CDI não está funcionando com Tomcat7 pelo maven + JSF

Question: Não estou conseguindo fazer o CDI funcionar, estou utilizando o plugin do tomcat7 do maven.
O beans.xml está criado e na pasta WEB-INF, o context.xml também e está na pasta META-INF, também coloquei as dependências do CDI, tanto implementação como especificação, também segue o código do bean com os imports do cdi e uma página jsf.
Segue o código
beans.xml

<beans>
</beans>

context.xml

pom.xml
<dependency>
    <groupId>javax.enterprise</groupId>
    <artifactId>cdi-api</artifactId>
    <version>1.2</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>org.jboss.weld.servlet</groupId>
    <artifactId>weld-servlet</artifactId>
    <version>2.3.2.Final</version>
</dependency>

MeuBean.java
import javax.faces.view.ViewScoped;
import javax.inject.Named;
import java.io.Serializable;

@ViewScoped
@Named
public class MeuBean implements Serializable {

    private Usuario usuario = new Usuario();

    public Usuario getUsuario() {
        return usuario;
    }

    public void setUsuario(Usuario usuario) {
        this.usuario = usuario;
    }

    public void testePrinta(){
        System.out.println(usuario);
    }

}

login.xhtml

Os seguintes logs do WELD aparecem:
INFO: WELD-ENV-001008: Initialize Weld using ServletContainerInitializer
nov 03, 2017 3:29:59 PM org.jboss.weld.bootstrap.WeldStartup <clinit>
INFO: WELD-000900: 2.3.2 (Final)
nov 03, 2017 3:29:59 PM org.jboss.weld.environment.deployment.discovery.DiscoveryStrategyFactory create
INFO: WELD-ENV-000020: Using jandex for bean discovery
nov 03, 2017 3:30:00 PM org.jboss.weld.bootstrap.WeldStartup startContainer
INFO: WELD-000101: Transactional services not available. Injection of @Inject UserTransaction not available. Transactional observers will be invoked synchronously.
nov 03, 2017 3:30:00 PM org.jboss.weld.event.ExtensionObserverMethodImpl checkRequiredTypeAnnotations
INFO: WELD-000411: Observer method [BackedAnnotatedMethod] com.sun.jersey.server.impl.cdi.CDIExtension.processAnnotatedType(@Observes ProcessAnnotatedType<T>) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.
nov 03, 2017 3:30:00 PM org.jboss.weld.environment.tomcat.TomcatContainer initialize
INFO: WELD-ENV-001100: Tomcat 7+ detected, CDI injection will be available in Servlets, Filters and Listeners.
nov 03, 2017 3:30:01 PM com.sun.faces.config.ConfigureListener contextInitialized

Answer: É só mover o arquivo beans.xml para a pasta
src/main/resources/META-INF


Exponential decay estimate for $(x,t) \in U_T$

Question:
Suppose $u$ is a smooth solution of
  $$\begin{cases}u_t - \Delta u + cu = 0 & \text{in }U \times (0,\infty) \\ \qquad \qquad \quad  \, \,u=0 & \text{on } \partial U \times [0,\infty) \\\qquad \qquad \quad  \, \, u=g & \text{on }U \times \{t=0\} \end{cases}$$ and the function $c$ satisfies $c \ge \gamma > 0$. Prove the exponential decay estimate $$|u(x,t)| \le Ce^{-\gamma t} \quad ((x,t) \in U_T).$$

This is from PDE Evans, 2nd edition: Chapter 7, Exercise 7.
Here is what I have worked on so far:
Let $v=e^{ct}u$. Then $v_t=e^{ct}u_t+ce^{ct}u$ and $\Delta v = e^{ct} \Delta u$. So our IBVP becomes $$\begin{cases}v_t - \Delta v = 0 & \text{in }U \times(0,\infty) \\ \qquad \quad v = 0 & \text{on } \partial U \times [0,\infty) \\ \qquad \quad v = e^{ct} g  &\text{on } U \times \{t=0\}. \end{cases}$$
The following convolution is a solution (see page 41 of textbook) to this auxiliary PDE: Edit: This convolution does not work; it does not satisfy the boundary conditions prescribed by the auxiliary IBVP. So the remainder of this solution does not follow.
$$v(x,t)=\frac 1{(4\pi t)^{\frac n2}} \int_U e^{-\frac{|x-y|^2}{4t}} [e^{ct} g(y)] \, dy,$$ which means in turn $$u(x,t)=e^{-ct} v(x,t)= \frac 1{(4\pi t)^{\frac n2}} \int_U e^{-\frac{|x-y|^2}{4t}} g(y) \, dy. \tag{$*$}$$
This is where I got stuck, because the $e^{-ct}$ factor is now gone. In an earlier attempt of this problem, I overlooked the $e^{ct}$ in the $e^{ct}g(y)$ part of the convolution, so I originally had this: $$u(x,t)=e^{-ct} \frac 1{(4\pi t)^{\frac n2}} \int_U e^{-\frac{|x-y|^2}{4t}} g(y) \, dy. \tag{$**$}$$
I was able to continue on from that point (while using that $g$ has compact support, so $|g(y)| \le M$):
\begin{align}
|u(x,t)|&\le e^{-ct} \frac 1{(4\pi t)^{\frac n2}} \int_U e^{-\frac{|x-y|^2}{4t}} |g(y)| \, dy \\
&\le Me^{-ct} \frac 1{(4\pi t)^{\frac n2}} \int_U e^{-\frac{|x-y|^2}{4t}} \, dy \\
&\le Me^{-ct} \cdot N \\
&= Ce^{-ct} \\
&\le Ce^{-\gamma t}.
\end{align}
My question is: Why am I getting stuck at $(*)$? I should arrive at $(**)$ instead, which would enable me to finish the proof.
Comment: Then you should consider $v = e^{d(t)} u$, where $d(t) = \int_0^t c(s) \, ds$.
Comment: Is $c$ a function of $t$ only, or a function of $t$ and $x$?  If the latter, you don't have $\Delta v = e^{ct} \Delta u$.  Maybe you are supposed to multiply both sides by $u$ and integrate over $D$, and then integrate by parts.
Comment: The problem doesn't say, but I think $c$ is based only on $t$. The process of deriving the solution was very similar to the one here: http://math.stackexchange.com/questions/543133/on-how-to-get-a-solution-for-a-nonhomogeneous-problem-for-the-heat-equation. Like in that other problem, I am trying to eliminate the $cu$ term of the PDE, when writing an auxiliary PDE.
Comment: Never mind, I found it: it is indeed $c(x,t)$, not $c(t)$. From page 372 of the textbook, the second-order differential operator is of the form $$Lu=-\sum_{i,j=1}^n a^{ij}(x,t) u_{x_i x_j} + \sum_{i=1}^n b^i(x,t) u_{x_i} + \color{blue}{c(x,t)}u.$$
Answer: I don't know if your approach could be made to work, but in my opinion you're going the wrong way. First of all, since $c=c(x,t)$, you can't just ignore the factor $e^{ct}$ when taking the laplacian of $v$. Moreover, the convolution integral will, in general, not be a solution of your problem: If $g>0$ then the convolution integral defines a strictly positive function on $\mathbb{R}^n$; in particular it can't satisfy the boundary conditions.
What you want to try is the maximum principle (See Theorem 9, Section 7.1 in Evans): Set $v=e^{\gamma t}u$, then $v_t-\Delta v =(\gamma-c)v$. Since $\gamma-c<0$ we can apply the maximum principle to get, for $(x,t)\in U_T$,
$$
-\max_{U}g^-=-\max_{\Gamma_T}v^-\leq \min_{U_T} v \leq \sup_{U_T} v =\sup_{\Gamma_T} v^+ = \sup_U g^+.
$$
This implies the desired inequality with $C\leq \sup_U |g|$.
Comment: How do we get $-\max_{\Gamma_T}v^-\leq \min_{U_T}v$? Shouldn't we need $v_t-\Delta v\geq 0$ for that?
Comment: sorry for insisting in such an easy question, but I am still missing something. I get the inequality $min(-v^-)\leq \min v$ holds on the same domain $U_T$, but then when I take the minimum on a smaller set don't I get something bigger i.e $\min_{\Gamma_T}\geq \min_{U_T}$?
Comment: @bibo: You're right, sorry you need the maximum principle here applied to the operator $L=\partial_t - \Delta -(\gamma -c)$ for which $v$ is both a subsolution and supersolution.


calling a ruby script in c#

Question: How do make a call to a ruby script and pass some parameters and once the script is finished return the control back to the c# code with the result?
Answer: void runScript()
{
    using (Process p = new Process())
    {
        ProcessStartInfo info = new ProcessStartInfo("ruby C:\rubyscript.rb");
        info.Arguments = "args"; // set args
        info.RedirectStandardInput = true;
        info.RedirectStandardOutput = true;
        info.UseShellExecute = false;
        p.StartInfo = info;
        p.Start();
        string output = p.StandardOutput.ReadToEnd();
        // process output
    }
}

Comment: This is a terrible answer. It has the classic "deadlock on redirection of both stdout and stderr" problem that the MSDN docs even go to lengths to warn people against. It is awful to write code that redirects both streams and then only consumes one of them.
Comment: @PeterDuniho I am not sure what to make of your comment since you have neither provided references nor a suitable fix. Would you recommend just removing the input redirect?
Comment: As my comment notes, this information is in the MSDN documentation. See e.g. [ProcessStartInfo.RedirectStandardOutput Property](https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandardoutput(v=vs.110).aspx). Search for the word "deadlock" on the page. You should have no trouble finding the information. Your code example does exactly what they warn against: redirecting both stdout and stderr, and then not reading either of them asynchronously (your example doesn't even read stderr at all, so it's not clear why you bothered with redirecting it).
Answer: Just to fill smaller gaps I've implemented the same functionallity with ability to access OutputStream asynchronously.
public void RunScript(string script, string arguments, out string errorMessage)
{
    errorMessage = string.empty;
    using ( Process process = new Process() )
    {
        process.OutputDataReceived += process_OutputDataReceived;
        ProcessStartInfo info = new ProcessStartInfo(script);
        info.Arguments = String.Join(" ", arguments);
        info.UseShellExecute = false;
        info.RedirectStandardError = true;
        info.RedirectStandardOutput = true;
        info.WindowStyle = ProcessWindowStyle.Hidden;
        process.StartInfo = info;
        process.EnableRaisingEvents = true;
        process.Start();
        process.BeginOutputReadLine();
        process.WaitForExit();
        errorMessage = process.StandardError.ReadToEnd();
    }
}

private void process_OutputDataReceived(object sender, DataReceivedEventArgs e)
{
    using ( AutoResetEvent errorWaitHandle = new AutoResetEvent(false) )
    {
        if ( !string.IsNullOrEmpty(e.Data) )
        {
            // Write the output somewhere
        }
    }
}

calling a ruby script in c#


Matrix Representation of Operators in Infinite Dimensional (Separable) Hilbert Spaces

Question: Suppose we have a separable Hilbert space (thus with a countable basis) and that we to represent an operator in matrix form, i.e:
$$A: H \rightarrow H \\ \; \; \; \; \; \;x \;\rightarrow \sum_{j \in \mathbb{N}}\left(\sum_{k \in \mathbb{N}} a(j,k)\cdot<x,e_k> \right)e_j$$
Given that the series of complex numbers $\sum_{k \in \mathbb{N}} a(j,k)\cdot<x,e_k>$ converges and that $\sum_{j \in \mathbb{N}}\left(\sum_{k \in \mathbb{N}} a(j,k)\cdot<x,e_k> \right)e_j$ converges in $H$ how do I prove without further assumptions that $A$ is a bounded operator?
I can prove that for each $j \in \mathbb{N}$ the sucession $(a(j,k))_k \in l^2 (\mathbb{N},\mathbb{C})$  and that for each $k \in \mathbb{N}$ the sucession $(a(j,k))_j \in l^2 (\mathbb{N},\mathbb{C})$.
However I always get to a point where I require that $\sum_{j \in \mathbb{N}}\sum_{k \in \mathbb{N}} \left |a(j,k) \right|^2 < \infty$ which is a condition that I can show to be necessary and sufficient for $A$ to be a Hilbert-Schmidt operator. Any help would be welcome.
Comment: Yes, I meant a bounded operator. I edited it, thanks.
Answer: As pointed out by Elias, the Uniform Boundedness Principle (UBP) is the key. But the assumption does not imply that $A$ be Hilbert-Schmidt, as it is satified by every bounded linear operator $A$ on $H$. Indeed, denoting by $P_n$ the finite rank projection onto $\mbox{span}(e_0,\ldots,e_n)$, we have that $P_n$ tends to the identity in the strong operator topology (i.e. pointwise). So for every $n$, $P_nAP_mx$ tends to $P_nAx$ as $m$ tends to $+\infty$, which proves that the coordinates of $Ax$ in the basis are given by the converging series $\sum_{k=0}^{+\infty} A_{jk}\langle x,e_k\rangle$ where $A_{jk}=\langle Ae_k,e_j\rangle$ are the matrix coordinates of $A$ in the basis. And then $P_nAx$ tends to $Ax$ as $n$ tends to $+\infty$, which proves the second part, namely that $\sum_{j=0}^{+\infty}\left(\sum_{k=0}^{+\infty}A_{jk}\langle x,e_k\rangle\right)e_j$ converges to $Ax$ in $H$.
Conversely, which is what you are asking about, assume that $(A_{jk})$ is a doubly infinite sequence such that $\sum_{j=0}^{+\infty}\left(\sum_{k=0}^{+\infty}A_{jk}\langle x,e_k\rangle\right)e_j$ has converging coordinates, and converges in $H$ for every $x\in H$. We will show altogether that this defines a bounded linear operator on $H$. For every $n,m$, consider the bounded linear operator $A^{(n,m)}$ defined by $A^{(n,m)}x:=\sum_{j=0}^n\left(\sum_{k=0}^mA_{jk}\langle x,e_k\rangle\right)e_j$. By the first part of the assumption, we see that $A^{(n,m)}$ converges to $A^{(n)}$ pointwise as $m$ tends to $+\infty$, where $A^{(n)}x:=\sum_{j=0}^n\left(\sum_{k=0}^{+\infty}A_{jk}\langle x,e_k\rangle\right)e_j$. By the UBP, it follows that $A^{(n)}$ is a bounded linear operator on $H$. Finally, note that the second part of the assumption yields that $A^{(n)}$ converges pointwise to $A$ given by $Ax:=\sum_{j=0}^{+\infty}\left(\sum_{k=0}^{+\infty}A_{jk}\langle x,e_k\rangle\right)e_j$ as $n$ tends to $+\infty$. By the UBP again, we can conclude that $A$ is a bounded linear operator on $H$.
Answer: Hint Use the Uniform boundedness principle or better yet  just do $x=\sum_{v=1}^{\infty}\sum_{u=1}^{\infty}a(u,v)e_u$ in $\sum_{j \in \mathbb{N}}\left(\sum_{k \in \mathbb{N}} a(j,k)\cdot<x,e_k> \right)e_j<\infty$  for proving $\sum_{j \in \mathbb{N}}\sum_{k \in \mathbb{N}} \left |a(j,k) \right|^2 < \infty$.
Comment: The series $\sum_{u=1}^{+\infty}e_u$ does not define a vector of $H$.
Comment: Hmm, we didn't learn the uniform boundedness principle so far. About the second aproach: how to we know that this particular x converges? If it does that would seem to prove that every bounded operator is Hilbert-Schmidt operator
Comment: Hmm, i got it by another means: using the Jensen inequality and using some results about $l^2$ sequences. Thanks for the help!
Comment: @julien I corrected my little mistake in my answer.
Comment: Hmm nvm, it doesn't work: i can prove that $\sum_{k \in \mathbb{N}} \left |a(j,k) \right |$ converges but i also need (in order for my argument to work) that $\sum_{j \in \mathbb{N}} \left |a(j,k) \right |$ which I cannot prove


Fingers 'stuck' on keys

Question: I keep finding that when moving my hand and playing a note - especially when crossing my index finger over my thumb - the note is already held down by another finger. i.e. I'm not lifting the note when it's played a beat or two earlier.
I'm aware I'm doing it but can't seem to avoid it, as a new player playing two hands at once I don't have the mental capacity to remember to concentrate what each finger is doing.
Is this a common beginner error and are there specific exercises I can do to train my fingers not to linger on keys?
Answer: Releasing notes after the correct duration is all part of practising. Just as the attack of a note starting at the right time is important, so is the release.
I practice this by slowing the tempo right down, by half or even more. Whatever you need to give yourself enough 'thinking time'. Then really focus on each note length and when notes in each voice get released. Only once you can play the passage precisely at a slow speed should you gradually speed it up.
Answer: Widor's suggestion of slowing down is great.  Another than can be used in conjunction with it or on its own is to play staccato.  Staccato obviously necessitates lifting your fingers back off the keys, and it will change the sound drastically to ensure that you are concentrating on it.  It also has the added benefit of building strength, which can help you maintain more control when you go back to playing normally.
Comment: I think my natural style is to play very legato so this might be good, by deliberately aiming to play staccato I might play more as the piece is intended.
Answer: Great responses.  I recommend the following (I used to do this often):
Take any given passage that involves that "finger stuck" problem.  Using the correct fingering as if you would normally play it (it would be helpful to write down the correct fingering if you are not comfortable remembering it), play those two notes (thumb and index finger) while also playing one (or two) note above and below those two note.
Slowly go up and down, up and down, gradually increasing tempo.  Then, play staccato as Matthew Read suggested.  Then, change the rhythm up.  Then change the rhythm in a way where one (or more) notes are staccato, but one (or more) notes are non-staccato.
By "pushing" your fingers to do more, and to experience different positions, you will develop better control.
Really it comes down to what Widor said: practice.  But yes, you must practice in a smart way (playing a short few notes in a variety of ways would be considered smart, IMO).
As a matter of fact, now that we are discussing this, I actually do this a LOT without even realizing it since I like to do a lot of improv.  I'll be at my piano, and I'll find myself playing the same phrase multiple times in different ways <- usually when I'm just having fun and "messing" around.
Comment: This is a good suggestion as well.  Sometimes learning to do something one way means learning all the other ways it can be done, and then not doing them :P
Answer: This is a common beginner problem, and there are exercises you can use to help you. It sounds like you want to gain independence of the fingers. In addition to finding a reputable exercise volume that includes finger independence exercises, you can do the following things:

Practice scales, two to four octaves up and down, all twelve keys, especially with contrary motion (one hand goes up while the other goes down).
Practice the passages you have difficulty with very slowly, maybe 1/4 tempo, which will reduce the amount of time it will take you to learn the passage. Also play one note at a time out of rhythm.
Play places where you have trouble one note group, and then the next, and then repeat (out of rhythm).
Comment: Make that all twelve keys, both major and minor.  Of course, you probably meant that, but most of my students will never work on the minor keys unless I keep pestering them about it.  :)

Fingers 'stuck' on keys


Correcting for outliers in a running average

Question: We have a daemon that reads in data from some sensors, and among the things it calculates (besides simply just reporting the state) is the average time it takes for the sensors to change from one value to another. It keeps a running average of 64 datapoints, and assumes that runtime is fairly constant.
My obvious solution for dealing with this would be to create a histogram of the data and then pick the mode. However, I was wondering if there were other methods that would yield better performance or would be more suited for operation with a running average. Some quick Wikipedia searches suggest algorithms for detecting outliers may be also suitable. Simplicity is a plus, since the daemon is written in C.
Edit: I scoped out Wikipedia and came up with these various techniques:

Chauvenet's criterion: using the mean and standard deviation, calculate the probability a particular datapoint would happen, and then exclude it if the probability is actually that bad is less than 50%. While this seems to be well suited for correcting a running average on the fly, I'm not quite convinced of its efficacy: it seems with large data-sets it would not want to discard datapoints.

Grubbs' test: Another method that uses difference from the mean to standard deviation, and has some expression for when the hypothesis of "no outliers" is rejected

Cook's distance: Measures the influence a datapoint has on a least squares regression; our application would probably reject it if it exceeded 1

Truncated mean: Discard the low end and the high end, and then take the mean as normal

Anyone have any specific experience and can comment on these statistical techniques?
Also, some comments about the physical situation: we're measuring the average time until completion of a mechanical washing machine, so its runtime should be fairly constant. I'm not sure if it actually has a normal distribution.
Edit 2: Another interesting question: when the daemon is bootstrapping, as in, doesn't have any previous data to analyze, how should it deal with incoming data? Simply not do any outlier pruning?
Edit 3: One more thing... if the hardware does change such that the runtimes do become different, is it worth it to make the algorithm sufficiently robust such that it won't discard these new runtimes, I should I just remember to flush the cache when that happens?
Comment: During bootstrapping provide 64 data points to seed the running average.
Comment: But what would those 64 data points be? It might not be good to assume a priori that the machine would run in X time.
Comment: Grubbs' test is superior to the other methods because it is computationally cheaper to determine regression coefficients than to sort and rank data.
Comment: The system has to know that it is being calibrated and bootstrapping itself.  Typical techniques are to seed the first values with the initial value plus or minus ten percent.  When the second value arrives the seeded values are the first and second values interleaved.
Comment: Interesting! I agree that I probably want to avoid sorting data, which would make implementing a truncated mean difficult.
Comment: If a known event is going to change the distribution it is essential to recalibrate.  In a mass spectrometer the device becomes more accurate as it adjusts itself to the compound being analyzed.  This is a well known method to increase accuracy while supporting a large variety of inputs.
Answer: If that example graph you have is typical, then any of the criteria you list will work.
Most of those statistical methods are for riding the edge of errors right at the fuzzy level of "is this really an error?"  But your problem looks wildly simple.. your errors are not just a couple standard deviations from the norm, they're 20+.  This is good news for you.
So, use the simplest heuristic. Always accept the first 5 points or so in order to prevent a startup spike from ruining your computation. Maintain mean and standard deviation. If your data point falls 5 standard deviations outside the norm, then discard it and repeat the previous data point as a filler.
If you know your typical data behavior in advance you may not even need to compute mean and standard deviation, you can hardwire absolute "reject" limits. This is actually better in that an initial error won't blow up your detector.
Comment: Even if it does need to be adaptable, put some EXTREME limits in there anyway. If a hardware glitch gives you 1e280 as a value, or NaN or +Inf, you may want to filter those out regardless.
Comment: Excellent! One thing though; although we do know the typical behavior, it is possible that the hardware will get switched out, so I feel that an adaptable program would be preferred.
Answer: The definition of what constitutes an abnormal value must scale to the data itself.  The classic method of doing this is to calculate the z score of each of the data points and throwing out any values greater than 3 z scores from the average.  The z score can be found by taking the difference between the data point and the average and dividing by the standard deviation.
Comment: How well would this method work for the pathological blue line case?
Comment: The pathological blue line case has a high standard deviation.  It would take a significant outlying value to get rejected.
Answer: I would compute a running median (robust alternative to mean) and a running mad (robust alternative to sd), remove everything that more than 5 mad's away from the median. More on this here (archived).
Comment: One thing to keep in mind is the computational complexity of the running median: The naive implementation requires a sort each time a new data point is added. In any case, all previous data points (at least in a certain window) have to be stored.
Answer: Another solution is to use the harmonic mean.
Your case is very similar to the example discussed here (archived).
Comment: Dude, harmonic mean FTW! People should really take a look into this one!
Answer: You need to have some idea of expected variation or distribution, if you want to be able to exclude certain (higher) instances of variation as erroneous. For instance, if you can approximate the distribution of the "average times" result to a normal (Gaussian) distribution, then you can do what ojblass suggested and exclude those results that exhibit a variation that is greater than some multiple of the standard deviation (which can be calculated on the fly alongside your running average). If you wanted to exclude results that have a 99.75 (or so) percent chance of being erroneous, exclude those that vary more than 3 standard deviations from the mean. If you only want 95% certainty, exclude those that vary more than 2 standard deviations and so on.
I'm sure a little bit of googling for "standard deviation" or "gaussian distribution" will help you. Of course, this assumes that you expect a normal distribution of results. You might not. In which case, the first step would be to guess at what distribution you expect.
Answer: The naive (and possibly best) answer to the bootstrapping question is "Accept the first N values without filtering." Choose N to be as large as possible while still allowing the setup time to be "short" in your application. In this case, you might consider using the window width (64 samples) for N.
Then I would go with some kind of mean and sigma based filter.


Uber jar not found in Kubernetes via spark-submit

Question: I have a very simple Spark job, but I can't get it to work in Kubernetes. The error I get is:
>     19/10/03 14:59:51 WARN DependencyUtils: Local jar /opt/spark/work-dir/target/scala-2.11/ScalaTest-assembly-1.0.jar does
> not exist, skipping.
>     19/10/03 14:59:51 WARN SparkSubmit$$anon$2: Failed to load ScalaTest.
>     java.lang.ClassNotFoundException: ScalaTest
>       at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>       at java.lang.Class.forName0(Native Method)
>       at java.lang.Class.forName(Class.java:348)
>       at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
>       at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:806)
>       at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
>       at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
>       at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
>       at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
>       at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
>       at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Project structure:
project/build.properties
project/plugins.sbt
src/main/scala/ScalaTest.scala
Dockerfile
build.sbt

build.properties
sbt.version=1.2.8

plugins.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.10.0-RC1")

ScalaTest.scala
import org.apache.spark.sql.SparkSession
import org.apache.spark.SparkContext

object ScalaTest {
  def main(args: Array[String]) {
    val spark = SparkSession.builder.appName("ScalaTest").config("spark.master", "local[*]").getOrCreate()

    import spark.implicits._

    println("hello")

  }
}

Dockerfile This is just a wrapper image based on the one built from the kubernetes folder within Spark binaries. Before building this image I make sure I run sbt assembly, which generates the Uber jar.
FROM spark:latest

WORKDIR /opt/spark/work-dir

COPY target/scala-2.11/ScalaTest-assembly-1.0.jar target/scala-2.11/ScalaTest-assembly-1.0.jar

build.sbt
name := "ScalaTest"

version := "1.0"

scalaVersion := "2.11.12"

val sparkVersion = "2.4.4"

libraryDependencies ++= Seq(
    "org.apache.spark" % "spark-core_2.11" % sparkVersion % "provided",
    "org.apache.spark" % "spark-sql_2.11" % sparkVersion % "provided"
)

Finally my spark-submit. Before I do this I pushed the image to ECR's registry, so EKS can pull that image. I also point to the location of the uber jar within my image.
~/spark-2.4.4-bin-hadoop2.7/bin/spark-submit \
    --master k8s://{K8S_ENDPOINT}:443 \
    --deploy-mode cluster \
    --name test-job \
    --conf spark.kubernetes.container.image={ECR_IMAGE}:latest \
    --conf spark.kubernetes.submission.waitAppCompletion=false \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.driver.pod.name=test-job \
    --class ScalaTest \
    local:///opt/spark/work-dir/target/scala-2.11/ScalaTest-assembly-1.0.jar

Also note, when I run the command below (spark-submit within my container, locally) it works as expected:
docker run --rm -it my-custom-image ../bin/spark-submit target/scala-2.11/ScalaTest-assembly-1.0.jar

UPDATE
Inspecting the assembled uber jar, I can see ScalaTest's class is there.
jar tf target/scala-2.11/ScalaTest-assembly-1.0.jar

...
ScalaTest$.class
ScalaTest.class
...

Answer: The solution to this is to leave the jar not in the working dir but in the jars folders. I haven't looked at the docs but probably this is an environment variable that can be changed. In any case, the Dockerfile should look like this:
FROM spark:latest

COPY target/scala-2.11/ScalaTest-assembly-1.0.jar /ops/spark/jars/ScalaTest-assembly-1.0.jar

And then change the spark-submit accordingly:
~/spark-2.4.4-bin-hadoop2.7/bin/spark-submit \
    --master k8s://{K8S_ENDPOINT}:443 \
    --deploy-mode cluster \
    --name test-job \
    --conf spark.kubernetes.container.image={ECR_IMAGE}:latest \
    --conf spark.kubernetes.submission.waitAppCompletion=false \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.driver.pod.name=test-job \
    --class ScalaTest \
    local:///opt/spark/jars/ScalaTest-assembly-1.0.jar


Installing @types/jquery results in an unwanted global $ declaration

Question: I have the following defined in package.json:
{
  "dependencies": {
    "@types/jquery": "^3.5.5",
  }
}

Which adds the following type declaration via @types/jquery/misc.d.ts:
declare const jQuery: JQueryStatic;
declare const $: JQueryStatic;

Now I have global $ and jQuery variables defined for all of my modules, but this is not only unwanted... it's wrong. Code similar to the following is blowing up at runtime because the TypeScript compiler believes that $ is a global when in fact it must be imported:
// $ is never imported here, and jQuery is not initialized globally in the browser

export function foo(node?:JQueryXML):void {
    node.children().each(function(_, el) {
        let node = $(el);
        if (node.attr("type") === VALUE) {
            doStuff();
        }
    });
}

Just to be sure, I tried commenting out those two declaration lines in @types/jquery/misc.d.ts and I got the expected error:
error TS2592: Cannot find name '$'. Do you need to install type definitions for jQuery? Try `npm i --save-dev @types/jquery` and then add 'jquery' to the types field in your tsconfig.

40    let node = $(el);

However, I've now effectively disabled all jQuery type checking. I've got an additional 50+ errors because all other modules where I correctly import $ are treating the variable as any.
How can I tell the TypeScript compiler that I want the type definitions from @types/jquery but that $ is not a valid global variable?
Comment: This is very annoying indeed. My project uses jQuery 1 and we can't upgrade it because plugins, but we import jQuery 3 in some places. For typescript, everything is v3, which makes it even more confusing than it is already.
Answer: You can explicitly tell TypeScript the types to search in the compiler options: https://www.typescriptlang.org/tsconfig#types
From the docs:

By default all visible "@types" packages are included in your compilation. Packages in node_modules/@types of any enclosing folder are considered visible. For example, that means packages within ./node_modules/@types/, ../node_modules/@types/, ../../node_modules/@types/, and so on.
If types is specified, only packages listed will be included in the global scope.

In this case the compiler is automatically including the global type from node_modules/@types/jquery.
Comment: I don't know how the jQuery TS types are defined as I used jQuery like 10yrs ago. However, I had a similar issue with the NodeJS global types. Sadly it maybe a problem on how the jQuery types are defined. At the beginning of the compilation TSC will scan for imports to resolve types, you can turn that off with the flag `--noResolve`. In that case it will only resolve local imports (and it will give you tons of errors). During the resolution phase TSC imports the req type def, if the definition is global.. you'll have it on the scope :( Perhaps there is an alternative @types/jquery
Comment: Aside from the fact that I still need jQuery types, I tried your suggestion with the following configuration: `{ "compilerOptions": { "types": [ "moment" ] } }` and TypeScript is still compiling without any errors. If I comment out those to constant declarations in misc.d.ts then I get an error, so I know that the file is being compiled by TypeScript. I really need a reliable way to disable those globals (`$` & `jQuery`) without sacrificing all the useful type information.
Comment: I've added additional details to my question.
Comment: If you removed it from `types` but the type is still there it may be the case of a source file that still includes the global jQuery. Try with `tsc --traceResolution` the output will be very verbose but it may help to find what's going on.
Comment: Note: `types` indicates types to automatically include, but you may still include the types by using an `import`, you can use `import $ from 'jquery'`. If a source file uses `import 'jquery'' it will include the globals as well.
Comment: I've tried running with `--traceResolution`, but I admit that I'm a little confused. I already know that jQuery is imported by other modules in my bundle. The issue is that I forgot to import it in this particular module but the linter didn't detect that, so at runtime `$` was undefined. `@types/jquery` appears to declare a global `$`, but my webpack configuration does not resolve a global `$`. I need TypeScript to catch these sorts of errors in the future, but I can't get it to flag a missing `import $ from "jquery"`
Comment: Thanks, I appreciate the help.


Is it possible to apply conditional formatting in Google App Maker?

Question: I have a table in Google App Maker, I am wondering is it possible to apply conditional formatting to rows of the table.
Example:
If the value of a drop down is "Yes" set that row's background colour to be red.
The documentation regarding this is useless as always and wasn't able to find any questions regarding this.
Image:

Greyed out Image:
Answer: The secret is in your bindings. If you wish to retain the regular assigned styles like 'app-ListTableRow' and 'hoverAncestor' then do the following:
Add a class in your Style Editor like this for example:
<code>.red {
  background: linear-gradient(to bottom, darkred, red);
}
</code>
On your table row 'Display' - styles enter the following binding:
<code>@datasource.item.CertificateisRequired === 'Yes' ? ['red','app-ListTableRow','hoverAncestor'] : ['app-ListTableRow','hoverAncestor']
</code>
That will do it.
If you have a dropdown in each row I would suggest to call it 'CertificateisRequired' instead of leaving a default name like 'DropDown1' or whatever App Maker assigns to it. Then adjust the binding in the row Display - styles to:
<code>@widget.descendants.CertificateisRequired.value === 'Yes' ? ['red','app-ListTableRow','hoverAncestor'] : ['app-ListTableRow','hoverAncestor']
</code>
Comment: You have to change the styles setting to binding instead of value. Then paste your binding in. You cannot add a binding as a list item in the styles list. Click on the equal sign in the styles dropdown and it gives you a dropdown of binding or value.
Comment: Hey Markus, thanks again for the help. If I understand correctly the above will look at the inherited datasource, look at the row(s) and look at the value in CertificateisRequired and if it is yes, apply the styling I set in the Style Editor. (That all makes perfect sense.) Only small problem is CertificateisRequired is not a field in the Certificate's datasource. It's just a plain old drop down widget with a value preset to be "No." Logic is solid though, will try working it out. (Thanks again, awarding as answer.)
Comment: Thanks Markus, tried to add it. The button to "add" style is greyed out though. Will add an image above. What I am trying to add:@widget.descendants.Dropdown_Cert_Required_Value.value === 'Yes' ? ['red','app-ListTableRow','hoverAncestor'] : ['app-ListTableRow','hoverAncestor']


Ping Link Local Address

Question: We have an embedded networking device product which we first develop and test on Windows platform. It was observed that based on given conditions the device got a Link Local address - 169.254.106.235.
But when I try to ping the above IP from command prompt it says 'Dest. Unreachable' or error. Now if I setup a loopback adapter with obtain configuration automatically - I see that the adapter has obtained IP 169.254.233.20
Now when I ping both my device and loopback adapter IP I start getting response from both - but then again when I delete or set up the IP manually for loopback adapter - the device becomes unreachable
How can we be able to ping my device in first condition - no loopback adapter setup?
Answer: As pointed out in your question Link Local Addressing - Communication with Routable Addresses there is no way to perform routing into the link-local address space.
This means the interface involved in the communication with your embedded device needs a link-local address as well. In case you do not feel well about using loopback interfaces, you can set your windows machine to DHCP on the physical interface pointing towards the embedded device. As there seems to be no DHCP server, windows will assign a link-local IP address.
Comment: RFC3927 does allow hosts with routable addresses to communicate with hosts with link local addresses, this is described in section 2.6.2.  But it isn't clear to me whether Windows can be configured to do this.
Comment: @HarryJohnston you are absolutely right. The trick might be given as well _In many network stacks, achieving this functionality may be as simple
   as adding a routing table entry indicating that 169.254/16 is
   directly reachable on the local link._
Comment: I'm pretty sure this is configurable, but I have no idea how. :-)  Via netsh, perhaps, or Powershell cmdlets.  I guess there's documentation somewhere.

Ping Link Local Address


MDS: Is Kruskal's Stress-1 affected by scale of the data, or the number of points?

Question: In Multidimensional Scaling, Kruskal's Stress-1 is a commonly used measure of fit.
It is defined as:
$\sqrt{\frac{\sum (d_{ij}-\delta_{ij})^{2}}{\sum d_{ij}^{2}}}$
where $d_{ij}$ represents the distances, and $\delta_{ij}$ represents the disparities.
I'm looking to use it to compare across studies in which there are differing numbers of data points, and in which the scales are different. Is this measure unaffected by the scale, and by the number of points? Why/why not?
As for what scale means, imagine that in one study the MDS related to distances between cities measured in miles, while in the other the distances between cities were measured in kilometres.
I would have thought that part of the point of normalization was to ensure that comparisons across studies with different numbers of data points could be made. However, I sometimes see diagrams like the following

That diagram shows that when tested on random data, Stress-1 increases with more points.
Comment: `measured in miles, while in the other... measured in kilometres` Most popular MDS algorithms (such as PROXCAL, ALSCAL) normalize the input dissimilarities to their sum equal the number of objects. At every iteration, transformed dissimilarities (the disparities)  are also normalized samely before the coordinates are computed and distances on the map are reckoned. So, there should be no effect of the _units of dissimilarities_ on the result, I believe. Just try yourself: MDS on kms vs MDS on miles - should not differ.
Comment: As for the _number of objects_ - yes it has some effect, as shown by your graphic and acknowledged in my comment.
Comment: I suppose it could be deduced from the formula, isn't it? But what do you mean by different "scale" in your context?
Comment: I edited the OP the clarify what I meant by "scale".
Comment: I see. Sure, it depends. The more are the points - hence dissimilarities - the greater is the constraint to adequately,with less loss, transform them to the disparities. Note that (1) the difference diminishes as the number of points grows. The trend is about the same for different dimensionalities. So, it is possible to model the trend between the lined shown on your pic and add a "standardizing" correction to your formula. But note the picture of yours will change with metric MDS as opposed to nonmetric. I expect so.
Comment: Thanks, this is really helpful. I've asked a related question [here](http://stats.stackexchange.com/questions/215058/mds-are-there-measures-of-stress-that-are-not-affected-by-the-number-of-objects).
Answer: Scale should make no difference. But, all else being equal, the greater the number of points, the higher the stress.
As ttnphns comments, the cause of this is that when you have fewer observations, the model will over-fit, so the stress is downwardly biased. As the number of observations grow, the extent of the bias reduces.
Pretty much every measure of goodness-of-fit with a fixed minimum and maximum, as in this case, suffers from the same problem. For example, R-squared goes down as the number of observations go up, all else being equal, and the Adjusted R-squared was developed to address this. While I do appreciate that it would be great to have measures that were not influenced by the number of observations, as the degree of "noise" is going to differ from problem to problem, this is probably not solvable (e.g., Adjusted R-squared is not used by people with a good knowledge of regression).
You can compare between different data sets by randomly sampling the number of observations in the larger data set. For example if data set 1 has 20 observations, and data set 2 has 30, randomly sample 20 from data set 2 and compare the stress with 1. If you repeat this multiple times you will be able to do a significance test comparing the stress levels.
Comment: Despite my (+1), I would question the "equivalence" approach via random samples that you describe in the last paragraph. The doubt is about our right to consider a subset of points as representative of the whole cloud of points.
Comment: (cont.) Say, we have 4 points ABCD. Your stance seem to claim that every triplet like ABC ABD ACD BCD each can "represent" the whole set; and so doing mds (say, 1dim mapping) on the 4 triplets and then averaging the stress is "equivalent" to 1dim mds mapping of ABCD, the only difference in the two stresses being due to that the averaged one is smaller because it was based on the analyses of 3 points while the second was based on the analysis of 4 points.
Comment: (cont.)   That philosophy is questionnable to me. I'd say, the 4 points ABCD is the _fixed_ population of points of which you have no right to sample (triplets or dyads).
Comment: To my mind the 4 points are a random sample and thus the 3 points are a random sample as a random sample of a random sample is a random sample. But, having said that, I am not familiar with the concept you mention of "have no right to sample", so perhaps I am misunderstanding something here.
Comment: For me, the 4 points in the mds are a not a random sample, nor a sample at all. They are the aspects of reality under the study. Like, for example, the set of variables/features in a MANOVA or other multivariate analysis: these are not a sample of features from some population of features; any selected subset of the features do not pretend to represent the entire set. The same is for a mds, - in mds our points under the study are the "features" of our study, not the units of a sample. We don't have sample here at all: mds is a mapping gimmick, not a sort of inferential statistical analysis.
Comment: It is a fair point that it is unlikely to be a random sample, but, how many random samples really existing in research anyway? But, it is to my min a sample, and not a population. Domain sampling theory and generalizability theory in psychometrics both accommodate the idea that stimuli in experiments can be considered as samples. Similarly, time series methods employ inference, and these are applied to time series data sets that would, I suspect, comply with your definition of a population. And, in the case of MANOVA, you can have random effects.
Comment: My simple thought was that just averaging over the results of analyses done on subsets cannot give clues to infer about the result that could be obtained on the whole set. In a case such as ours with MDS. Without special assumptions and rules (how to synthesize) we may not extrapolate the combination of results from {ABC, ABD, ACD, BCD} onto the expected result from ABCD. I consider it to be exactly the _gestalt_ situation. A gestalt does not break up into sample units.
Comment: I agree. But, ultimately the choice for user1205901 is to do what he/she wants, by making strong assumptions, or, not be able to do anything. In such situations I am always with George Box and the all models are wrong stream of workflow.


Jquery animate opacity and margin on other element

Question: I have an link that when clicked, has to show 2 other links.
But when I click my link, nothing happends. Really nothing.
I have loaded in jquery before I have my script.
This is my jquery script:
   $("#imd").toggle(function(){
    $("#anw").animate({opacity:1},200);
    $("#3d").animate({marginTop:75}, 200);
},function(){
    $("#anw").animate({opacity:0},200);
    $("#3d").animate({marginTop:0}, 200);
});

This is my CSS:
#anw {
    width: 50%;
    background-color: #0f0;
    opacity: 0;
}

And this is my HTML markup

Interactive Media Design

AppsWebsites

3D Modelling

Contact and about me

My apologies if this is a very dumb question, but I have no clue.. Also the documentation I can find is not filling my needs.
Comment: What link do want to hide when clicked and what links do want to show when clicked?
Answer: You need to work with a click handler:
$("#imd").click(function(){
    $("#anw").toggle(function(){
        $(this).animate({opacity:1},200);
    }, function() {
         $(this).animate({opacity:0},200);
    });
    $("#3d").toggle(function() {
        $(this).animate({marginTop:75}, 200);
    },function() {
        $(this).animate({marginTop:0}, 200);
    })
});

See the Fiddle: https://jsfiddle.net/8Lpvty1w/
Answer: Working fiddle for what you want to do: https://jsfiddle.net/rr9Lvjgh/
Essentially:
 $("#imd").on('click', function(){
  // code to change stuff here
 });


How to practice my ears to identify scales or notes?

Question: I know it comes by practice; my teacher said this. But I want exercises for this, so that I will come to know which note or scale is being played.
Where should I start from?
Comment: It's not clear whether you want to achieve absolute or relative pitch. The latter is far easier to attain, and is a better start point.
Answer: What I'd suggest is to pick up your instrument (any instrument) and play a scale and sing it. Because when you simply play a scale, you'll simply listen to it. But when you sing it, you'll relate the sound you are hearing with the notes.
After you've done that, play the scale up and down moving in thirds, then in fourths, fifths etc.
( E.g the C major scale moving up in thirds would be C-E, D-F, E-G etc)
This way you'll be able to hear all the intervals included in the  scale as well. Again, you won't simply be listening to the intervals, you'll be able to relate what you theoretically know with what you play.
This has helped me a lot in my ear training.
Comment: Don't just sing 'lah' though. Either sing the note names, which will help in that particular key, or sing the note numbers, which will give the same intervals in any key. Thus C-E = 1-3. D-F = 2 -4 in C will be the same as, in F, F-A =1-3, G-Bb =2-4, etc.
Answer: I break down ear training into three things:
1. Intervals
Identify the intervals.  There's the eight we're used to hearing most of the time, as well as the extended intervals.  Work on the ones inside the octave first, then venture out above the octave.  The easiest way to start recognizing intervals is to associate them with melodies or intervals you encounter in real life.  This includes the beginning of the Star Wars theme or "Here Comes the Bride", etc.  Here's a great reference to associations to get you started.  Most intervals we here are going up, so start with intervals ascending.  Once you feel comfortable with that, try them going down.  Then check out the intervals above an octave.
This is just like a language.  Repetition is key.
2. Scales
When working on scales you have three you'll hear often; major, minor and melodic minor.  You may even encounter others, especially if you're venturing into jazz.  Start small and listen to the scale ascending and descending (I'm looking at you, melodic minor!) because it can change slightly.  Once you feel comfortable identifying the standards, venture to pentatonics, altered scales, and non-western scales.  It's fun!
3. Chords
Chords are similar to scales in that you have a few staples you need to be able to recognize before moving on to more advanced stuff.  Focus on types (Major, Minor, Augmented and Diminished) of triads just getting the basics.  Once you feel comfortable with those start expanding the chord extensions.  Start to recognize the same 4 with 7ths, then 9ths, then 11ths, then 13ths.  After you get there (be sure to listen to what's on top of the triad) you can start to take a look at altered chords including 6/9, sus4, etc.

Practice all of these with - and away from - your primary instrument.  And don't focus on a key center, try to be as abstract and random with your interval/chord/scale keys as possible.  This makes you focus on the note relationships and not diatonic analysis.  There's associations you'll subconsciously make with the instrument you play and these intervals, but the retention rate skyrockets when you can do this away from the instrument and work on you "inner ear".
The key when working on this is again repetition.  If you don't use it you'll lose it.  if you do it enough it'll become second nature and you'll be able to understand and speak a completely different language with fluency.  As you move along, try to analyse the stuff you hear, intervals in nature, or chords, or scales are everywhere.  Try to identify what you hear, then use handy mobile tools (see below) to check your answers.  For grins, try to recognize pitches away from an instrument as well.  You'll start to notice a lot of environment sounds revolve around standard pitches and intervals.
All three of these things can easily be practice with mobile applications (you always have a phone, tablet or music player on you most of the time).  My particular favorite (though entirely opinionated) is Tenuto.  It's available on mobile devices as well as the browser and comes from the good people at musictheory.net
Answer: my suggests :

ear music that you love and find the scales (memory auditive based on example)
use apps in iOS appstore or android store (lot of tools)
create your own audio files that contains scales examples for replay those
listen passive (on working) and listen active (in subway)
work on your instrument based on few differents scales for memory since 1 week
you have also this : Golden Ears


Statistical test for non uniform counts

Question: I have data on counts for about 1000 categories in a sample. I want to get an estimate if the counts are somewhat uniformly distributed across 1000 categories, or most of the counts are cominf from only few categories and rest are either 0 or minimal.
Any idea what could be a good statistical test, and visualization method? if these were not 1000 but 5 categories, pie-chart would be perfect to get a visual feel.
Also, if I have different samples and want to find relative non-uniform property one sample vs the other, what could be a good way.
Thanks
I want to elaborate my question. Suppose i have only 10 categories instead of 1000, i want to see if the counts look like

Blockquote

5,5,5,5,5,5,5,5,5,5 or

Blockquote

0,0,0,0,0,50,0,0,0,0.

Blockquote

It is possible that all 50 counts are assigned to a single category rather than distributed across 10. I want to reject this sample only accept if they are reasonably distributed across all 10 categories. What could be a good statistical test, and visualization way for this?
Comment: How many data points do you have in total?
Comment: I have a 1000 data points, corresponding to 1000 categories.
Comment: Hm. I may have been unclear. What is your *total count*, i.e., the total number of samples? For instance, in your example, this total count would be 50. (This is important because the $\chi^2$ test [Ertxiem proposes](https://stats.stackexchange.com/a/457972/1352) needs a sufficiently large number of expected counts per bin.)
Comment: Lets say I have only one sample, total counts is variable but would be well above 5000. I want to find if this 5000 belongs to only few categories or distributed across all 1000 categories in that sample.
Answer: If the data sample is large enough (I would say at least 10 times the number of categories), then you may apply a chi-square test of homogeneity for an uniform distribution.
Regarding the graphical methods, consider a bar chart showing a subset of categories, for instance, the top 5 and the bottom 5 categories.
Answer: Ertxiem's answer is precisely what you want. Calculate a $\chi^2$ test with $1000-1=999$ degrees of freedom.
I personally am a big fan of simulating the null hypothesis a couple of times and plotting the results of such simulations, to get a feeling for the randomness that the null hypothesis would imply - and then comparing these plots to the actual data you have. In the present case, I would simulate distributing $n=5,000$ items into $k=1,000$ bins, tabulating the numbers of items in the bins and plot the top 5 and the bottom 5 bins, as Ertxiem proposes. You can do this, say, 20 times and arrange the resulting histograms in a $4\times 5$ matrix:

For instance, we see that the fullest bin typically has about 14-15 items in it. Now, you can insert your actual histogram at a random position in this matrix. Does it "stand out"? For instance because the largest bin contains not 14-15, but 20 items? If so, then it is sufficiently far away from the null distribution that you can safely say there is something there. You can do a fun little exercise with this, by showing your 19 null distribution and one observation plots to random colleagues and ask them to identify the "special" case. If people can do so consistently, then there is something there.
This will also work if your bin counts are "too uniform". We wouldn't expect all bins to contain only 3-7 items, so if your fuller bins are too empty (and your emptier bins too full), then this illustrates a different departure from uniformity.
(Note: I didn't come up with this kind of "visual significance test". See Buja et al., "Statistical Inference for Exploratory Data Analysis and Model Diagnostics" (2009, Philosophical Transactions: Mathematical, Physical and Engineering Sciences))
R code:
<code>n_sims <- 20
set.seed(1)
n_items <- 5000
n_bins <- 1000
y_max <- 20 # set through trial and error

opar <- par(mfrow=c(4,5),las=2,mai=c(.1,.5,.1,.1))
    for ( ii in 1:n_sims ) {
        sim <- factor(sample(1:n_bins,n_items,replace=TRUE),levels=1:n_bins)
        barplot(c(sort(table(sim),decreasing=TRUE)[1:5],
                NA,NA,
                rev(sort(table(sim),decreasing=FALSE)[1:5])),
            xaxt="n",lwd=2,col="gray",ylim=c(0,y_max))
        text(7.2,1,"...",cex=2,font=2)
    }
par(opar)
</code>

Statistical test for non uniform counts


Python keycodes to break out of a screen terminal session

Question: I've been looking online for a way to break out of a screen terminal session.  The keys to break out of a session are:

Ctrl+A then D
OR
Ctrl+A then Ctrl+D

I've got the Ctrl+A part:

child.send(b'\x01')

But cannot find the keycode for Ctrl+D
Answer: Just check the ASCII table. It is child.send(b'\x04') (EOT)
Comment: Well... That worked.  Thank you very much for the quick response.
Answer: CTRL/D  is b'\x04'. There are many references, of which this was the first I came across.
Comment: Thank you very much - Especially for the reference - Now I understand it.


Diebold Mariano test (in R)

Question: As I asked in my answer to this question: does anyone know if the DM test (in R in this case) is supposed to be made with h=h-1?
If not, am I supposed to make several prediction sets (with h observations each) and perform h-step predictions on them in order to have more degrees of freedom?
That is, am I supposed to make n h-step predictions for the DM test?
When I attempt to perform the test with h=18 (there are 18 obs.) it says that DM variance is 0 (it consumes the degrees of freedom?).

Error in dm.test(error.311arima, error.013arima,  :
    Variance of DM statistic is zero

The residuals are different for the time series predictions.

EDIT
I have "checked under the hood" of the dm.test function in R, and apparently what happens is that when it executes the following operation:
m.test
function (e1, e2, alternative = c("two.sided", "less", "greater"),
    h = 1, power = 2)
{
    alternative <- match.arg(alternative)
    d <- c(abs(e1))^power - c(abs(e2))^power
    d.cov <- acf(d, na.action = na.omit, lag.max = h - 1, type = "covariance",
        plot = FALSE)$acf[, , 1]
    d.var <- sum(c(d.cov[1], 2 * d.cov[-1]))/length(d)
    dv <- d.var
    if (dv > 0)
        STATISTIC <- mean(d, na.rm = TRUE)/sqrt(dv)
    else stop("Variance of DM statistic is zero")

the d.var part yields a negative number, i.e. the variance estimator is negative???
I presume it is due to a lack of covariance stationarity on the errors, but then what should I do? I can only think of manually differencing the series (or the error of the series) and then use those forecasts, which should be in turn, covariance stationary.
error 311arima  error 013arima
-0.8758334  -0.8623334
-0.571964484    -0.614464484
-1.5118295  -1.3041295
-2.034418946    -1.918818946
-1.422745494    -1.805045494
-1.777216545    -2.240316545
-1.671314551    -2.260914551
-2.61727251 -3.40957251
-4.29605616 -5.10285616
-4.87524124 -5.78274124
-6.29766468 -7.27736468
-7.458823814    -8.455323814
-0.983687537    -1.110287537
-0.49127652 -0.92037652
0.652971123 0.048671123
2.52594615  1.73144615
0.651429228 -0.409670772
3.416551133 2.252951133
3.302610752 2.023910752
0.929007023 -0.460192977
1.959733687 0.522733687
-0.710917583    -2.217517583
-2.09341202 -3.64921202
-2.320514415    -3.910114415

The answer to this question can be found in a later question here: Diebold-Mariano test for multiple prediction horizons
We cannot compare forecasts for VECTORS of differing (in this case increasing) points in time, we must use a rolling window approach with a fixed h to compare these two models' h-point forecasts along time
Answer: The answer to this question can be found in a later question here: Diebold-Mariano test for multiple prediction horizons
We cannot compare forecasts for VECTORS of differing (in this case increasing) points in time, we must use a rolling window approach with a fixed h to compare these two models' h-point forecasts along time

Diebold Mariano test (in R)


$\sum_{n=1}^\infty \frac{1}{n^p}$ converges for any $p>1$

Question: I read that the harmonic series $\displaystyle\sum_{n=1}^\infty\dfrac{1}{n}$ diverges but for any $p>1$ the series $\displaystyle\sum_{n=1}^\infty\dfrac{1}{n^p}$ converges. I found this quite difficult to comprehend and could not find a proof by myself. Is there a nice proof of this?
Comment: Use Cauchy Condensation test. (The answer ILoveMath posted)
Comment: yes, the integral test.
Comment: @WillJagy ... oh. I don't know what I was thinking.
Comment: By comparison test with $\sum_{n=1}^\infty\frac1{n^p}\ge\sum_{n=1}^\infty\frac1n\forall p\le1$, it must diverge for those values.  If it converges for $p=a$, it converges for all $p>a$ be comparison too.
Answer: Consider the series $\sum a_n $ where $(a_n)$ is decreasing and positive. Also, consider the series $\sum 2^n a_{2^n}$. Consider their partial sums $(s_n), (t_n)$
Note if $n < 2^n$, then $s_n = a_1 + a_2 + ... + a_n \leq a_1 + (a_2+a_3) + (a_4 + a_5 + a_6 + a_7) + ... + (a_{2^n} + ... + a_{2^{n+1}-1}) \leq a_1 + 2a_2 + ... + 2^n a_{2n} = t_n \implies \boxed{ s_n \leq t_n }$
If $2^n < n$, then $s_n  \geq a_1 + a_2 + (a_3 + a_4) + ... + (a_{2^{n-1}+1} + ... + a_{2^n} ) \geq \frac{1}{2} a_1 + a_2 + 2 a_4 + ... + 2^{n-1}a_n = \frac{1}{2} t_n \implies \boxed{ 2s_n \geq t_n }$
Since series of nonnegative terms converges iff its sequence of partial sums are bounded, then by the work above we obtain the following
$$ \boxed{ \sum a_n \; \; \; \mathbf{Converges \; \; iff} \; \; \; \sum 2^n a_{2^n} \; \; \; \mathbf{Converges} }$$
Thus, with $a_n = \frac{1}{n^p}$, then $a_{2^n} = \frac{1}{2^{np}}$, thus
$$ \sum 2^n \frac{1}{2^{np}} = \sum \left( \frac{1}{2} \right)^{n(p-1)} = \sum \left( \frac{1}{2^{p-1}} \right)^n$$
Clearly this series $\mathbf{converges}$, being a geometric series, iff $p-1>0 $ iff $p > 1$ cause otherwise it would a divergent geometric series.
Answer: One can use the integral test to ascertain the convergence of $\displaystyle\sum_{n=1}^\infty\dfrac{1}{n^p}$ where $p>1$. (link to integral test: https://en.wikipedia.org/wiki/Integral_test_for_convergence)
Let $p>1$ be a real number. Let $f:[1,\infty)\to \mathbb{R}$ be a function defined as $f(x):=1/x^p$. The function $f$ is non-negative, monotone decreasing and continuous. By the integral test for convergence, the series$$\displaystyle\sum_{n=1}^\infty f(n)$$ converges if and only if the integral $$\displaystyle\int_1^\infty f(x)\mathrm{dx}$$ converges.
We have $$\displaystyle\int_1^\infty f(x)\mathrm{dx}=\displaystyle\int_1^\infty\dfrac{1}{x^p}\mathrm{dx}=\displaystyle\lim_{a\to\infty}\int_1^a\dfrac{1}{x^p}\mathrm{dx}$$
But we have $$\displaystyle\lim_{a\to\infty}\int_1^a\dfrac{1}{x^p}\mathrm{dx}=\displaystyle\lim_{a\to\infty}\left.\dfrac{x^{1-p}}{1-p}\right|_1^a=\displaystyle\lim_{a\to\infty}\left(\dfrac{a^{1-p}}{1-p}-\dfrac{1}{1-p}\right)=\displaystyle\lim_{a\to\infty}\dfrac{a^{1-p}}{1-p}-\displaystyle\lim_{a\to\infty}\dfrac{1}{1-p}$$
Since $1-p<0$, we have $$\displaystyle\lim_{a\to\infty}\dfrac{a^{1-p}}{1-p}=0$$
Thus $$\displaystyle\int_1^\infty f(x)\mathrm{dx}=\dfrac{1}{p-1}$$, and the integral $\displaystyle\int_1^\infty f(x)\mathrm{dx}$ converges.
By the integral test the series $\displaystyle\sum_{n=1}^\infty\dfrac{1}{n^p}$ also converges.
Comment: And note that $n^{-s} - \int_n^{n+1} x^{-s}dx = \int_n^{n+1} \int_n^x s t^{-s-1}dt dx = \mathcal{O}(n^{-s-1})$ therefore $\sum_{n=1}^\infty n^{-s} - \int_n^{n+1} x^{-s}dx = \lim_{N \to \infty} (\sum_{n=1}^N n^{-s}) - \frac{1-(N+1)^{1-s}}{s-1}$ converges for $Re(s) > 0$


insert webform component values into a custom table

Question: i am having trouble storing webform component values into a custom database table,
i have created a custom module to insert the data into the table but its not working and i can't figure out where my problem is can someone please help
here's my code  thanks
<?php
function customform_form_alter(&$form, $form_state, $form_id) {
    if ($form_id == 'webform_client_form_24') {
        $form['#submit'][] = 'customform_additional_insert';
        if ($node = menu_get_object()) {
            $form['submitted']['node_id']['#value'] =$node->nid;
            $form['submitted']['amount']['#value'] = @$node->field_needed_amount['und'][0]['value'];
            $form['submitted']['full_name']['#value'] = @$node->title;
            $form['submitted']['image']['#value'] = @$node->field_student_photo['und'][0]['fid'];
        }
    }
}

function customform_additional_insert($form, &$form_state) {
    $data = $form['submitted'];

    // Insert a record into a 3rd-party module table when a submission is added.
    db_insert('sponsorship')
        ->fields(array(
            'n_id' => $form['submitted']['node_id']['#value'],
            'user_id' => 1,
        ))
        ->execute();
}
?>

Comment: I can't help but feel that this is duplication of data and should be avoided. Why do you need to store the data in a separate table? Can't whatever needs its take the data directly out of the webforms tables?
Comment: thanks for the response Felix. thats what i tried first to get it from the webform submission data table but i could not get it. can you help with that
Answer: I would suggest taking the data straight out of the webforms DB tables.
To understand how to do that we need to look at how data is stored in the DB.
Each time a new webform is created a row for it is created in the webform table. Each webform has a unique identifier called nid.
Each component added to the webform is stored in the webform_component table and has a unique identifier of cid (cid is only unique per webform, nid is also required to make the primary key).
Now when a user fills in the webform a row is created in the webform_submission table (with a unique id of sid) and each fields value is stored in webform_submitted_data. Each row in the webform_submitted_data table has a nid (the webform ID), sid (submission ID) and cid (the component ID).

So, if you wanted to get all of the data from a particular submission with the name of the field (or component), uid and the data submitted you can use the following query:
SELECT c.`name`, sd.`data`, s.uid
FROM webform_submitted_data sd
INNER JOIN webform_component c ON c.cid = sd.cid AND c.nid = sd.nid
INNER JOIN webform_submissions s ON s.sid = sd.sid
WHERE sd.sid = {YOUR SUBMISSION ID}

Or to show all of the values submitted for a particular field with uid and date then you could use:
SELECT sd.`data`, s.uid, s.submitted
FROM webform_submitted_data sd
INNER JOIN webform_component c ON c.cid = sd.cid AND c.nid = sd.nid
INNER JOIN webform_submissions s ON s.sid = sd.sid
WHERE sd.cid = {YOUR COMPONENT ID}

Answer: I know this is a bit old, but if anybody else is trying to do this check out webform_mysql_views. Doesn't create a table, but a database view can be really helpful if you're doing this for integration purposes.
http://drupal.org/project/webform_mysql_views


How can I create a field to hold data on users' countries and states?

Question: I'm trying to create a content type that has multiple countries available to be selected and then multiple states/territories within those countries based on the country, but I'm not sure how to best accomplish this.
When a user is creating a piece of content they need to be able to select their country, then based on the country they select, they can select their state or territory with only those states/territories for that country available as options. I have two such countries that will have multiple states/territories - the U.S. and Canada.
I'm creating a text field using the field module that lists all the U.S. states when a parent item is selected using a list (text) and a "select text" drop-down. I also need to create a way to present the user with the Canadian territories when selecting Canada as the territory.
Right now I have two fields - "Country" and "State". Can you recommend the best way to accomplish presenting the user with the correct options for the state/territory based on their country selection? I'm drawing a blank on this one.
Answer: In Drupal 7 you should use the dedicated addressfield module for that purpose.
The composite address field provides a country drop down box with other address fields such as state loaded via ajax depending on the selected country.
USA

Canada

It's a requirement of the Commerce module for address selection.
You can hide some of the address sub-fields if needed.
Comment: I'm working on this using the addressfield module you suggested and I'm curious how to hide the "address 1 & 2" fields when adding a node?
Answer: Probably two text fields with widget select, and use form #ajax to update the region list when the country is selected. See Dynamic select list in the form (dependent dropdown) for a similar answer.
Another option might be Hierarchical Select which does this "out of the box" - you could use a vocabulary for the country/region structure. Consideration here is that you'll need to look up the parent term of the region to find the country, and maybe you'd rather have an easy way to query "content from Canada" than "content from any region which has a parent term of Canada".


Cron Authentication Error

Question: I can not seem to find out what cron.php is doing exactly that it needs to have an ssl connection with Google.
I keep getting the following error:

ERROR: Authentication with Google failed. Reason: BadAuthentication

I've Googled it and basically come up with nothing that seems to be similar to my issue.
Is there some cron.php configuration parameters I should be setting somewhere?
Thanks in advance,
SOLVED
The "Search" module is the culprit. Here's how I found it.
I hacked my cron.php file to add a couple functions it was calling so I could add logging information to help me track down the module. I copied both the module_invoke_all() and drupal_cron_run() functions into my cron.php, renaming them dev_module_invoke_all() and dev_drupal_cron_run(). The dev_module_invoke_all() is the logging is added and is shown below. The change I made in my dev_drupal_cron_run() was simply to change the module_invoke_all('cron') call to dev_module_invoke_all('cron'), so I am not pasting it. Then the cron.php call to drupal_cron_run() is changed to dev_drupal_cron_run() to use my functions.
I could then look in the "Recent log entries" under the Report menu to see the last module cron_hook() called. For me, it was the "Search" module. I disabled it, re-ran cron, and it successfully completed.
This is with a Drupal 6 installation.
<code>function dev_module_invoke_all() {
  $args = func_get_args();
  $hook = $args[0];
  unset($args[0]);
  $return = array();
  foreach (module_implements($hook) as $module) {
    /*This logs a message out to the Recent Log Entries report.*/
    watchdog('cron', "Calling module: " . $module . " cron_hook: " . $module . '_' . $hook);
    $function = $module . '_' . $hook;
    $result = call_user_func_array($function, $args);
    if (isset($result) && is_array($result)) {
      $return = array_merge_recursive($return, $result);
    }
    else if (isset($result)) {
      $return[] = $result;
    }
  }
  return $return;
}
</code>
Even more info. I added watchdog logging to the search.module's search_cron() to determine which module it was calling and tracked the issue to the node module. I then added watchdog logging to the node.module _update_index function to log out which node was being indexed. I then re-ran the cron and found the offending nodes. There turned out to be a couple of issues: A PHP page trying to do openssl authentication which was my original error. That page had been deprecated, but was still in the node list. I deleted the page.
Dan
Comment: Google Analytics module?
Comment: That was my thought as well, so I disabled it. Getting the same result.
Answer: Check if you have a module that uses any of the Google's API (translate, maps, youtube, calendar,...).
Seems that one of the modules implements hook_cron() where is trying to access to one of the Google services through its API.
If that is the case at Google Cloud Platform you can configure a key to solve the issue.
If that is not the problem, try to check the logs to see if you can get more info about this error.
Comment: I'll go through them and look again, I got rid of the gmap and statistics module on the test install but still had the issue.
Comment: I may have found the culprit. Found in constant_contact's changelog: "17.) Installation improved, it now checks openssl and the fsockopen function are working correctly by trying to establish a connection with https://www.google.com/."
Comment: Looks like that hole goes nowhere, disabled, then moved module out of install and still get same error. Is there a way to get any debugging info from the cron to figure out at which point this is occurring?

Cron Authentication Error


Dimension of rows space and columns space of a matrix

Question: The rank of the below matrix confused me. Since it has $3$ independent rows, so the dimension of rows space should be $3$. However, it also has $7$ independent columns, then the dimension of columns space is $7$ also?
\begin{pmatrix}
1& -2& 0& 19& -6& 0& -37\\
0& 0& 1& -6& 2& 0& 6\\
0& 0& 0& 0& 0& 1& 3\\
0& 0& 0& 0& 0& 0& 0\\
\end{pmatrix}
Answer: It actually doesn't have $7$ independent columns (for example the second column is $-2$ times the first)
Answer: "Dimension of a matrix" makes no sense.  Dimension is a property of vector spaces, not of individual vectors or matrices.
Your matrix has three independent rows (and no more), so the dimension of its row space is $3$.
If a matrix has seven independent columns (in fact, yours doesn't) and no more, then the dimension of its column space will be $7$.
Answer: As mentioned David Sir, ""Dimension of a matrix" makes no sense"-it is true fact. I think it may be "Rank of a matrix".  However in detailed solution of the problem we see that only $1st$ $3rd$ and $6th$ columns are linearly independent, remaining all rows can be expressed as a linear combinations of these three columns. So $dim (\text{column space})=3$. Also $dim (\text{row space})=3$, since it has three independent rows also. Moreover for any $m\times n$ matrix $A$

$$dim (\text{row space of A})=dim (\text{column space of A})= rank(A).$$


how to get flash's url, height, width in a HTML web page?

Question: I am new to Web C++ programming. I am trying to use IE8's IWebBrowser2 interface to obtain a webpage's embedded flash's url, height, width information. This flash can be generated by javascript. So I did like this: from IWebBrowser2 I got IHTMLElementCollection,then I iterated through the collection to get IHTMLObjectElement2,and finally called get_data() to get flash's url.
But when I tried to extract flash object's url for "http://www.abc.com", get_data() returned "null". Why is this?
So what's the difference between IHTMLObjectElement2 and IHTMLElement2? What's the right way to obtain IHTMLElement2 from IWebBrower2?
Also, anybody knows other good forums to ask web programming questions?
Your help is highly appreciated!
Answer: You would grab these in Javascript. Place an ID on the element that the flash is contained in and place that ID in these:
document.getElementById('divID').offsetHeight
document.getElementById('divID').offsetWidth

Note that these are dynamic and if the flash applet changes, doesn't load, or breaks, the value can change.


Analytic functions whose all derivatives keep $\mathbb{Z}$ invariant

Question: Is there a real  analytic function $f:\mathbb{R} \to \mathbb{R}$ which is  not a polynomial function but all its derivatives $f^{(n)}$ satisfy $f^{(n)}(\mathbb{Z})\subseteq \mathbb{Z}$?
This  question was  included in a more  flexible question as follows but according to its answers  and comments, I realize that the smooth version of this question was obvious.
Smooth or analytic functions which keep $\mathbb{Z}$ invariant
Answer: Functions satisfying $f^{(n)}(\mathbb Z)\subseteq\mathbb Z$ are called utterly integer valued. There is a simple construction of an utterly integer valued 1-periodic entire function given in Sato, Daihachiro. Utterly integer valued entire functions. I. Pacific J. Math. 118 (1985), no. 2, 523--530. I will just give the idea.
Consider a series of the form
$$f(z)=\sum_{n=0}^\infty a_n\sin^n(2\pi z).$$
Formally at least, $f^{(n)}(0) = a_n n!(2\pi)^n + R(n)$ where $R(n)$ depends only on $a_0,\dots,a_{n-1}.$ Each $a_n$ can be chosen inductively such that $f^{(n)}(0)$ is the closest non-zero integer to $R(n),$ giving $|a_n|\leq 1/n!(2\pi)^n.$ This rate of growth guarantees that the series defines an entire function, which by construction is utterly integer valued.
Comment: Thank you very much for your very perfect answer.


disable/understand java applet class cache behavior

Question: I have a java applet that I am transitioning from distributing unsigned to signed.  The signed version seemed to start up slower, and in investigating this, the slowdown is in a series of connections to my web server that are fast when unsigned, but slow when the applet is signed.  I've struggled to get traction on this, as I can't figure out why it's making the connections in the first place, much less how to disable the behavior.  I'm also unclear on what creates the classes, and searches for java things involving "Customizer" have been unproductive.
Here's the relevant output from the java console (with a little obvious censorship):

basic: Applet loaded.
  basic: Applet resized and added to parent container
  basic: PERF: AppletExecutionRunnable - applet.init() BEGIN ; jvmLaunch dt 279063 us, pluginInit dt 1299510 us, TotalTime: 1578573 us
  network: Cache entry not found [url: MYHOSTURL/games/java/lang/ObjectCustomizer.class, version: null]
  network: Connecting MYHOSTURL/games/java/lang/ObjectCustomizer.class with proxy=DIRECT
  network: Connecting MYHOSTURL:80/ with proxy=DIRECT
  network: Connecting MYHOSTURL/games/java/lang/ObjectCustomizer.class with cookie "STUFF"
  network: Cache entry found [url: MYHOSTURL/games/UnitValueCustomizer.class, version: null] prevalidated=false/0
  cache: Adding MemoryCache entry: MYHOSTURL/games/UnitValueCustomizer.class
  cache: Resource MYHOSTURL/games/UnitValueCustomizer.class has expired.
  network: Connecting MYHOSTURL/games/UnitValueCustomizer.class with proxy=DIRECT
  network: Connecting MYHOSTURL:80/ with proxy=DIRECT
  network: Connecting MYHOSTURL/games/UnitValueCustomizer.class with cookie "STUFF"
  network: Connecting MYHOSTURL/games/UnitValueCustomizer.class with proxy=DIRECT
  network: Connecting MYHOSTURL:80/ with proxy=DIRECT
  network: Connecting MYHOSTURL/games/UnitValueCustomizer.class with cookie "STUFF"
  cache: Resource MYHOSTURL/games/UnitValueCustomizer.class has expired.
  network: Connecting MYHOSTURL/games/UnitValueCustomizer.class with proxy=DIRECT
  network: Connecting MYHOSTURL:80/ with proxy=DIRECT
  network: Connecting MYHOSTURL/games/UnitValueCustomizer.class with cookie "STUFF"
  network: Connecting MYHOSTURL/games/UnitValueCustomizer.class with proxy=DIRECT
  network: Connecting MYHOSTURL:80/ with proxy=DIRECT
  network: Connecting MYHOSTURL/games/UnitValueCustomizer.class with cookie "STUFF"
  network: Cache entry not found [url: MYHOSTURL/games/java/lang/ObjectCustomizer.class, version: null]
  network: Connecting MYHOSTURL/games/java/lang/ObjectCustomizer.class with proxy=DIRECT
  network: Connecting MYHOSTURL:80/ with proxy=DIRECT
  network: Connecting MYHOSTURL/games/java/lang/ObjectCustomizer.class with cookie "STUFF"
  network: Cache entry not found [url: MYHOSTURL/games/BoundSizeCustomizer.class, version: null]
  network: Connecting MYHOSTURL/games/BoundSizeCustomizer.class with proxy=DIRECT
  network: Connecting MYHOSTURL:80/ with proxy=DIRECT
  network: Connecting MYHOSTURL/games/BoundSizeCustomizer.class with cookie "STUFF"
  network: Cache entry not found [url: MYHOSTURL/games/BoundSizeCustomizer.class, version: null]
  network: Connecting MYHOSTURL/games/BoundSizeCustomizer.class with proxy=DIRECT
  network: Connecting MYHOSTURL:80/ with proxy=DIRECT
  network: Connecting MYHOSTURL/games/BoundSizeCustomizer.class with cookie "STUFF"
  network: Connecting MYHOSTURL:3000/ with proxy=DIRECT
  My program's first output

(Note: the UnitValueCustomizer cache entry found is a junk file I put on the web server just to see what would happen - which was that everything operated normally despite downloading garbage instead of a class.)
The listed classes (nor the word "Customizer") do not appear in the java source for the application, nor in the .class or .jar files generated in compilation.  The class names without the "Customizer" are from Miglayout.
So any hints on how to get rid of this by whatever means are welcome, including but not limited to:

How to tell the VM not to look for anything outside the provided applet .jar
What language feature produces these phantom classes & how to avoid that
Any information on this class loading mechanism or what it's called
Answer:
How to tell the VM not to look for anything outside the provided applet .jar

This requires the codebase_lookup parameter as mentioned in Special Attributes of Applets.
E.G.
<APPLET ...>
<PARAM name="codebase_lookup" value="false">
</APPLET>

Questions 2. and 3. can be best answered as, it is a feature of the JVM that can be configured otherwise by the codebase_lookup parameter.  The class-loaders that load either way are the exact same URLClassLoader, but just configured with a different class-path.

codebase_lookup: false includes only the Jars defined in the archive attribute.
codebase_lookup: true (default if unspecified) includes not only the Jars defined in the archive attribute, but also the applet codebase itself - to search for resources that are not in a Jar.


ionic2-calendar Error: Module not Found: Angular is not exported from package Swiper

Question: I updated the package of my project

Ionic + angular to generate app for IOS and Android
Ionic: 7
Angular: 16

Since this recommendation

ion-slides was deprecated in v6.0.0 and removed in v7.0.0. We
recommend using the Swiper.js library directly.

I migrated to the last version of swiper(version= 9.3.2)
But after update package.json and run ionic build I got this issue
./node_modules/ionic2-calendar/fesm2020/ionic2-calendar.mjs:8:0-37 - Error: Module not
found: Error: Package path ./angular is not exported from package
/Users/admin/Projects/<name_project>/node_modules/swiper (see exports field in
/Users/admin/Projects/<name_project>/node_modules/swiper/package.json)

I reviewed node_modules/swiper and I did't find a path to angular
And checking the code of ionic2-calendar  I see that they try import swiper/angular
import * as i3 from 'swiper/angular';
import { SwiperModule } from 'swiper/angular'

Checking this answer  it looks like Swiper Version 9 doesn't have support to Angular and we need to migrate to the new way to use swiper in angular. I did that in my code but in the third part code like ionic2-calendar  I didn't do that
What is the recommendation in this case? Should I downgraded the version 9 and use the version 8?
Answer: I downgraded Swiper from version 9 to Version 8.4.7 that works for me
The last version of ionic2-calendar was updated 4 month ago almost the same Date of the update of Swiper to version 9, so probably to use  the last version of Swiper we nee to wait for a new update ionic2-calendar


Alternative proof of isomorphism between linearly ordered sets

Question: The following theorem theorem is proved in Hrbacek and Jech's Introduction to Set Theory:

Theorem: Let $(A, \prec)$ be a nonempty linearly ordered set with the properties:

(a) For every $p \in A$, there is $q \in A$ such that $q \succ p$.
(b) Every nonempty subset of a has a least element (in the order    $\prec$).
(c) Every nonempty subset of $A$ that has an upper bound    has a greatest element (in the order $\prec$).

Then $(A, \prec)$ is isomorphic to $(\mathbb{N}, <)$.

We need to show there is a biyection from $\mathbb{N}$ to $A$ that preserves order. My questions (below), specifically, are about the part where they prove their function is surjective. So i'm gonna sketch the first part of the proof and then will reproduce verbatim the argument I am referring to:
Sketch of proof: A sequence $f: \mathbb{N} \to A$ is constructed recusrively as follows:

(i) $f_0 = a =$ the least element of $A$ (it exists by (b))
(ii) $f_{n+1} =$ the least element of $A$ greater than $f_n$. (it exists by (a) and (b))

Clearly $f_n \prec f_{n+1}$ for every $n \in \mathbb{N}$. Then $f_m \prec f_n$ whenever $m < n$ (by induction) and $f$ is one to one.
We now prove the range of $f$ is $A$. Their argument is this:

If not, $A - \operatorname{ran}{f} \neq \emptyset$; let $p$ be the
least element of $A - \operatorname{ran}{f}$. The set $B = \{q \in A: q \prec p \}$
has an upper bound $p$, and is nonempty (otherwise, $p$
would be the least element of $A$, but then $p = f_0$). Let $q$ be the
greatest element of $B$ (it exists by (c)). Since $q \prec p$, we have
$q = f_m$ for some $m \in \mathbb{N}$. However, it is now easily seen
that $p$ is the least element of $A$ greater than $q$. Therefore, $p = f_{m+1}$
by the recursive condition (ii). Consequently, $p \in \operatorname{ran}{f}$, a contradiction.

Question 1: is the boldfaced line the only part where they use the fact $p$ is the least element of $A - \operatorname{ran}{f}$? It seems to me it is.
Question 2: Is the following alternative "proof" of this fact correct?
Proof: Take $p \in A$ (notice I do not assume $p$ is a least element) and define $B = \{f_n \in A: f_n \prec p\}$. If $p = a = f_0$ then we are done, so suppose $a = f_0 \prec p$; hence $B \neq \emptyset$ since $f_0 \in B$. Also, $B$ has $p$ as an upper bound so it has a greatest element $f_n$.

If $f_{n+1} \prec p$, then $f_{n+1} \in B$ and $f_n \prec f_{n+1}$, which is impossible since $f_n$ is the greatest element of $B$.
If $p \prec f_{n+1}$, then $f_n \prec p \prec f_{n+1}$ and $f_{n+1}$ is not the least element of $A$ greater than $f_n$.

We conclude then that $f_{n+1} = p$ (we are using the fact $\prec$ is a linear order here) and $p \in \operatorname{ran}{f}$.
Comment: For the first question, we need $p=\min(A\setminus\operatorname{ran}f)$ to ensure $p$ is the least element greater than $q$. I think no other part of the proof uses this fact any further.
For the second question, your proof looks correct.
Answer: Since $q \prec p=\min(A\setminus \operatorname{ran}(f))$ it implies that $q \in \operatorname{ran}(f)$ or else the minimum would be $\prec$ than itself.
(or quite formally, if $q \prec p$ then $q$ cannot be in $A \setminus \operatorname{ran}(f)$, so $q \in \operatorname{ran}(f)$ and $q=f_n$ for some $n$ etc.).
Why do you need an alternative argument for that? It only complicates matters.
And if $p = \max B \prec q$ we also see that there can be no $r \in A$ with $p \prec r \prec q$, as $p$ is already the max of $B$ and $r$ would have been in $B$ too (being $\prec q$). So that exactly tells us that $q = \min\{r \in A \mid r \succ p\}$ (here we use that the order is linear) which equals by construction $f_{n+1}$ if $p=f_n$. So $q \in \operatorname{ran}(f)$ after all, a contradiction.


Unusual transformation of percentage data

Question: I am reviewing some support vector machine (SVM) work by someone who has left my company. The target data is a heavily left skewed set of percentage values. He chose to make an unusual transformation that I don't understand - and can't find anywhere else.
transformed_x = log10(max(x) - x +1)

This appears to be a more successful way to make the data normally distributed than more common transformations such as logit and arcsin - but I don't really understand why? Only power transformations determined from Tukey/Box-Cox (x^19.3 !!) seem to do better.
As a side question - is it only the target data in SVM that should be normally distributed, or should I also address the input variables too? He has only done the target variable.
Comment: This transformation, when parsed into its components, is not terribly unusual: the direction of the data was flipped and then a log transformation (with a start value of 1) was applied: this is a simple generalization of a Box-Cox transformation.  A power of $19.3$ certainly is ridiculous and is exceptionally unlikely to be of any use--but no reasonable Box-Cox analysis would produce such a power in the first place.  The motivation for the original transformation could lie in what kind of quantity `x` represents.  What does it measure?  See https://stats.stackexchange.com/questions/4831.
Comment: When you have percentages of a whole, it can be reasonable to re-express them in terms of the complementary percentages: that's close to what subtracting from the maximum does. The choice of the maximum (rather than 100%) as well as adding 1 is *ad hoc* but not unusual: the proof of the pudding lies in whether it is *effective* and *generalizable*. For more about the art of finding an appropriate Box-Cox transformation see https://stats.stackexchange.com/search?q=+box+cox+ladder; also see https://stats.stackexchange.com/questions/6150; https://stats.stackexchange.com/questions/30728.
Comment: @whuber - sorry answered before the edit. For your question: I have a dataset that is a series of (mineral) samples. For each sample is a range of measured parameters that can be values that are always > 0 (e.g. residue at some screen size). They can be percentage values, such as proportion of mineral that is some oxide like SiO2. Or other things. Then the target is the brightness (% reflectance at some wavelength) when the mineral is milled a certain way. He was able to reasonably predict the brightness and I'm just making sure I understand what and why he did.
Comment: @whuber. Thanks again for your help - it's all clear to me now. And depressingly obvious once you and explained it. Should I provide my own answer to my question - which might help others with a similar misunderstanding - or just leave it?
Comment: what is min(x) and max(x)?
Comment: The max is 91.16 and the min is 68.03.
Comment: i'd guess that your predecessor simply picked the function that fit the data well. the fact that you have $(-x)$ makes this look like lognormal distribution but flipped around vertical axis
Comment: @whuber thanks, I see what you're saying about his transformation. It still seems funny to me that it performs noticeably better than a straight log or logit transform? I just did a Box-Cox sweep in R with increasing lambda until I found a peak. It's my first go with SVM or with Box-Cox, so appreciate I might be doing something unreasonable. I'm never comfortable with a mixture of percentage and non-percentage values in PCA either. But I've only had a relatively quick play with that too (I'm no statistician - just a scientist given the usual basics and a desire to learn a bit more).
Comment: Thanks so much for all your tips and links. Will read them with great interest. I did have a search before posting the question, but didn't find anything (I thought) relevant.
Comment: I think an answer would be welcome.
Answer: Thanks to @whuber and @Aksakal, I can answer my own question. It's actually fairly simple and I was over-complicating things because the variable of interest was a percentage and this threw me off into too complicated topics -- I hope this answer will assist others who have come across similar problems, and I learnt some more general things I'll mention too.
First I'll explain a simple case without percentage data.
Let's say we have some measurement that always gives a small number, but never goes below zero. Such as sieve residue when screening minerals at a particular size. Clearly this type of measurement will often give a right skewed distribution of data -- much like a log-normal distribution. Obviously then, it might be sensible to take a log transform to try to convert the data into a normal distribution. In fact, I've done this very thing in the past -- so it's a bit embarrassing I didn't get this solution myself anyway. For example, check slide 9 of this presentation.
In this screening scenario, we are talking about a potentially real log-normal distribution -- because there's no upper bound on the value. But in the original question I was dealing with (left-skewed) percentage data, and tied myself up in knots thinking that only something like a logit transform should be appropriate -- because the data must be bounded at both ends -- but, as noted in my original question, this is not what the original author of the SVM work did (and it doesn't work well).
When I look at the data, from an empirical point of view, and don't worry about the hard and fast principle of the data being bounded, my data won't ever get too close to either 0 % or 100 %, so I don't need to worry about it and can do something else. Which is what the original author obviously realised.
The first thing to do is to flip the data to convert it to right-skewed -- to something that has a chance of being near to a log-normal distribution (and hence can be transformed to a normal distribution by a simple log transform).
This is the part max(x) - x in my original question, and converts

to

The + 1 is then there to avoid the error caused by having a zero value by doing this simple flip (i.e. when max(x) is the x).
Thinking about this, after understanding the flip, I wondered why the original author did max(x) - x + 1, and not 100 - x, given that we know physically the maximum theoretical value is 100 %? What I've realised (I think), is that you get the best possible log transformation -- i.e. the most normal data -- when the offset you apply gives your minimum datum as exactly 1. Looking at a Q-Q plot such as

If the offset is less than 1, then the data at the bottom end of the Q-Q plot tend to stray further from normality (log10 varies quickly as x goes below 1). As the offset becomes > 1, then a similar thing happens, but at the upper end of the Q-Q plot.
So -- maybe this is particular to my case, feel free to comment/edit if that's the case -- an offset of 1 is the optimum value to add to get the most normal data after the log10 transformation. Hence, max(x) - x + 1 is better than 100 - x, unless max(x) = 99.
As long as none of my data gets too close to the upper bound of 100 %, then a simple flip will give something that can be log transformed. I would imagine if the data was right-skewed (and not too near zero) then the flipping would be unnecessary and the log transform would be suitable straight away.
It appears the original author of the SVM work has taken a very empirical look at the data, realised all this way quicker than I did, and gone for the simplest option. One that may have some physical validity
This is another thing I've learned thanks to @whuber's and @Aksakal's comments. Don't obsess about finding the best transformation -- either formally (because you know your data is bounded in some way the transform implies it isn't, as long as you are far enough from the bounds, it's fine) -- or numerically.
The latter point is relevant because a Tukey power ladder (or Box-Cox) on the data implies the best transform is a power transform of over 19! Similarly, if I do a ladder for 100 - x (just for fun) it gives -2 as the best option, but this doesn't seem particularly physical. If I do it for max(x) - x + 1 then, sure enough, it proposes a log transform (lambda = 0).
So, now I understand why the original author did what he did, and, thanks to @whuber and @Aksakal, have also learnt:

Don't obsess about what's formally/mathematically correct, if you data can be represented more simply.
Keep in mind what is physically sensible. You want to balance what seems physically sensible, with what gives you the best results.
Be practical/empirical and if something simple, that is easy to understand/work with, gives you good enough results -- stick with that.

Edit: typos/clarity.


Excluir valores repetidos de uma variável

Question: Vou apenas dar um exemplo mais simples para não ter que postar o código todo, não sei se isso e possível fazer.
$exemplo = "exemplo exemplo"

echo $exemplo;

Ao exibir essa variável com echo ele retorna "exemplo exemplo" , teria algum comando ou alguma coisa q possa usar para ele retornar apenas "exemplo" sem repetir a palavra?
Obs: nao posso alterar o valor da variavel nem por ,  pois o modo que uso para capturar ela me retorna exatamente como esta acima.
Comment: Os valores repetidos são sempre separados por "espaço"? No caso essa variável é um string normal certo? Você poderia transformar essa variável em um array, e no array retirar os iguais.
Answer: Você pode transformar em um array usando o implode() e depois utilizar o array_unique() para remover todas palavras duplicadas, e depois voltar a string normal utilizando implode().
Exemplo:
$exemplo = "exemplo exemplo exemplo exemplo";
$exemplo = implode(" ", array_unique(explode(" ", $exemplo)));
echo $exemplo; // Saida  = "exemplo"

Comment: @MatheusRocha você vem com problema A e recebe resposta para problema A, quando você aceita a resposta não pode reeditar sua pergunta colocando outros problemas, além da pergunta ficar sem foco perde a qualidade.
Comment: so para ver se entendi, vo usa explode para transformar em um array depois usa array unique e depois volta a uma string usando implode
Comment: @MatheusRocha exatamente.
Comment: estou capturando a variavel utilizando um script( postei ele na publicação) e acho que isso esta influenciando , pois o codigo n esta funcionando no meu programa.