Which are outliers?

Question: I am in the process of solving a Machine Learning challenge, and I want to do it the right way.
I did some exploratory data analysisand I wanted to check the distribution of the data.
As displayed in the figure below, the box plots of each of the numerical continuous dataset features.
According to my understanding, several features do contain outliers, but after reading some articles, I encountered cases where outliers are mistaken for a Pareto distribution, and removing them leads to information loss.

Could you please explain which contain and which do not contain outliers?
Comment: Not my problem, but expecting a good data analysis without a context is a bizarre exercise.  This perhaps exposes a cultural difference between (some kinds of) machine learning and (what I regard as good practice in) statistics.
Comment: Ah ok. It helps if you tell us what type of data you think this is (stocks? weather? biology?) etc., to develop some intution about correlations or dependencies. The boxplots for X27, X31, X33 don't show much, better to histogram the few discrete values (/intervals).
Comment: @RockPaperLz-MaskitorCasket  Leo Breiman's paper on two cultures is easy to find and 21 years young. Others may be able to give references more recent and as good or better.
Comment: The starkest contrasts are between one approach done badly and the other done well (and so essentially unfair). As someone more on the statistical side than in machine learning, I am sometimes gobsmacked by some mentions on CV of machine learning, especially if someone has picked up a sense that in essence data PLUS code $\implies$ results, without any need to know context, think about goals or explore data. But mainstream statistics thoughtlessly applied can lead to nonsense too, as in applying techniques utterly unsuited to the data or ridiculous over-emphasis on significance tests.
Comment: [Kaggle Corporate Bankruptcy Prediction 2021](https://www.kaggle.com/c/1056lab-corporate-bankruptcy-prediction-2021/data); Bankruptcy data from the Taiwan Economic Journal for the years 1999–2009, yes?
Comment: Negative, the dataset was given to us without mentioning the source. However, it is not the one you shared. I named the features manually, and it contains 34 features, labels included.
Comment: "I encountered cases where outliers are mistaken for a Pareto distribution" Do you mean "I encountered cases where a Pareto distribution is mistaken for outliers"? Also, "outlier" and "data point that should be removed" are very different.
Comment: @NickCox A discussion of differences between machine learning and statistics would make a fascinating read.
Comment: @NickCox Thank you for that reference.  I'm open to additional reading on the topic too.
Answer: Outliers, like beauty and much else, are often in the eye of the beholder.
Identifying outliers is often treated as if it were a matter of knowing that your data are a mix of gazelles (regular beasts) and giraffes (probably large, awkward and unwanted), so the problem is to tell which is which.  In real applications, it may not be obvious that there are two  such distinct subsets, even in abstract principle. And even if the principle is accepted, it can be hard in practice to tell them apart: imagine how you would proceed  if the variables were height, weight, and number of legs and there are known to be young giraffes in the data.
Box plots are routinely misused in this context. First of all, although box plots were (re-)introduced by John W. Tukey in the 1970s, he himself played with several different conventions, and later people haven't stopped inventing different variations -- which could be worthwhile experiment. Unfortunately users often fail to explain in reports which detailed rules their software uses -- which is not helpful.
Here I guess that the convention being followed is one Tukey introduced: plot data points individually whenever any is more than 1.5 IQR from the nearer quartile. Here IQR is interquartile range or upper quartile MINUS lower quartile. Tukey himself regarded this just as a way of identifying data points that need to be thought about, and not at all as a reliable criterion for identifying points that should be rejected. Indeed, sometimes outliers (and skewness) are a signal that data should be analysed on a transformed scale and/or that they need resistant or robust summaries. .
I won't try to say which of your data points are outliers, because

The whole idea that some points are expected to be BAD outliers or problematic, and the rest GOOD, needs to be argued in each case. For example, in banking some transactions are rogue and most are honest, so there is strong concern to identify rogues, which is rarely easy. In many other fields, strongly skewed distributions are entirely expected and the larger outliers are known extremes (the Amazon, or Amazon, say).

There are no outliers except in relation to an implicit or explicit model of what is expected. If one distribution is expected to be (roughly) normal and another (roughly) Poisson or anything else, I have quite different expectations of whether outliers are likely and how to detect them. (This is the kind of point made in the question with the example of a Pareto distribution.) Here the data are just labelled X without even a signal for whether they are versions of the same (kind of) variable or whether they are quite different.

Box plots alone are inadequate to make the decision even if #1 and #2 are satisfied.  I'd much prefer dot or quantile plots. I'd also want to know the number of data points, which is impossible to discern precisely. But by failing to show simple patterns box plots can signal that you need another and a deeper look. Some of the box plots above are degenerate in that median and quartiles are identical, or very close, so that the box collapses to a line: in that case every other data point is plotted individually. (If the IQR is zero, every point beyond the quartiles must be plotted individually!)  Such a pattern could be anything from the quirks of small samples to some special situation: the box plots alone cannot do more than flag an oddity. Specifically X27 X31 X33 spring out as demanding careful scrutiny.

Specifically the numerical labels for these data suggest that the data were standardized first (likely by (value MINUS mean) / SD): this pre-processing is over-valued and inhibits or prohibits reading off basic characteristics of the data, such as

the data being all positive

the data being all positive or zero

the data being positive, zero or negative

whether only certain values occur (e.g. discrete grades or counts that are integers). This is one of the worst problems, as box plots are often hard work with such data. Repeated values plot as one symbol or contribute to compressed boxes. X33 seems possibly an example as there are perhaps only 3 distinct values; they may even be nominal codes with no intrinsic meaning.
Comment: +1.  Marvelous discussion.  I have concerns about (2), though, because I think it is the opposite of what Tukey maintained.  Many of his methods automatically identify "outliers" (and even "far outliers").  The difference is that outlier identification becomes a tool to help analyze the data, perhaps by suggesting ways to re-express or smooth them.  Key elements of the underlying conception are (1) that "outliers" often occur, (2) they do not necessarily reflect "rogue" or "bad" data, and (3) are treated as *adding useful detail to the gross descriptions afforded by summary statistics.*
Comment: @whuber I don't disagree and I've modified or added wording to give a more nuanced summary. I am particularly disturbed at a meme I find on CV: points more than 1.5 IQR from quartiles should just be deleted!
Comment: Or it could be a genuine but unusual value. I find it hard to tell you what's right there. For example, I would look at the values of other variables for the same observation to see whether there was a strong impression of something very unusual going on. Again, the context that's missing here is important. If the data are people filling in questionnaires that's one thing. If the data are produced by an automatic weather station, that's another.
Comment: Thank you Nick for the outstanding explanation. However, I'd like to ask one more question, the data contains 400 samples, 100 of which is not labeled, so the task is to predict the labels of the remaining 100. The data consists, of both num. continuous values and cat. nominal values. The latter will be encoded. My question is the following, concerning the nominal data, if you encounter an attribute which consists of two or three values (e.g. V1, V2, V3) but among all instances, it only contains one instance of V3, would you replace this with the mode? since it could be a data entry mistake.
Comment: As a matter of curiosity only, which software was used for the box plots?
Comment: Alright, I will try to experiment and see what happens in each case.
I am using seaborn package in python for the box plots.
Answer: You have two questions
What are outliers?

Which are outliers?

Should outliers be removed?

According to my understanding, several features do contain outliers, but after reading some articles, I encountered cases where outliers are mistaken for a Pareto distribution, and removing them leads to information loss.

There can be different interpretations of 'outliers'

Outliers as extreme cases within a population: Outliers can be referred to the cases within a population in the outer regions. For instance the book 'Outliers' from Malcolm Gladwell speaks about outliers as extreme cases, cases that are relatively far from the mean.

Outliers as measurement errors: In statistics and experimental science/research, outliers can also refer to erroneous measurements. Data points that are not representing the population of interest because of some error in the sampling method or experimental execution.

The latter type of outliers, experimental errors, you might want to remove.
But the former type of outlier, extreme cases among the population of measurements, are not always the same as these latter types.
Wrong measurements/sampling
All measurements have some error* due to small variations in the population and experiments. But some measurements introduce error to a much larger extent and are a bad representation of the population that is being sampled or used for experimentation.
It is these types of measurement or sample errors that we may want to remove from the data because they cause bias and/or high variance in estimates of population parameters.
*It might be better to speak about 'disturbance' rather than 'error'. The variation of measurements is an error in a context of a physics or chemistry experiment, where some fixed value is being measured. But, in psychology, nutrition, health, econometrics, the variations are natural since there is no fixed value and a varied population is being investigated. We do not want to filter out all the errors if these errors are what is representing the population. Discarding data in preprocessing is about the wrong measurements.

For example, imagine some experiment that determines and compares the growth rate of two different strains of bacteria.

Some of the experiments might have a bias.
e.g. the temperature was not set correctly causing a consistent/systematic error.

Some of the experiments might have a large variance
e.g. some batches of bacteria were not kept under optimal conditions and have a relatively large variance in the measurements.

Outliers as a way to detect erroneous measurements
Detecting these cases of error in measurements and sampling is part of preprocessing the data and data cleansing.
One way to detect the erroneous measurements is by finding out the outliers in the first sense, finding extreme cases in the population of measurements.

Outliers are extreme cases in a population.
When outliers in a population of measurements can be considered as wrongful measurements/sampling, i.e. they are a part of the measurements/sampling that is not representing the population of interest, then they could be removed.

Should they be removed?
The choice to remove the outliers depends on whether they are likely wrong measurements/sampling and whether they are likely to influence the statistical analysis.

Outliers in the population of measurements may not necessarily be equal to wrong measurements. If you are removing outliers too aggressively, then you might also remove data points that are not wrong measurements/sampling. The result is that the amount of data is decreasing. And also you can potentially introduce bias by having a selective/subjective sampling process that wrongly represents the population that is under investigation because there is too much censoring of extreme data points. Removing outliers in the measurements should be a balance between advantages and disadvantages.
Besides removing outliers, you can also perform an analysis that is more robust towards outliers. For instance, you can consider the wrong measurements as 'correct' measurements (since all measurements have some 'error') but just with a different type of error. One alternative way to deal with it is, for instance, using a different cost function in regression (e.g. use absolute deviations instead of squared deviations).

In your example, it is difficult to say because there is no information about the data (what does it represent?). But we see so many 'outliers' that we may wonder whether they are not just naturally part of the population. They are outliers in the first sense (extreme data points) but that does not necessarily make them wrong measurements.
Comment: +1 Thank you for taking the time to answer my question. You made it clear that the context of data plays an essential role to decide whether to handle extreme data points as anomalous or not.
Answer: An example to illustrate the points made well above.
On most days, the Dow Jones Index increases slightly. About once a decade it drops by more than 20%. An investor who ignores the "outliers" will not do well.
Comment: Not an answer to the question, but rather a comment on Nick's answer?
Comment: The point to be made here is, roughly, "Do outliers represent information of interest, or are they just the result of insufficient sampling?"

Which are outliers?


Name of this type of equation and solutions to solve it iteratively?

Question: I have the equation of the form:
$$
\textbf{X}_{n+1} = \textbf{X}_{n} + \textbf{X}_{0}
$$
This equation come from physical optics when trying to solve the MFIE (below) iteratively.
$$
\textbf{J}(\textbf{r}) = 2\vec{n} \times \textbf{H}^{inc}(\textbf{r}) + 2\vec{n} \times \textbf{H}(\textbf{r})
$$
Where $\vec{X}_{0}$ is my incident plane wave (This remains a constant vector) and the $\vec{X}_{n}$ is the current which has been calcuated. The sum of these gives the next iteration of the current, $\vec{X}_{n+1}$. This was just some background to help get a better picture.
My questions are:

What type of equation is this?
What methods are there to solve an equation like this iteratively? Some of the methods I have looked at are only valid for linear equations of this type, $Ax = b$.
Comment: I would look at methods that solve matrix equations for system of difference equations, namely the implicit numerical schemes. As you can imagine alot of time dependent equations require solve equations of the form above. Set your initial matrix i.e $n=0$ and then solve as if the r.h.s was constant.
Comment: Do you think that time matters here ?
Answer: This ''vectorial affine recurrence relationship'' (such is ''a'' name that can be given):
$$\tag{1}V_{n+1}=AV_{n}+b$$
has a standard method of resolution.
First of all, look at the ''candidate vector'' say $V_{\infty}$ that will be the limit if any. $V_{\infty}$ should verify the fixed point equation:
$$\tag{2}V_{\infty} \ = \ A V_{\infty}+b$$
(and $V_{\infty}$ will be found as a solution to the equivalent linear system $(I_n-A)V_{\infty}=b$).
Now, make the subtraction (1)-(2):
$$\tag{3}V_{n+1}-V_{\infty} \ = \ A(V_{n+1}-V_{\infty})$$
where $b$ has disappeared; (3) can be written so, with an auxiliary sequence:
$$\tag{4}W_{n+1} \ = \ A W_{n+1} \ \ \text{with} \ \ W_{n}:=V_{n}-V_{\infty}$$
We are now faced with a classical problem, with a discussion on the eigenvalues of $A$:
Let us call ''spectral radius'' (and denote by $r$) the maximum of the absolute values of the eigenvalues of $A$:

if $r<1$, $W_n \to 0$, then $V_n \to V_{\infty}$.
if $r>1$ $W_n$ is divergent, then $V_n$ is divergent.
if $r=1$, different cases may arise but in general there is no convergence either

(for example if there is an eigenvalue equal to -1, one can observe permanent oscillations).
Comment: Have you just converted it to a normal system of linear equations in (4)?
Comment: I don't understand what you mean for (4) which is not truly speaking system of linear equations. Could you explain a little more ?
Comment: Maybe, by giving some more context, I could understand better your needs. By the way, what is the dimension of the ambient space $\mathbb{R}^n$ n=2,3 ....?


Best set of variables in linear regression when the response is repeated

Question: I have monthly data about customer care in a bank branch. The variables are presented as follows:
Global evaluation: expressed in percentages.
X1, X2, ... X14 : Yes or no questions taken from a survey passed to the clients.
The data have the following structure:
For each month there is a unique global evaluation, but there is approximately 100 observations in the variables X1, X2, ..., X14.
I need to know what is the best set of variables X1, X2, ... , X14 that explain the response variable. ¿ Is it correct to apply linear regression to this problem?
Note: The global evaluation is assigned independently of the values of the variables X1, X2, ..., X14. Finally, there is only 7 months.
Answer: I'm not sure what the best way is but you could apply a linear kernel in each dimension (so the overall kernel is simply the product if these) with Gaussian noise. You then actually have a Gaussian over the gradients in each dimension and can determine the set of $k$ features which are most likely to contain the largest gradients - ie their values influence the outcome the most. It's not totally indifferent to automatic relevance detection for linear kernels (I am only mentioning linear kernels because that's the type of model you mentioned in the question).
This is just how I would naturally go about the problem but not necessarily the best way.


Saving plots generated from list using names in list using lapply

Question: I have a list of data tables, each of which contains data in the same format and each of which has a unique name. Example below:
> head(list$table1)
        gene     logFC negLogPval
    1   CROT -1.546082   6.405688
    2 CASC15 -1.718302   5.501062
    3  ITGA8 -2.839048   5.159019
    4   LDB2 -1.258781   4.646456
    5  PTGS1  1.009361   3.791273
    6  FABP5 -1.186742   3.189549

I want to make a separate plot from each of these data tables in the list and then save, which I can do using the following function:
volc = function(input){
  ggplot(input, aes(logFC, negLogPval)) +
  geom_point()
  ggsave(paste0("Volcano_", names(input), ".png"), device = "png")
}

If I run this on a single data table as below:
volc(list$table1)

Then it generates a plot as expected, but instead of taking the name of the object to name the file, it takes the name of the first column in the data table instead, i.e. saves the file as "Volcano_gene.png" instead of "Volcano_table1.png".
This means that if I run this on my list of data tables using lapply, as below:
lapply(list, volc)

It runs through the code on each of the tables, but saves all of them as "Volcano_gene.png", overwriting the previous table as it goes, meaning the only plot that gets saved is the very last one generated.
I am sure I am missing something simple with the names() function, but can anyone help me fix this so that it saves the plots with names as expected?
Comment: If you want some pretty Volcano plots, see [EnhancedVolcano](https://bioconductor.org/packages/devel/bioc/vignettes/EnhancedVolcano/inst/doc/EnhancedVolcano.html) package.
Comment: Will check it out, I have been playing with colouring my plots with thresholds etc manually, I just left them out of the example code. This might be a simpler method though!
Answer: You don't want the column names of the data.frames in that list but their names, i.e., names(list). I suggest iterating over these names:
volc = function(inputname){
  ggplot(list[[inputname]], aes(logFC, negLogPval)) +
  geom_point()
  ggsave(paste0("Volcano_", inputname, ".png"), device = "png")
}

for (x in names(list)) volc(x)

Comment: Ah I see. the `names(list)` was applying the `names()` function to the individual data tables in the list, not to the list itself.
Is there a similar way to do this using `lapply` rather than a `for` loop? I usually use `lapply` as I prefer the syntax, but perhaps `for` is just easier in this circumstance.
Comment: A `for` loop is the appropriate construct if you call a function purely for side effects as you do here.
Comment: Maybe I am misunderstanding the terminology, but how is this example using it for the side effects? The `ggplot` is called for its function of drawing a plot, and the `names()` is called for its function of providing the names of items in a list. Right? I am new to coding so maybe side effects has a different meaning to what I am used to?
Comment: Saving files and drawing plots are side effects as opposed to returning a value.
Comment: Hmm ok, that doesn't seem obvious, but thanks.

Also I finally tried to run your version to see if it worked well as the Map() version by Tommy, but it came back with the following error:
`Error in list[[inputname]] : object of type 'builtin' is not subsettable`
EDIT: nevermind, I solved it myself almost as soon as I posted, I forgot to rename "list" to my actual listname.
Comment: It is just what I have been using until now, perhaps you are right and for is better in this situation
Answer: The problem with your code is that when you apply your function to each data.frame in your list then names(input) gives you the column names of the data.frame. See this example.
data(mtcars)
l = list()
l$a = mtcars[, 1:2]
l$b = mtcars[, 2:3]

lapply(l, names)
$a
[1] "mpg" "cyl"

$b
[1] "cyl"  "disp"

To resolve this change the function to take a second argument name.
volc = function(input, name) {
  ggplot(input, aes(logFC, negLogPval)) +
  geom_point()
  ggsave(paste0("Volcano_", name, ".png"), device = "png")
}

Now you can use Map to archieve your goal:
Map(volc, list, names(list))

Comment: I see, thanks for the help. I responded to Roland before I read this, but this is a better way of doing it across the whole list without using `for` loops.


Build a IIS and SQL Server replicated cluster

Question: I have four Microsoft Windows Server 2012 R2 Standard virtual machines on three physical nodes. Two will be dedicated for IIS, and two for SQL Server 2014 Standard.
I have to replicate them (IIS1 to IIS2 and SQL1 to SQL2) to have an high availability system.
I already have a load balancing/failover system, because I can have this services on the perimetral firewalls, and if it possible I would like that every nodes have local storage replicated to/from the other node, so if it possible I wouldn't use a shared storage like SAN.
What configuration do you advise to me?
Answer: Hyper-V on 2012 and 2012 R2 has built-in replication of virtual machines. Just enable it and replicate the VM's.
Comment: I cannot do that because I'm not using Hyper-V and I cannot replicate virtual machine. Cannot I replicate from node to node?
Comment: @Mat Well, for SQL you could use AlwaysOn (SQL Enterprise License required), mirroring, log shipping, replication. Do you really need to keep IIS replicated? Can't you just deploy the web application to both servers when you release new updates?
Comment: Are you referring to "Alwayson Failover Cluster Instances"? I see that for 2 nodes it's available on SQL Server 2014 Standard, isn't it? Will mirroring or replication work both sides? Will both nodes be active? Yes for IIS, I think I could deploy two times.
Comment: @Mat No, FCI is for shared storage clusters. AlwaysOn can use a clustered SQL instance and provide AlwaysOn replication to another cluster or host (like a remote site). It still requires a enteprise license.
Comment: Ok, thank you very much. I have a standard license for SQL Server. What is the best I can use so?
Comment: @Mat I wrote several options for you in the first comment reply. Please read it.
Answer: For iis hosting you can use shared configuration :
http://www.iis.net/learn/manage/managing-your-configuration-settings/shared-configuration_264#02
For site content you must use "staging" and production concept .
One server is "staging" and you push your site with webdploy from a local iis on your laptop to this staging server for exemple . And from staging to prod when staging is ok :)
Some script to add but easy .
For sql2012 if you want easy to do "high availibility" you must use sql always on FCI( fail over cluster) :
https://msdn.microsoft.com/en-us/library/ms189134.aspx
But and old solution for sql replication is just use sql logshipping for replication easy to use and low prerequire .
https://msdn.microsoft.com/en-us/library/ms187103.aspx
And you could backup from this logshiping target server without loading "prod" database.
For small scale deploy webdpoy ,shared config ,sql logshiping is a robust solution .


React state defind as property deep clone returns 'is undefined'

Question: I'm trying to write a simple ToDo app.
Clicking on TodoItem should remove a item from state but when I do this I'm getting a TypeError: this.state is undefined. I don't know what is wrong.
My code looks like this:
---- App.js  ----
import React from "react";
import TodoItem from "./components/TodoItem";

class App extends React.Component {
  state = {
    items: {
      item1: { text: "Nazwa item1", isDone: false },
      item2: { text: "Nazwa item2", isDone: false },
      item3: { text: "Nazwa item3", isDone: false },
      item4: { text: "Nazwa item4", isDone: false }
    }
  };

  removeItem(key) {
    const items = { ...this.state.items }; // TypeError: this.state is undefined
    console.log(items);
  }

  render() {
    return (
      <React.Fragment>

          {Object.keys(this.state.items).map(key => (
            <TodoItem
              removeItem={this.removeItem}
              index={key}
              key={key}
              item={this.state.items[key]}
            />
          ))}

      </React.Fragment>
    );
  }
}

export default App;

---- TodoItem.js  ----
import React from "react";

const TodoItem = props => {
    return (

 props.removeItem(props.index)}>
            {props.item.text}
            {props.item.isDone ? "zrobione" : "niezrobione"}

    );
};

export default TodoItem;

Answer: The issue is that this is bound to your function and not to the class.
To prevent that you can use an arrow function to keep the binding of this to the class, or calling .bind method
That should make it work:
 removeItem = (key) => {
    const items = { ...this.state.items };
    console.log(items);
  }

or that:
removeItem={this.removeItem.bind(this)}

Here is a more in-depth article
Comment: Maybe, if they have the right transpiling setup (which they probably do). But a good answer says **what** it changed, and **why**, and **how** it works. (And probably removes the hopefully-now-erroneous comment.) In any case, this is an opt-repeated duplicate well covered by previous answers.
Comment: Changing to arrow function solved a problem.
Comment: @T.J.Crowder I am a fool for these points. :(


Express $z$ in terms of $x$ and $y$, i.e., find $z= f(x,y)$

Question: I've been banging my head against the wall for a while now:
$x = s^2 - t^2$
$y = s + t$
$z = s^2 + 3t$
Express $z$ in terms of $x$ and $y$.
Answer: Hint: What is $\dfrac{x}{y}$ equal to?
Comment: Follow up question - what's $(s + t) + (s - t)$ equal to?
Comment: If you know $s-t$ and $s+t$ then surely you can find $s$ and $t$.
Comment: $x/y$ = $s - t$

I've done that and tried incorporating it but just don't see how it works.
Comment: @Hans: great question (+1)
Answer: $$x=s^{2}-t^{2}=(s+t)(s-t)$$
so
$$s+t=\frac{x}{s-t}$$
$$s-t=\frac{x}{s+t}=\frac{x}{y}$$
$$(s+t) + (s-t) = 2s=\frac{x}{s-t}+\frac{x}{y}$$
$y=s+t$, so $t=y-s$ and therefore:
$$2s=\frac{x}{2s-y}+\frac{x}{y}=x(\frac{1}{2s-y}+\frac{1}{y})$$
$$2s=x(\frac{y}{2sy-y^2}+\frac{2s-y}{2sy-y^2})=x(\frac{2s}{2sy-y^2})$$
$$1=\frac{x}{2sy-y^2}$$
$$2sy-y^2=x$$
$$2sy=x+y^2$$
$$s=\frac{x+y^2}{2y}$$
From $z=s^2+3t$ we have:
$$z=(\frac{x+y^2}{2y})^{2}+3t$$
$y=s+t$ so $t=y-\frac{x+y^2}{2y}$ and finally:
$$z=(\frac{x+y^2}{2y})^{2}+3(y-\frac{x+y^2}{2y})$$
Pretty sure this is correct...
Comment: Welcome aboard.
Answer: Try to express $s$ and $t$ as functions of $x$ and $y$ from the first two equations. Then plug these expressions into the third equation.


Bijection between $2^{\Sigma^*}$ and $\mathbb R$

Question: Is there a Bijection in  between $2^{\Sigma^*}$ and $\mathbb R$? Here $2^{\Sigma^*}$ denotes the set of all languages over a finite alphabet.
If I have an uncountable set, then its powerset will also be uncountable, but there is no bijection between them.
Of course, sometimes I can find a bijection between two uncountable set: take any uncountable set and itself. And, there is a bijection between $[0,1]$ and $\mathbb R. $ But my question is there is there any possibility of bijection between two uncountable set $2^{\Sigma^*}$ and $\mathbb R$ where $2^{\Sigma^*}$ is the set of all languages. I know that the set $2^{\Sigma^*} $ is uncountable by cantors theorem.
Comment: Well, [this question](https://math.stackexchange.com/questions/1719885/bijection-between-0-1-and-the-space-of-binary-sequences) shows that $2^{\mathbb N}$ has the same cardinality as $\mathbb R$. So all you need to show is that $\Sigma^*$ is countable.
Comment: $\Sigma^*$ is easy to proof countable but there is possible one-to-one correspondence possible between $2^{\Sigma^*}$ and $\mathbb R$?
Comment: There's a one to one correspondence between $2^{\mathbb N}$ and $\mathbb R$, so yes.
Comment: But what about $2^{\Sigma^*}$ and $\mathbb R$?
Comment: $\Sigma^*$ is countable, so $2^{\mathbb N}$ is in a one to one correspondence with $2^{\Sigma^*}$...
Comment: @Don if your comments insert into answer I will accept it.
Comment: It depends on your vocabulary is it countable?
Comment: @Math how is it countable?
Comment: @User4567 I mean the alphabet your working on? what is it?
Comment: is it {0,1}? something else?
Comment: @Math you are saying alphabet about of $2^{\Sigma^*}$?
Comment: @Math {0,1} or {a, b} whatever.
Comment: @User4567 ok consider my answer
Answer: Say you are working over the  alphabet, $A=\{ 0,1 \},$ the set of all finite strings over $A$ obviously has cardinality of the naturals, now in order to get the cardinality of all languages over $A$ you want the cardinality of the  power set of the set of all strings  which is the cardinality of the the power set of the naturals. It is a well known theorem that the power set of the naturals is equinumoruous with $\mathbb{R}$ whose proof can found in text on real analysis. Note that you can replace $A$ with any finite set.


AWS ALB sticky cookie issue

Question: recently we had switched from aws elb to aws alb; but we are facing issue in aws alb; cookie stickiness is not working at all; for each request (event ajax request on the page) generates a new cookie;
if we switch back to aws elb again cookie stickiness working perfectly fine.
Answer: AWSALB is a cookie generated by the Application load balancer in the AWS. It works slightly different from AWSELB.
The ASWALB cookie is load balancer generated cookie. Once the load balancer routes your request to one of the servers, it generates a new cookie and sends it to the client in the response header. Though AWSALB works fine when you just send the webapp_session for consecutive request, it is better to integrate the AWSALB cookie along with the webapp_session and send it for each request to the server. This way, the load balancer will identify your target server and direct your request to the same target in each request call to the server. This preserves the stickyness of the server.
However, if you send AWSALB along with the webapp_session in the request header for each request to the sever, even if the target that processed your previous requests gets busy for future requests, the AWSALB generated in each call will help the load balancer identify your credentials and route your request to the new target. I faced similar situation and this helped me solve the problem.
The "STICKY SESSION" topic in this https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#sticky-sessions will perhaps help you have some idea.
Answer: For Application LB you have to set up session stickiness in target group not directly in LB like in Classic LB. Session stickiness doesn't work when your browser blocks AWSALBCORS, AWSALB - this seems to be obvious, but Internet Explorer can block these cookies and doesn't show any warning/info in console.
Comment: yes that's it, i just activated in the target group and worked fine :)

AWS ALB sticky cookie issue


Revit/AutoCAD Plugin connection to Webservice/Server possible?

Question: just a quick and basic principle question. Is it possible to create a Revit or AutoCAD plugin that can communicate with a Webservice/Server that is created by me (REST or Soap for example)?
For example the user uses my Plugin enters some numbers or other things. These numbers are send to my server, calculated and the result is send back to the plugin? Is that possible or blocked by Revit/AutoCAD.
Thanks for possible replies and have a nice day.
Answer: Yes, sure.
The Building Coder provides several examples.
Here is one:
https://github.com/jeremytammik/FireRatingCloud
Answer: Of course it is possible. Why on the hell would AutoCAD or Revit would block things like that?
Comment: Thanks for the reply. Maybe security or money reasons  I thought.


Every time I restart mysql, I see the following in the error logs immediately after the restart

Question: 2017-06-29T17:38:11.740837Z 0 [Note] Event Scheduler: Loaded 0 events
2017-06-29T17:38:11.741470Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.18-0ubuntu0.16.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
2017-06-29T17:38:11.741496Z 0 [Note] Executing 'SELECT * FROM INFORMATION_SCHEMA.TABLES;' to get a list of tables using the deprecated partition engine. You may use the startup option '--disable-partition-engine-check' to skip this check.
2017-06-29T17:38:11.741511Z 0 [Note] Beginning of list of non-natively partitioned tables
2017-06-29T17:38:11.882383Z 0 [Note] End of list of non-natively partitioned tables
2017-06-29T17:38:11.889889Z 5 [Note] Access denied for user 'root'@'localhost' (using password: NO)

I am on Ubuntu 16.04 LTS. I do not know what is running the SELECT or what is trying to login as the root user. I do not suspect malware as these are brand new installed machines. These lines come instantly after each restart.
How do I further debug what is running the SELECT and what is being denied access for root@localhost? I have turned off all of my processes.
Thanks!
Answer: You have MySQL 5.7.18. Note from Reference Manual Chapter 22 Partitioning:

As of MySQL 5.7.17, the generic partitioning handler in the MySQL
  server is deprecated, and is removed in MySQL 8.0, when the storage
  engine used for a given table is expected to provide its own
  ("native") partitioning handler. Currently, only the InnoDB and NDB
  storage engines do.
Use of tables with nonnative partitioning results in an
  ER_WARN_DEPRECATED_SYNTAX warning. Also, the server performs a check
  at startup to identify tables that use nonnative partitioning; for any
  found, the server writes a message to its error log.
To disable this
  check, use the --disable-partition-engine-check option.
To prepare for migration to MySQL 8.0, any table with nonnative
  partitioning should be changed to use an engine that provides native
  partitioning, or be made nonpartitioned. For example, to change a
  table to InnoDB, execute this statement:
ALTER TABLE table_name ENGINE = INNODB;

Based on the next two lines it looks like you don't have these tables, so you probably don't have to worry about this issue, unless the unsuccessful root login is related to that test. If you want to be sure, log in as root and execute the query from the test manually, i.e. (tables with other engines):
SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE ENGINE NOT IN ('INNODB', 'NDB');

Whether you want to use the --disable-partition-engine-check option after this manual test is up to you. However, it's just a couple of log lines and it might be good to stay informed if someone adds such tables, ever, until it's time to upgrade to MySQL 8.0.
Answer: The messages tell you exactly what is going on:
2017-06-29T17:38:11.741496Z 0 [Note] Executing 'SELECT * FROM INFORMATION_SCHEMA.TABLES;' to get a list of tables using the deprecated partition engine. You may use the startup option '--disable-partition-engine-check' to skip this check.

In other words, mysql is itself executing 'SELECT * FROM INFORMATION_SCHEMA.TABLES;'. It does that in order to get a list of tables using the deprecated partition engine.
If you want it not to do that, you can use the startup option --disable-partition-engine-check to skip this check.
Comment: OK, so I don't need to worry about that, but what is the root login, or rather how do I investigate that?


Why I have to use python-dotenv in order to load env variables to Python?

Question: I am trying to read in a python script some variables that I have in an environment file myfile.env:
LOCALHOST='1'

If in my python script I just run:
import os
print(os.environ)

It prints:
environ({'PATH': '....', 'HOSTNAME': '...', 'PYCHARM_HOSTED': '1', 'PYTHONUNBUFFERED': '1'})

But in these array I don't have the LOCALHOST variable. I found that I can use python-dotenv and now I see it and working, but I don't get why I have to use it, and it's not feasible to load them only using os module.
Answer: Because a .env file is not the environment.
os.environ gets data from the environment which, somewhat oversimplified, is a string-to-string (name-to-value) mapping maintained by the operating system for each process, and inherited by child processes. Things are typically put there using export statements in a shell – they will then be inherited by all processes started from that shell — but there are other ways.
.env files are an application-level thing; the operating system does not know about them and they are not automatically loaded into the environment (and hence not picked up by os.environ). That is what python-dotenv takes care of for you.
Comment: Great thanks, and using docker, and setting env_file parameter doesnt load that file to the `os.environment`?


$X_n \to_p X$, $g(\cdot)$ discontinuous at points with positive probability, when $g(X_n) \to_p g(X)$?

Question: Suppose $X_n$, $n = 1,2,...$ is a sequence of random variables which stochastically converge to a random variable $X \in \mathcal{X}$ . Suppose $g(\cdot)$ is discontinuous at $x_0 \in \mathcal{X}$ (e.g. $sgn(x)$ and $x_0 = 0$) and that $P(g(X) = x_0) > 0$. Is there any mode of stochastic convergence of $X_n$ to $X$ which implies $g(X_n) \to_p g(X)$ (e.g. sup convergence)?
Answer: If $X_n=\frac 1 n, X=0$ then $X_n \to X$ in every conceivable  type of  convergence but the conlusion fails as soon as $g$ has  a jump at $0$.


laravel routes. `action="{{'register'}}"` what does this code do

Question: I am new in Laravel coding, I have a problem in routing. I am using Laravel 5.5. this is the code that I need an explanation on what it does and how it works.
<form class="form-horizontal" method="POST" action="{{ route('register') }}">

This code is from the registration view. The form post username, name, password, token to the server.
I would like to know how the data is handled, and where the data is routed to, through the action action="{{'register'}}".
Comment: this controller uses a trait `RegistersUsers`, look at that trait has register function, which you are searching may be
Comment: have a look at this `App\Http\Controllers\Auth\RegisterController`
Comment: the registercontroller.php only has the create function and validate function. what exactly should i be looking at.
Comment: thanks for the help
Answer: From the docs:

Named routes allow the convenient generation of URLs or redirects for specific routes. You may specify a name for a route by chaining the name method onto the route definition:

Route::get('user/profile', function () {
    //
})->name('profile');

You may also specify route names for controller actions:

Route::get('user/profile', 'UserController@showProfile')->name('profile');

Once you have assigned a name to a given route, you may use the route's name when generating URLs or redirects via the global route function:

{{ route('profile') }}

Comment: @NaveedSheriffdeen `register` is a part of auth routes. Run `php artisan route:list` command to see all registered routes with their names.
Comment: my route.php file has `Route::get('/', function () {
    return view('register');
});
` so how is the `action="{{'register'}}"` code running without any routes named register
Comment: @NaveedSheriffdeen when you use
{{ route('profile') }}
profile is your name inside route file like
Route::method('url','Controller/function')->name('profile')
Once you have assigned a name to a given route, you may use the route's name when generating URLs or redirects via the global route function:


Pagina de Arvore - Oracle Apex

Question: Estou com dificuldade para criar uma página de arvore no oracle apex.
Criei a página, mas o apex exibe os resultados, porém não agrupados pelo nó especificado.
É uma relação de títulos pagos. Gostaria de criar o nó Fornecedor e abaixo dele os títulos relacionados.
Criei uma view para buscar os títulos:
CREATE OR REPLACE VIEW V_TIT_FORN AS
SELECT P.NOME, T.CODIGO, T.CODFORNEC, T.PARCELA, T.DTVENCTO, T.VLRPARCELA, T.VLRPAGO
  FROM PESSOA P
  ,    TITULO T
 WHERE P.CODIGO = T.CODFORNEC
   AND T.TPTIT = 0;

E o select de criação da página ficou assim:
select case when connect_by_isleaf = 1 then 0
            when level = 1             then 1
            else                           -1
       end as status,
       level,
       "NOME" as title,
       null as icon,
       "CODIGO" as value,
       null as tooltip,
       null as link
from "#OWNER#"."V_TIT_FORN"
where DTVENCTO > TO_DATE('01/03/2016','DD/MM/RRRR')
start with "CODFORNEC" is NOT null
connect by prior "CODIGO" = "CODFORNEC"
order siblings by "NOME"

Notem que coloquei no start with como NOT NULL. Se
Comment: start with "CODFORNEC" is NOT null.
Comment: Pesquise um pouco sobre o bloco with usando search breadth first. Talvez seja isso o que você precise
Answer: Desculpe postar de forma tardia...mas vai aí a forma como funcionou.

select case
      when connect_by_isleaf = 1 then
       0
      when level = 1 then
       1
      else
       -1
   end as status
  ,level
  ,name as title
  ,null as icon
  ,id as value
  ,null as tooltip
  ,decode(level,4,'f?p=&APP_ID.:34:'||:APP_SESSION||':::RP,34:P34_CODIGO,P34_PARCELA:'||titulo||','||parcela,null) as link
from (
    select to_char(p.codigo) id,
           to_char(null)     parent,
           p.NOME            name,
           0                 titulo,
           0                 parcela
      from pessoa p,
           titulo  t
     where p.codigo = t.codFornec
       and t.vlrpago = 0

    UNION

    select distinct t.codFornec || ':' || to_char(t.dtvencto,'dd/mm/rrrr') id,
           to_char(t.codFornec)                                            parent ,
           'Data: ' || to_char(t.dtvencto,'dd/mm/rrrr')                    name,
           0                 titulo,
           0                parcela
      from titulo t
     where t.vlrpago = 0

    UNION

    select t.codFornec || ':' || to_char(t.dtvencto,'dd/mm/rrrr') || ':' || t.codigo id,
           to_char(t.codFornec) || ':' || to_char(t.dtvencto,'dd/mm/rrrr')           parent ,
           'A.P.: '|| t.codigo                                                       name,
           0                 titulo,
           0                 parcela
      from titulo t
     where t.vlrpago = 0

    UNION

    select t.codFornec || ':' || to_char(t.dtvencto,'dd/mm/rrrr') || ':' || t.codigo ||':'|| t.parcela id,
           to_char(t.codFornec) || ':' || to_char(t.dtvencto,'dd/mm/rrrr') || ':' || t.codigo          parent ,
           'Parcela: '|| t.parcela ||' - '|| to_char(t.vlrParcela,'999G999G999G999G990D00')            name,
           t.codigo          titulo,
           t.parcela         parcela
      from titulo t
     where t.vlrpago = 0

    )
start with parent is null
connect by parent = prior id
order siblings by name

Pagina de Arvore - Oracle Apex


Show that a closed path is homotopic to a constant path

Question: Let $\delta$ in $(\mathbb{R}^2-\alpha)$ where $\alpha$ is a path joining $0$ to $\infty$ which is injective, i.e., it is a simple path, it does not intersect itself. Show that $\delta$ is homotopic to a constant path in $(\mathbb{R}^2-{0})$. I am assuming the path are defined on the interval $[0,1]$.
Here, homotopic means that there is a continuous application from $\delta$ to the constant path, with fixed extremities, i.e. I should be able to crunch $\delta$ to a point without moving one of its point which we will assume to be $\delta(0)=\delta(1)$ (since $\delta$ is closed). Also, obviously, the crunching should not involve an intermediary path crossing $0$.
This means, assuming the paths are parametrized on the interval $[0,1]$, that I should be able to find a continuous application $H$ from $[0,1]^2$ to $\mathbb{R}^2-0$ such that $H(0,t)=\delta(t), H(1,t)=\delta(0), H(s,0)=\delta(0)=H(s,1)$ (this last double equality encodes fixed extremities,
 the $H(s,t)$ are intermediary paths which should continuously go from $\delta$ to the constant path)
A hint is given, telling me to consider $\delta(t)-\alpha(s)$, but I don't know what to do from here, since it does not even fix extremities and goes to infinity. It does not intersect $0$ though, which is a good thing at least. What can I pick for $H$ that will work for any path $\delta$? If its interior is convex then this is easy, but otherwise this can become complicated...
Comment: As stated, this isn't true (if $\alpha$ intersects itself, it may not even be connected !), but its connected components will be simply connected indeed.
Comment: Just a note : by Jordan's theorem, there is a homotopy in $\mathbb{R}^2\setminus\alpha$ as well (though I'm not sure you're allowed to used that theorem)
Comment: @Max I shouldn't be using this theorem, anyway this is obvious even from a cursory  glance that a plane without a branch from $0$ to $\infty$ is simply connected. I think I am supposed to give an explicit formula using $\delta(t)-\alpha(s)$ for $H$. I've been on it for hours, and this is a question from an old exam that's supposed to last 2 hours. I'll have to sit for this exam in a week. I wish I were dead, I made a tragic mistake by picking a geometry class this year. The profs there simply brush everything off as "obvious from a glimpse", but us students are not held to the same standards.
Comment: @Max Yes, thank you, here it is implied that $\alpha$ is injective, i.e., it does not intersect itself.
Answer: This doesn't directly give a homotopy $H$, but the argument is at least elementary.  Consider, as in the hint, the family of loops $$\gamma_s: S^1 \to \mathbb{R}^2 \setminus \{0\}, \quad t \mapsto \delta(t) - \alpha(s)$$ for $s \in [0, \infty)$.
Since $\alpha(s) \to \infty$ and the image of $\delta$ is bounded, there exists $S \geq 0$ such that $\lvert \alpha(S) \rvert > \lvert \delta(t) \rvert$ for all $t$.  I claim that the degree of the map $\gamma_S: S^1 \to \mathbb{R}^2 \setminus \{0\} \simeq S^1$ is zero.  If not, then for some $t$, $\delta(t) - \alpha(S)$ must be perpendicular to $\alpha(S)$, i.e., $0 = \langle \delta(t) - \alpha(S), \alpha(S) \rangle = \langle \delta(t), \alpha(S) \rangle - \lvert \alpha(S) \rvert^2$.  But by Cauchy-Schwarz, we have $\lvert \langle \delta(t), \alpha(S) \rangle \rvert \leq \lvert \delta(t) \rvert \cdot \lvert \alpha(S) \rvert < \lvert \alpha(S) \rvert^2$ for all $t$ by our choice of $S$.  So indeed $\deg \gamma_S = 0$.
Now, the function $[0,\infty) \to \mathbb{Z}$, $s \mapsto \deg \gamma_s$ is locally constant.  Since $[0,\infty)$ is connected, we deduce that in particular $\deg \gamma_0 = \deg \gamma_S = 0$.  Since $\gamma_0 = \delta: S^1 \to \mathbb{R}^2 \setminus \{0\}$, this means that there is a homotopy from $\delta$ to a constant path in $\mathbb{R}^2 \setminus \{0\}$.
Comment: Thank you, this makes sense. I probably misinterpreted the hint.


How to find out if a person likes some type of clothing?

Question: Imagine, I want to build a more or less intelligent agent, which finds out, which clothing a person likes.
Clothing has certain characteristics like type (shirt, T-shirt, pants, skirt etc.), color and style.
Let's assume that first I want to find out, what types of clothing a person likes. For that purpose, I show to him or her several items of different types (i. e. several shirts, several T-shirts etc.) and ask him to tell me, if he or she likes that piece.
At the end, I will have following information:

Type of clothing
Number of times a piece of clothing of that type was presented to a person
Number of times the person liked a piece from that type

From that information, I need to find out, if that person likes that type of clothing or not.
How can I test the hypothesis like User X likes shirts apart from using cumulative binomial probabilities (see below) ? Are there other, better ways to do it?
Binomial probability method
At highschool I learned that such hypotheses can be tested using the Binomial probability.
In order to test, whether the user like a particular type of clothing, I do following:

Calculate the cumulative binomial probability.
If cumulative binomial probability is higher than 50 % for a particular data set (number of times the user saw the picture, number of times she liked it), then I regard the hypothesis "User likes type of clothing X" as confirmed. If the probability is less than or equal to 50 %, then this result may have occured by accident.

Example. Let's say I have shown 10 different shirts to a user. I assume that the success (user likes a particular shirt) is equal to 50 %. The table below shows different situations, which may occur - from when the user did not like any of the shirts to the one, where she liked all of them.
The column "Bin. prob. > 50 %" shows whether in that particular situation I regard the hypothesis that the user likes shirts as confirmed.

The cumulative binomial probabilities are calculated using Google Docs function =BINOMDIST(A2,10,0.5,TRUE).
Comment: You have now asked several rather broad questions, ranging from how to analyze data to how to design experiments. (There is an enormous literature on both topics.)  Please formulate *one specific question* at a time, research this site (and others, if you like) to see whether it has been answered, and edit your post to focus it on what you need to know.
Comment: Your procedure is equivalent to deciding that somebody likes something when they respond positively to it at least half the time. Although that could be a useful *criterion,* it is not an admissible hypothesis test. Moreover, it does not seem to have any way to cope with the complication that each article of clothing has *multiple* characteristics: type of clothing, color, style, etc. How do you propose to separate out the relationships between those different attributes and what people like?
Comment: You'd probably get better results if you asked people if they owned something like the sample piece.
Comment: I thought of 1) running a series of tests for finding out what types of clothing the user likes, then 2) running an additional series of tests to find out what color he/she likes and repeat this procedure for all other attributes. That is, show them 10 scarfs (to determine, whether she likes scarves or not), show them 10 red items (to determine whether she like red stuff), show them 10 blue items etc. The problem with this approach is that it might take too many attempts to figure out, what the user likes. That's why I'm asking. If there is accessible literature on how to do it,please tell me.
Answer: It sounds like you want a system that, given some training data about a person's likes and dislikes, is able to predict whether the person will like something.  That problem is known as classification.  The basic idea is to collect a data table where each row is an article of clothing, the columns describe the article via numerical or discrete attributes (such as 'color'), and one column gives the label ('like' or 'dislike').  Then you can apply an algorithm such as logistic regression to learn a classifier from this data.
Answer: I share other users' confusion as to what the heart of your question is. But you reference an 'intelligent agent' you'd like to build, so I gather that your ultimate goal is to suggest products a user is likely to prefer. Systems that perform this task are called recommender systems.
Collaborative filtering is a common approach to this problem. Taking your example, say you wanted to predict whether or not I would like a certain shirt. As you get at in your comment, designing an experiment to measure my likes and dislikes could be rather impractical. Very simplified, a collaborative filter would attempt to find one of:

users like myself, and suggest their items
items similar to ones I like, based on other users' histories

As for the approachable literature mentioned in your comment, an oft-cited and wonderful resource is Andrew Ng's open course on machine learning. This video begins the series on recommender systems and collaborative filtering.


Search elasticsearch string field by wildcard

Question: My elasticsearch index has a "message" field of type string. I'm trying to search for records that contain a partial match on the message field. For example, if the record contains "Hello World", then searching for "hello", "Hello", "o w", etc... would result in a match.
Here is a sample record in the index:
{
        "_index": "applicationlogs",
        "_type": "upstream",
        "_id": "d7Hj12EBs1AV7_j_56mg",
        "_score": null,
        "_source": {
          "@timestamp": "2018-02-27T09:33:16.7817021-06:00",
          "level": "Information",
          "messageTemplate": "Elasticsearch log level configured to Information",
          "message": "Elasticsearch log level configured to Information",
          "fields": {
            "SourceContext": "Rpr.Agg.Program",
            "ApplicationVersion": "184.108.40.206",
            "ServiceName": "BDF_SERVICE_SWBR2DEV-JTC01",
            "MachineName": "SWBR2DEV-JTC01",
            "Application": 8
          }
        },
        "sort": [
          1519745596781
        ]
      },

Here is the relevant part of the class that mirrors the index:
public class ElasticsearchLogEntry
    {
        [Text(Name = "@timestamp")]
        public DateTime TimeStamp { get; set; }
        public string Level { get; set; }
        [Text]
        public string Message { get; set; }
    }

And here is the relevant part of the query that I have so far, which only matches on the full string:
public async Task<IEnumerable<ElasticsearchLogEntry>> GetAllLogEntries(string search, int skip, int take)
        {
            var filters = new List<Func<QueryContainerDescriptor<ElasticsearchLogEntry>, QueryContainer>>();
            if (!String.IsNullOrEmpty(search))
                filters.Add(fq => fq.Match(m => m.Field("message").Query(search)));

            var response = await client.SearchAsync<ElasticsearchLogEntry>(i => i
                .Index("applicationlogs")
                .Type("upstream")
                .Query(q => q.Bool(bq => bq.Filter(filters)))
                .Sort(s => s.Descending(so => so.TimeStamp))
                .Skip(skip)
                .Take(take));
            return response.Documents.AsEnumerable();
        }

Any ideas how I can get it working correctly?
Answer: I had assumed that I was simply getting 0 results back, but it turns out I was actually getting an error:

{ServerError: 400Type: parsing_exception Reason: "[match] query does
  not support [type]"}

I thought this was because I'm setting a Type in my query, however even after removing it I still get the error. I think this is because I'm connecting to a ES 6 instance on Amazon while using the NEST / ElasticSearch.Net 5.x nuget packages. See this.
Unfortunately, the whole reason why I'm using the 5.x nuget packages is because 6 didn't work for me or the other developers on my team. The api calls always give this error:

UnexpectedElasticsearchClientException: Method not found:
  'Elasticsearch.Net.PostData`1
  Elasticsearch.Net.RequestData.get_PostData()'.

So this leaves me in a bit of a bind...
EDIT: They released an update to the ES AWS nuget package that supports 6.x, so I was able to upgrade everything and get past these errors.
Comment: As you've found, NEST 5.x is not compatible with Elasticsearch 6.x. When updating to NEST 6.x, ensure that Elasticsearch.Net package is also updated to a version compatible (usually same version as NEST).
Comment: @RussCam Unfortunately I can't upgrade to NEST 6.x because the ElasticSearch AWS package isn't compatible with it (https://github.com/bcuff/elasticsearch-net-aws/issues/33). Any workaround that you can think of?
Comment: If you do not have multiple types in your index, try commenting out the .Type("upstream") line.
Comment: @Justin easiest approach might be to build Elasticsearch AWS package against NEST 6.x, use that and then open a PR against the project
Comment: They released an update to the ES AWS package to support 6.x, so everything is working now.


why does y axis sometimes change from normal histogram to kernel density?

Question: Consider the distributions I have plotted below. They are of the same variables, one in normal histogram form and another in kernel density (Epachanov).

As far as I know, the auc of the kernel curves is 1, so I'm not sure why the y-axis would need to change sometimes for that. Yeah, the main thing is I'm not sure why sometimes the y axis is the same and sometimes the y axis is different.
Question
Why did the y axis remain the same for the left two (0 to .8) but changed for the right two (histogram is 0 to 14, kernel is 1 to 7)? Feel free to explain about how the y axis factors into interpretation if you don't mind.
Answer: For the LHS plots, the areas under the curves are n near $1$, so you don't see a range change in y-axis. We can inspect it visually, by reading $y$ values for each bin, and multiplying it with the step size, $0.25$ for the blue histogram. For example, the it has an approximate area of
$$(1/3\times3+1+2+3+3.5+5+7\times2+3.5+1+1.5)/10\times 0.25\approx 0.91$$
This is my simple approximation, you can come up with another, or directly calculate it from the data. But, it'll be around $1$. The situation is similar with the red histogram. KDE is not the same as normalized histogram but for simplicity you can think like we normalize the curve with 1/AUC. If AUC is close to $1$, then normalization won't affect the value on the y-axis a lot.
It's harder to do visual inspection for the red plot on RHS, however, clearly, the AUC won't be close to $1$ so that the resulting KDE curve's y-axis values will be in different range. Also, some jumps might not well represented in KDE smoothed PDF. And, for the blue curve in RHS, y-axis range didn't change significantly. In the histogram, most of the values fluctuate around $1$, and the AUC is therefore $1$ since the support spans $[0,1]$. The KDE also produces a curve with y-value near $1$.


Average time for a tetris field to fill

Question: I have a tetris field, infinitely wide (irrelevant) and N blocks high. Now random blocks start falling down, and I do not move them.
Given that the field is N blocks high, the blocks aren't moved, the blocks fall one block per second and the blocks are the default blocks (they always "spawn" with the same orientation), how long does it take on average before the game is over?
Comment: It seems like this problem would require many cases, finding the probability of the cases and calculating the expected time.
Comment: Great to see questions like this! Thanks nightcracker! :)
Answer: There are $7$ different kinds of blocks. The height of the stack will generally increase by $2$ in each step, but only by $1$ in the following cases: A green block preceded by any block; a purple block preceded by a red block; and a blue block preceded by either a yellow or a gray block. I'll treat the events of certain pairs following each other as independent; they're not, but it doesn't make a difference for the average height after time $t$ (by linearity of expectation) and only a small difference for the average time $t$ for height $N$.
So we have $49$ different pairs that are assumed to occur with equal and independent probabilities of $1/49$; for $1\cdot7+1\cdot1+1\cdot2=10$ of these pairs the second block increases the height by $1$, and for $49-10=39$ of them by $2$. Thus the average height after time $t$ will be $\frac{10+2\cdot39}{49}t=\frac{88}{49}t$. A good estimate for the average time to reach height $N$ is therefore $\frac{49}{88}N$. But we can be a bit more precise than that.
The average time $t_N$ to reach height $N$ satisfies:
$$t_N=\frac{10}{49}t_{N-1}+\frac{39}{49}t_{N-2}+1\;,$$
since there is a probability of $\frac{10}{49}$ that the next step will have height $N-1$ left and a probability of $\frac{39}{49}$ that it will have height $N-2$ left, and in either case the current step takes time $1$.
We can solve this inhomogeneous linear recurrence relation by adding a specific solution of the inhomogeneous relation to the general solution for the homogeneous relation and then imposing initial conditions. We can guess a specific solution for the inhomogeneous relation from our earlier estimate: It is simply $t_N=\frac{49}{88}N$. For the general solution of the homogeneous equation, we have to solve the characteristic equation
$$\lambda^2-\frac{10}{49}\lambda-\frac{39}{49}=0\;,$$
with solutions
$$\lambda_{1,2}=\frac{5\pm\sqrt{1264}}{49}\;,$$
$$\lambda_1\approx 0.8276, \lambda_2\approx -0.6235\;.$$
Then the general solution of the inhomogeneous relation is
$$t_N=\frac{49}{88}N+c_1\lambda_1^N+c_2\lambda_2^N\;.$$
The initial conditions are $t_0=0$ (if there is no height left, the game is over) and $t_1=10/49$ (if there is only one row left, the game is over unless the next block only increases the height by $1$, with probability $10/49$). Substituting these into the general solution yields
$$c_2=-c_1=\frac{10-49^2/88}{\sqrt{1264}}\approx 0.2431\;.$$
Thus, the correction terms proportional to $c_1$ and $c_2$ are quite small to begin with and die off quickly, so by the time the height reaches $N$ the result is well approximated by the initial estimate $t_N=\frac{49}{88}N$. The height in the image seems to be $N=20$, so this would lead to an average time $t_N=\frac{245}{22}\approx11.1$ until the game is over.
Comment: very rigorous (+1)! now you could go even further and consider that the blocks fall down from the top with a given velocity (say on line per second) and calculate the actual duration (considering that the last lines are faster filled due to the shorter falling way)
Comment: Nice answer! Would you recommend any textbook that teaches probability at this level?
Comment: @Mark: I'm sorry, I don't know any.
Comment: Eehm, a pair of 2 squares increases height by 4 and a pair of a long one and a L increases height by 3.
Comment: @nightcracker: My formulation may have been ambiguous. In talking about pairs $(A,B)$ of colours, I was analyzing by how much a block of colour $A$ increases the height when preceded by a block of colour $B$, not about the total height increase due to the pair.
Comment: @nightcracker: I edited the answer to make it clearer; thanks for pointing that out.
Comment: @joriki Perhaps a book that is at least as rigorous as your arguments?
Comment: @Mark: Sorry, I don't think I've ever read a book about probability, and if I did, it was probably in German and a long time ago :-)
Answer: For a lower boundary, consider the average height of these blocks (assuming as you stated no rotated version is spawned) is (6 x 2 + 1 x 1)/7 = 13/7 = 1.857..., the answer is N/(13/7) = 7N/13, e.g. for N=26 its 14 blocks. This is not exact though, you'll have to include the chances of e.g. a purple block dropping on a red one, yielding a total height of only 3 and so on.
Luckily, only all combinations of two sequential blocks have to be considered, since without rotations only the blue and purple blocks following white&yellow or red blocks respectively can result in a merged line and it's not possible that any three block combination merges any further. That's 3 out of 7² two-block combinations, the average height of all two-block combinations has therefore to be corrected from 13/7 to (13/7 x 7² - the 3 lines counted too much) / 7², = 88/49 ≈ 1.796. So now the correct result is 49N/88, for N=26 that's 14.477 blocks.
Comment: That's incorrect -- more comprehensive answer to follow...
Comment: Clearly if you have a white followed by a blue block you only gain 1 block of height with the blue block
Comment: @joriki: @Alexander thanks, I didn't consider these first...
Comment: @Alexander, @joriki: corrected

Average time for a tetris field to fill


Checkbox dinâmico não aparece a seta de checado

Question: Tenho um checkbox que mesmo selecionado não aparece o item de checado. Este atributo é criado dinamicamente. Veja a imagem:

Quando clicado ele teria que estar selecionado assim:

Segue o código quando eu clico no checkbox:
$(document).on("click",".preco-prod-adicional",function(e)
{
  e.preventDefault();

  if ($(this).attr('checked') == 'checked')
  {
    alert('selecionado');
    $(this).attr('checked',true);
    var precoproduto = "";
    var valortotal = "";
    var subtotal = 0;
    var resultadoitem = "";
    var replacetotal = "";

    precoproduto = $(this).val();
    valortotal = $(".total-v .v-lor").text();
    subtotal = somarValorTotal(precoproduto, valortotal);
    replacetotal = subtotal.replace(',','.');
    resultadoitem = $(".right-imagem .valor-total .total-v .v-lor").html(replacetotal);
  }
  else
  {
      $(this).attr('checked',false);
  }

});

HTML da checkbox:
<input type='checkbox' class='preco-prod-adicional' name='checkadicional' value='" + precoprodadicional[0] + "' id='" + prodadicional[i] + "'/>

Comment: Adiciona o html do checkbox também, por gentileza. E pq você marca checked quando já testou e o elemento é checked?
Comment:
Comment: Eu ponho porque ele não aparece o checked quando checa. Ele é criado dinamicamente. Muito estranho!
Comment: Tu tá usando estilo comum ou algum outro estilo diferentão?
Comment: Não. Ele esta´em um modal sendo criado dinamicamente pela seguinte variável:   mostraAdicional +=
Comment: Porquê mudar o estado da `checkbox` no click ?  uma `checkbox` já muda de estado por defeito quando se clica
Answer: Se a ideia é fazer algo quando ele fica checked ou unchecked, melhor será utilizar o evento change e verificar o estado de checked com .is(":checked")
Exemplo:

$(".preco-prod-adicional").change(function(){

  if ($(this).is(":checked"))
  {
    //código para quando está checked
    console.log("Está checked");
  }
  else
  {
    //código para quando não está checked
    console.log("Não esta checked");
  }

});

<input type='checkbox' class='preco-prod-adicional' name='checkadicional' value='" + precoprodadicional[0] + "' id='" + prodadicional[i] + "'/>


parse xml node items minus the first

Question: I am trying to read an external xml file and return a table which I can do. I need however to set a var that can limit the number of results starting with the second node. I can get the number I want I can set that no problem. It is starting with number two thats the problem. Any help would be great.
Here is the parser I am using to write the table. I need help placing the limitation to remove the first node.

document.write("
");
var x=xmlDoc.getElementsByTagName("item");
for (i=0;i < number;i++)
  {

  document.write(""+x[i].childNodes[0].nodeValue+"	");

      }

simple xml list
<item>
  <title>One</title>
  <description> </description>
  <link></link>
  <image></image>
</item>
<item>
  <title>Two</title>
  <description> </description>
  <link></link>
  <image></image>
</item>
<item>
  <title>Three</title>
  <description> </description>
  <link></link>
  <image></image>
</item>
Answer: If I understand correctly, the solution would be to modify the starting value of i of your loop, so that the loop starts at i = 1. The effect of this would be to skip the first <item> in your XML:
// Initialize i to 1
for (i=1;i < number;i++)
{
   ...
}

Comment: ok so I have tried that but it still returns the entire list.
Comment: Can you please include more of your code in the original question?
Comment: Hey, I have two parsers one for vertical one for horizontal I didn't change it in one and checked the other.... Thanks. I just spent several hours beating my head against the wall.... Thanks again

parse xml node items minus the first


Particularly unconventional fielding positioning in baseball/softball

Question: I have never seen fielders in positions that would hardly be recognisable as the "standard" positions, and probably for good reason. (I mean a positioning where a casual observer might have difficulty identifying who is playing which position.)
However, are there known (modern) instances where teams/coaches (competing on the highest levels) chose for a "radically" different field placement on specific occasions?
(Examples might be: 1) the third-baseman is placed in the outfield, or 2) the second-baseman is placed in the outfield, or 3) the right-fielder is placed between the third-baseman and the shortstop, or ...)
Answer: Ron Roenicke, manager of the Milwaukee Brewers, has brought in an outfielder to be a fifth infielder on numerous "sudden-death" occasions.
For example, the game is tied, the Brewers are the road team, and the home team has a man on 3B with one out.
If the runner at third is the only runner, then Ron Roenicke typically walks the next batter (and will walk another if that first walked batter steals second) to set up the double play.
Then he brings in one of the outfielders to fill a gap in the infield.
It appears that this happened most recently in a game in Pittsburgh on June 30, 2013 – and it worked!

Pedro Alvarez came to the plate with the bases loaded and one out in
  the 13th. The Brewers (32-48) shifted an outfielder in for a fifth
  infielder and Alvarez hit into a 3-6-3 double play to end the inning.
– Jenn Menendez / Pittsburgh Post-Gazette

Roenicke has employed this strategy at least once in each of his first three years managing the Brewers.
Try searching for "fifth infielder" brewers to find other instances.
Comment: Excellent! Many thanks, also for the search suggestion. (I found that "5-man infield" also gives good results.) I also found the Rays (even in the 2008 World Series) and the Marlins doing similar things.
Answer: One of the defensive ploys that manager Joe Maddon of the Tampa Bay Rays has employed is to shift the field heavily to right field when left-handed hitters (such as Boston Red Sox's David Ortiz) come to bat. He was one of the first coaches to use to this against Ortiz and now almost every MLB employs a similar defensive shift. (Source)
Also last year Toronto Blue Jays third-baseman Brett Lawrie boasted an extremely high fielding percentage based on the fact that he was shifted between the shortshop and second basemen when left-handed hitters came to bat. (Source)
Answer: (I can't find a reference right now)
I seem to remember during the 2014 post-season (possibly during the World Series?) that the Kansas City Royals shifted all the outfielders and the Short Stop between 1st and 2nd, where the batter had a steady history of hitting, but left most of the rest of the field uncovered.
Sure enough, the batter hit right into the middle of the fielders, and they got a trivial out.
Comment: Nowadays this happens pretty regularly - it was less common when the question was initially asked, but see [this article](http://m.mlb.com/news/article/107573504/players-coaches-discuss-impact-of-the-infield-shift) for example; the number of shifts has dramatically increased in the last few years.


Not able to run command via wsl.exe

Question: I'm trying to run this commands from a python script:
def raw(path_avd_py, path_avd, snp_name, out_file):
    if OS == 'Windows':
        cmd_raw = f"wsl.exe -e sh -c 'python3 {path_avd_py} -a {path_avd}
        -s {snp_name} -o {out_file}'"
    else:
        cmd_raw = f'python3 {path_avd_py} -a {path_avd} -s {snp_name} -o {out_file}'
    subprocess.Popen(cmd_raw, shell=True)
    time.sleep(25)
    return None

def idiffer(i_path, raw_1, raw_2, path, state):
    if OS == 'Windows':
        cmd_idiff = f"wsl.exe -e sh -c 'python3 {i_path} {raw_1} {raw_2}'"
    [...]
    file = os.path.join(path, f'{state}.idiff')
    with open(file, 'w') as f:
        subprocess.Popen(cmd_idiff, stdout=f, text=True)

If im executing cmd_raw with subprocess.run from a python-shell (Powershell), things are working. If im try running this via script, this exception occurs, using different shells:
-e sh: avdecrypt-master\avdecrypt.py: 1: Syntax error: Unterminated quoted string
-e bash: avdecrypt-master\avdecrypt.py: -c: line 0: unexpected EOF while looking for matching `''
avdecrypt-master\avdecrypt.py: -c: line 1: syntax error: unexpected end of file
I already tried os.system, os.run([list]) no change.
Thanks for the help!
Comment: It's not clear why you are invoking `sh` or `bash` here, or `wsl` for that matter. Running Python as a subprocess of itself is also frequently something you should avoid; instead, `import` the code you want to run, and simply call the relevant function directly.
Comment: I'm not able to compile some dependencies in windows, which are necessary to run this script. To compile those under unix-flavoured systems are no problem. So i have to call some function from wsl. Thanks for the reply.
Comment: Still you should not need `sh` to run Python, or Python as a subprocess of itself.
Answer: For those who have a similar question, I found a solution, which is working for me:
Apparently calling scripts with some argv has to be in one single quotation mark and can be executed via run (in my case important, because the process has to be terminated). This leads to a form like:
cmd = ['wsl.exe', '-e', 'bash', '-c', '-a foo -b bar [...]']
subprocess.run(cmd, shell=True)

Lib shlex is helping here and formatting the strings like subprocess is needing it:
cmd_finished = shlex.split(cmd)

https://docs.python.org/3/library/shlex.html

Not able to run command via wsl.exe


Standard error and p-values of exponential smoothing weights

Question: Is there any justfification for producing a standard error of a single exponentially weighted coefficient?
If yes, how can we interpret the p-value?

Background
I use SAS ETS to estimate a single exponential smoothing model. When i select the optimize option for the estimation of the smoothing weight I get the estimated value (optimal), a standard error, a t-value and a p-value. How is the standard error calculated (e.g. what is the formula for it) and what is the menaing of the p-value. In the users guide it says that:

The standard errors associated with the smoothing weights are
  calculated from the Hessian matrix of the sum of squared,
  one-step-ahead prediction errors with respect to the smoothing weights
  used in the optimization process.

As far as I know the optimal weight is calcualted with a non-linear optimization procedure (heuristcs search with objective function to be minimzed the MSE or somthing similar). Based on this the statistical concepts are not applicable in this case (e.g. if we repeat the calculations in another sample then in the 95% of the cases etc.) So what is the meaning of the p-value and the standard error calcualtion? Do we have to check whether the smoothing coefficient is significant or not or we just disregard the standard error and the p-values?
Is there any use of the standard error, the t-value and the p-value?
Answer: There is an underlying statistical model where the SES weight is a parameter. So one use for the standard error is to get a confidence interval on the parameter. However, since the parameter is not really of direct interest, there is little point in doing this. The p-value on the test for whether the parameter is equal to zero may be of interest in testing whether the underlying process is a random walk. But there are better tests for that.
So while there is some meaning in these calculations, there is little value to be gained. Consequently, I don't think there is much point in providing the standard errors or p-values in software output.
Comment: Hi Rob, Thanks for your answer. Do you have in mind any material on the web or somewhere else on the method used to calculate the standard error (The standard errors associated with the smoothing weights are calculated from the Hessian matrix of the sum of squared, one-step-ahead prediction errors with respect to the smoothing weights used in the optimization process)?
Comment: Most likely SAS is assuming an ARIMA(0,1,1) model with Gaussian errors for the SES forecasts. So look at a time series book on ARIMA models. Note that other models are possible. See Chatfield et al (JRSSD, 2001).


How can I interrupt the mysqld front process?

Question: I want to ask, how is it possible to interrupt a simple mysqld command in the terminal.
By a mistake I typed in mysqld in the console and press enter.
Now I get  the Note that mysqld starting as a process (Process ID).
How can I interrupt such a command process in the future?
CTL - C, Z, \ and so on are not working here?
If you have a answer, it would be great to explain it also.
Comment: Open another shell and kill it.
Comment: Ok, thanks. Ok I thought its possible in the same shell. Could you explain me, why it is not possible in the same shell? Is there no signal existing?
Comment: this is a normal behavior, since not all program can handle ctrl+c
Answer: connect a second time and kill it
if its local, open a second terminal.
you can use
ps auxwww|grep mysql
kill -9 pid

or
if killall is available
killall -9 mysqld


Showing Independence for X and frac(X + Y)

Question: Suppose that we have independent samples $X_1, X_2 \sim \text{unif}[0, 1)^d$. I'm asked to show that $Y_1 = X_1$ and $Y_2 = X_1 + X_2 - \lfloor X_1 + X_2 \rfloor$ are also independent uniform samples from $[0, 1)^d$.
I'm able to show that $Y_2$ is uniform on $[0, 1)^d$ by using a convolution for $X_1 + X_2$ and then reasoning about $\lfloor X_1 + X_2 \rfloor$ to obtain the PDF.
However, I'm not sure how to formally show that $Y_1, Y_2$ are independent. This post indicates that identical distribution does not imply independence. I also don't think the joint PDF is calculable given only the marginals without assuming independence?
Any advice on what approach to use to show independence that uses the fact that $X_1, X_2$ are independent?
Comment: If $X_1$ and $X_2$ are independent, it should not be too difficult to show that given $X_1 = x_1$ you have $X_1 + X_2$ conditionally uniformly distributed on $[x_1,x_1+1)$ and so you have  $X_1 + X_2 - \lfloor X_1+X_2\rfloor$ conditionally uniformly distributed on $[0,1)$.  Since this latter conditional distribution does not depend on $x_1$, you get independence of $Y_1$ and $Y_2$
Comment: Thanks! I had something written down to the effect that the final PDF doesn't depend on $X_1$ but wanted to know if there was a more formal treatment (since this was just my intuition).
Answer: This is a collection of $d$ identical statements, one for each coordinate, so it suffices to examine the case $d=1$.
Let $0 \le y \lt 1$.  Define the random variable $$X_2^{(y)} = \left\{\matrix{X_2+1 & X_1 \le y \\ X_2&\text{otherwise.}}\right.$$  $(X_1, X_2^{(y)})$ has a uniform distribution on a region consisting of the unit square $[0,1)^2$ from which a triangle bounded by $(0,0), (y,0), (0,y)$ (yellow in the figure) has been removed and translated one unit upwards to the triangle bounded by $(0,1), (y,1), (0,1+y)$.

Writing $[z] = z-\lfloor z\rfloor$ for the fractional part of any number $z$,
$$Y_2 \le y \iff [X_1+X_2] \le y \iff 1 \le X_1 + X_2^{(y)} \le 1_y.$$
The probability of $X_1 + X_2^{(y)}$ within this region is concentrated in the parallelogram with vertices $(0,1), (1,0), (1,y), (0,1+y)$ (the blue region in the figure).  If, simultaneously, $X_1\le x$ (for $0 \le x \lt 1$), the probability is the area of this parallelogram to the left of $X_1=x$.  That region, too, is a parallelogram: its height is still $y$ but its base is only $x$, whence its area is $$\Pr(Y_1 \le x,\ Y_2 \le y) = xy = \Pr(Y_1 \le x)\Pr(Y_2 \le y).$$
We have seen how, throughout the support of $(Y_1,Y_2)$, its distribution function factors into the product of the marginal distributions.  Consequently $(Y_1,Y_2)$ is independent.


Using Auth::user()->user() in Laravel 5.8 or later

Question: I am trying to upgrade a Laravel project from Laravel 5.1 to 5.8. I noticed that instead of Auth::user(), Auth::user()->user() has been used in the project, and in Laravel 5.8 I get a null exception in some cases because Auth::user is null. Shall I replace all instances of Auth::user()->user() with Auth::user()?
Answer: You should either use this-
Auth::user();

Or this-
auth()->user();

Please make sure that you have no method/relation with name user() in your user model which may be the reason you have user()->user() like syntax. If that's not the case then feel free to replace it with auth()->user()
Comment: Care to explain why `auth()->user();` would be a better syntax over the other one?
Comment: It does not give any advantage over Auth::user() other than being fluent. That's why I find (prefer) it better. Laravel is all about fluent syntaxes.
Answer: If Auth:user() is null, that means that the system didn't get an authenticated user.
But if you want to get the authenticated user you should to use Auth::user() or auth()->user() thats retrive the User model
Answer: Check your route if you added the auth middleware and try using this code.
auth()->user();

and that will retreive the authenticated user same as Auth::user();


What kind of learning algorithm is suitable for classifying unknown number of groups of unlabeled data?

Question: I'm looking for a learning algorithm to analyze data, then group it and define how many classes are fit to distinguish the data set. Type of experiment is in the context of evolutionary biology, how a set of bacteria evolve over the course of time.

The data is initially unlabeled so I'm assuming I should look into unsupervised learning algorithms.
I'm trying to classify instances of the training set into groups, so I thought clustering types of algorithms would be more suitable (following a pre-processing of PCA for an easier grasp of the data).
The number of groups are unknown, so something like K-Means might be more troubling since the number of centroids is preselected before the analysis step.

I did some reading/research and stumbled upon Hierarchical Clustering, which seemed like a great candidate for this problem.
Comment: You said "change over time." I wanted to see if you meant you are interested in clustering one variable measure at multiple time points. In that case, like @alto said, growth mixture modeling could be useful if you have enough data.
Comment: Hmm.  So is this empirical data, or simulated data?  If empirical, can you tell us about the nature of the experiment?  If you are growing cultures, then knowing how the population varies spatially and something about the structure of the data can substantially improve the quality/relevance of your clustering.
Comment: You mention how bacteria change over the course of time. So are you interested in clustering one variable measured repeatedly? That is, clustering bacteria trajectories? If so, there are ways to use the time-dependent nature of the dataset to your advantage in clustering.
Comment: @EngrStudent, It definitely isn't a simulation. Mostly spatially and texture variation -wise. I'm analyzing visual stimulus (so it will include computer vision algorithms as well).
Comment: @dmartin, not sure what you mean by one variable
Comment: I still don't understand your data.  So are you measuring colony size, count, color/density as a function of time or some evolutionary marker? Lets say I had a "reactor" where I fed my e.coli some broth doped with ... chopped up bits of MRSA, and also exposed them to some x-rays.  I want to see if they get better at metabolizing MRSA over time, so I then put them in limited space, in competition with live MRSA and chart "competitiveness" vs. time using colony morphography measures.  Would my data look anything like yours?
Comment: It can, but yes similar concept.
Answer: If you have only one or two feature variables (1D or 2D dataset), I would hesitate to recommend clustering methods, because you may even not know whether there will be clusters or how many clusters. I would rather try kernel density estimation on the data to observe the peaks of probability density. The python code with SciPy package  is shown on the page.
Regarding the clustering methods, I think this interesting page indicates how the characters (size, density, spherical, metric,...etc) of the data set matter when choosing different algorithms.
Answer: If you don't know the number of clusters ahead Bayesian nonparametrics may be appropriate. Loosely these methods work by assuming there is an infinite number of clusters and using a prior, such as the Chinese restaurant process, which favors only using a small number of these clusters for any given finite data set. This allows one to do things like fit a Gaussian Mixture Model without specifying the number of mixture components.
There is a lot of good, freely available information out there on the subject, but I recommend starting with A tutorial on Bayesian nonparametric models by Gershman and Blei. It is a short and fairly easy read.
Comment: +1, sounds like exactly what I'm looking for. Plan to apply this one.
Answer: Before applying clustering algorithms like K-means you could try to visualize the data to get some insights about the number of clusters. You could apply Self Organizing Maps (SOM - https://en.wikipedia.org/wiki/Self-organizing_map) first. With SOM you can analyse your data in 2D or 3D. This way you can find a good range of clusters to apply K-means or other clustering algorithms.
Answer: I think what you are looking for is indeed a clustering algorithm.
In hierarchical clustering, you will also have to choose the number of groups unless you trust a criterion to stop the tree.
What you can do for kmeans (and also hierarchical) is calculating a few metrics (like inertia, gap statistic or silhouette) to compare each models with a different number of clusters. (ex calculate silhouette coefficient for kmeans for k = 2, 5, 10, 15 ,20 and choose k for which silhouette coefficient is maximum).
Now these two algorithms kind of supposed spherical clusters so other algorithms might be more adapted.
The best algorithm will highly depend on your data so if you provide more details on what you are working with (marketing , genomic ...) , people may be able to guide you a little bit more.
Comment: what is your data precisely ? I mean what are individuals and variables ?
Comment: See question for latest update!
Comment: Main concern against using K-Means, as you mentioned spherical clusters, is that the clusters are bound to be biased to similar variance. This might be troubling since this type of data is very likely to have uneven clusters.
Answer: Excellent question I must say.
This is a situation I run into many times myself and unfortunately have not found a satisfactory answer.
As @Scratch has said, you can use the hierarchical clustering algorithm, since it is deterministic and then you can try to come up with a heuristic which traverses the hierarchy and tries to use a metric such has membership size of a cluster to extract a final cluster list.
Another method which I've not quite gotten to work is to use clustering resampled data sets and then looking to see if there is affinity between two elements. Here is a question I had asked which might give you some hints on how to accomplish this.


Differences between groups and rings when understood as categories

Question: A group can be understood as a category with a single object $*$ whose hom-set $\text{hom}(*,*)$ forms a group with composition as the group multiplication.
A ring can be understood as a category that is preadditive with a single object $*$. Preadditive means that for a category $C$, $\text{hom}_C(A,B)$ forms an abelian group that is distributive with respect to composition of arrows.
The group structure appears in different ways in these two categories. It seems weird to me that the group multiplication is the composition of arrows for groups as categories but for rings it is some binary map of arrows in the same hom-set. Why is this the 'best' way to understand these structures as categories? Is there some way to understand a ring as a category where composition of arrows forms a group (like the categorical sense of a group)?
Comment: You seem to expect category theory to provide a "best" way to understand the definition of an algebraic structure. Category theory is a very useful organising principle, but it isn't a panacea or a required dogma.
Comment: To Rob's comment, you can see that you're never going to realize a nonassociative operation in the same way as here, for groups and rings.
Comment: @RobArthan You make a good point. What I meant was "the best way to understand these structures as categories." I've updated the question to reflect this.
Answer: I think the basic way to understand a ring as a category is as a monoid in the category of abelian groups.
On one hand this says it is one abelian group $A$ and abelian group homomorphisms $Hom(A,A)$. But if you look at what it says when you identify the elements of $Hom(A,A)$ as "elements of the ring," it just says everything about distributivity of multiplication over addition.  The category axioms provide for the associativity of multiplication in the ring monoid.
A group is a monoid in the category of sets with the extra property that its arrows are isomorphisms.
So both are realized as monoid over different categories, and the group specializes by having extra requirements on its arrows.  The closest analogue for a ring is a division ring, of course, where all its nonzero arrows are isomorphisms.
Added
Here are the details for what a monoid in a category is.. It is more involved than I remembered but it is worth working through as it provides a convenient viewpoint for many objects including rings, algebras, graded algebras, topological monoids etc.
Comment: Thanks! I'm not quite sure what you mean by "a monoid in the category of abelian groups." Is there a different or more precise way to put it? My understanding is that $Hom(A,A)$ forms an abelian group under addition of homomorphisms. Let $f,g,h \in Hom(A,A)$. How can I see $(f+g)\circ h = (f\circ h) + (g\circ h)$?
Comment: @rschwieb: Surely you don't really mean "a monoid in $\newcommand{\C}{\mathcal{C}}\C$ is a one-object subcategory of $\C$"? That would make your answer's main claim false!  "A monoid in a category $\C$" usually assumes (in my experience) that $\C$ is a *monoidal* category — then a monoid is a 1-object $\C$-enriched category; explicitly, an object $A$ with maps $A \odot A \to A$, $\mathbb{I} \to A$ satisfying associativity/unit laws. Under this sense, it's true that a ring is a monoid in $\mathrm{AbGp}$, in the monoidal structure with $\otimes$ (not $\times$).
Comment: @fred The addition points to a complete discussion of what a monoid in a category is. Peter's right that we need at least what is known as a monoidal category.  It looks a little daunting but I assure you it is not as hard as it looks.  It is really just a rephrasing of the axioms to be in terms of arrows rather than elements.
Comment: @PeterLeFanuLumsdaine My impression as a cat. th. beginner is that it is perfectly fine to talk about monoids as categories without thinking of monoidal categories. (Mac Lane and Leinster do this in their books.) By the definition of a category, a single object category has an identity arrow and closure under composition of arrows that is associative; the monoid structure is evident. But perhaps Peter's explanation is the more useful mode in which monoidal structures often appear in practice?
Comment: @fred I do now remember the diagram definition of monoidal category he's referring to, and how the arrows basically encode all the distributivity. It may be more likely that over time I confounded that with simply "a one object subcategory" which is temptingly simple.  Maybe once I get my hands on a counterexample I will know one way or the other.  Thankfully my potential error is still only confined to the comments.
Comment: @PeterLeFanuLumsdaine You're right of course. I retract my comment, and revise to add a link to a complete discussion.
Answer: Instead of comparing groups and rings, one needs to compare monoids and rings:

An one-object category is the same as a monoid.
An one-object preadditive category is the same as a ring.

We see from this that the multiplication in a monoid needs to be compared to the multiplication in a ring.
The addition in rings has no corresponding structure in monoids.
Every ring has an underlying additive group, and some monoids happen to be groups. But that doesn't matter for the above observation:
the multiplication in a group is a special case of the multiplication in a monoid, and therefore needs to be compared to the multiplication in a ring (and not the addition).
The addition in rings has no corresponding structure in groups.
Answer: I don't know if this quite answers your question, but the point you focus on as your dissatisfaction seems to me to be a misunderstanding:

It seems weird to me that the group multiplication is the composition of arrows for groups as categories but for rings it is some binary map of arrows in the same hom-set.

Viewing a ring as a one-object pre-additive (i.e. $\newcommand{\AbGp}{\mathrm{AbGp}}\AbGp$-enriched) category, its multiplication is precisely composition of arrows, exactly like in groups-viewed-as-one-object categories.  The point is that in single-object categories (plain or enriched), composition is exactly "some binary map of arrows in the same hom-set", $\newcommand{\C}{\mathcal{C}}\C(*,*) \otimes \C(*,*) \to \C(*,*)$.
So I'd say the picture you set out is indeed a very good way to look at them:

monoids (so in particular, groups) are 1-object categories;
rings are 1-object categories enriched in $(\AbGp,\otimes)$ (aka monoids in $(\AbGp,\otimes)$);
the forgetful functor $(\AbGp,\otimes) \to (\mathrm{Set},\times)$ is lax monoidal; so it sends monoids in $(\AbGp,\otimes)$ to monoids in $(\mathrm{Set},\times)$, sending a ring to its underlying multiplicative monoid.
Comment: @fred (1) General 1-object categories are *monoids* — they don't automatically have inverses — so rings and monoids are both instances of "1-object enriched categories", or equivalently "monoids in a monoidal category"; then groups are a subclass of ordinary monoids.  (2) Yes: If you define preadditive cats with "composition is bilinear" as in CWM, distributivity comes precisely from that bilinearity.  If you define them as "$(\mathrm{AbGp},\otimes)$-enriched cats", it comes from the fact that maps out of a tensor product correspond to bilinear maps.
Comment: That's helpful, thanks! To make sure I'm understanding properly, could you clarify two lingering concerns of mine? (1) Composition plays the role of multiplication for both rings and groups as 1-object categories but for rings each arrow need not have an inverse under composition while arrows forming a group do need an inverse. Do I have this correct? (I just want to make sure I'm interpreting your use of "exactly" properly.)
Comment: (2) My other concern is the ring distributivity. In Mac Lane's CWM (p.28 ed.2) I read that a preadditive category is defined s.t. $\mathcal{C}(a,b)$ is an additive abelian group for which composition is bilinear. I.e. for $f,f':a\rightarrow b$ and $g,g':b\rightarrow c$, $(g + g') \circ (f + f') = g\circ f + g\circ f' + g' \circ f + g' \circ f'$. The ring distributivity are therefore baked into the definition of a preadditive category, correct?