re-analysis of someone else's data: 5 treatments x 4 non-exclusive outcomes

Question: I am trying to re-analyze (ETA: used loosely; the original study performed no statistical analysis) some published biological data (below). They used 5 treatments and scored presence/absence of 4 non-exclusive and potentially non-independent outputs (A-D); an individual could have 1, 2, 3, or 4 of these responses, it's possible that having 1 precludes or predisposes an individual to other(s), and that may vary by treatment.
They did not provide raw scoring data, nor did they track replicates. Different treatments have different sample sizes.
I am trying to figure out whether their conclusions about the different treatments have merit but I'm not sure what statistical test(s) would be valid.
I typically use R but shouldn't need code. I'm just feeling stuck for how to handle these data.
(    A  , B ,  C ,  D  , total)
treatment1 369, 44, 10, 54, 434
treatment2 67, 37,  0, 13,  86
treatment3  62, 19, 8,  22, 76
treatment4  45, 13, 0,  5,  60
treatment5  40, 5,  0,  0,  45
Comment: Without a more detailed understanding of the data than I get from your description, its difficult to suggest a method of analysis. Can you say how many subjects are represented here? Exactly what does '369' mean? If it is a count of something, then please describe what was counted. Similarly, what do the $0$s in treatments 2, 4, and 5 mean? In what way are the numbers 369, 67, 62, 45, 50 related? What does the grand total $434 + \cdots + 45$ represent?
Comment: I guess my layout was unclear. For example, in treatment 1: 369 individuals had outcome phenotype A, 44 outcome B, 10 outcome C, 54 outcome D; there were 434 total individuals counted. (or, to answer the question you asked: 369, 67, 62, 45, 40 are the numbers of individuals in treatments 1-5 respectively that had outcome A).
Comment: If I understand what you mean by "What does the grand total 434+⋯+45 represent?" It would be the total number of individuals included in the experiment (701).
Comment: Thanks. I thought I understood. But then I expected the last entry in each line to be the total of the previous four entries. For example, this is the case for the last row. But not for other rows. Also, adding the 20 numbers I take to be cell counts, I get a total of 840 individuals not 701. // I supposed that the 20 counts would make a 5-by-4 contingency table, and that appropriate analysis would be a chi-sq test for indep, of treatments and phenotypes, but I can't go ahead with that until you explain the 'total` column. // Afraid 'non-exclusive' is explantation. Then don't see what to do.
Comment: That's exactly the thing I'm struggling with – an individual can have more than one A-D so the total number of individuals in each treatment =/= the sum of all the counts for that treatment. But in treatment 1, with 434 total individuals, 369 of them had outcome A, etc, so the total number of outcomes counted is 477 (369+44+10+54) So some individuals certainly have more than one outcome. There are 701 individuals total across all treatments (434+86+76+60+45=701). The chi-squared tests I am used to require mutually exclusive categories.
Answer: If you want to know whether treatments and phenotypes are independent
categorical variables, then consider the counts (exclusive of totals)
to make a contingency table, and do a chi-squared test of independence.
First, let's temporarily ignore the double counting and pretend that the
number of subjects per row is the total of the counts per row. That amounts
to pretending there were 840 subjects.
TABLE = matrix(c(396, 44, 10, 54,  67, 37, 0, 13,
                   62, 19, 8, 22,  45, 13, 0, 5,
                   40, 5, 0, 0), byrow=T, nrow = 5)
TABLE
     [,1] [,2] [,3] [,4]
[1,]  396   44   10   54
[2,]   67   37    0   13
[3,]   62   19    8   22
[4,]   45   13    0    5
[5,]   40    5    0    0

sum(TABLE)
[1] 840

The expected counts are $E_{ij} = R_iC_j/G,$ where $R_i, C_j, G$ are
corresponding row and column totals, and the grand total $G=840.$ Then under $H_0$ that row and column categories are independent, the chi-squared statistic is
$$Q = \sum_{i,j}\frac{(X_{ij}-E_{ij})^2}{E_{ij}} \stackrel{\text{aprx}}{\sim}
\mathsf{Chisq}(\nu = (r-1)(c-1)),$$
where the approximation depends of having "most" $E_{ij} > 5$ and all
of them larger than 3. Here $\nu = (5-1)(4-1) = 12.$
However, for our data, several of the $E_{ij}$ in the lower-right corner
are too small. In this case upon request, R will simulate the distribution
of $Q$ to give a reasonably accurate P-value. Omitting the false start with
a warning about $E_{ij}$'s being to small and moving directly to simulation,
we have a highly significant result with P-value near 0 (as small as the
number of iterations of the simulation permits). Without the simulation
the P-value is also very near 0, 1.478e-12.
[Notice that $\nu$ is missing from the output because the chi-squared distribution was not used.]
Results = chisq.test(TABLE, simulate.p=T);  Results

        Pearson's Chi-squared test with simulated p-value (based on 1e+05  replicates)

data:  TABLE
X-squared = 82.334, df = NA, p-value = 1e-05

However, this result is questionable because of the multiple counting, so we
need to try to adjust for multiple counting in some way.
Below I propose a few possible methods of adjustment. I have not encountered
this problem before, so maybe someone can find a better method than any of them.
Adjust statistic Q: Suppose for the moment that every count $X_{i,j}$
were doubled. Then every $E_{i,j}$ is doubled and so is the statistic $Q.$
Thus, a 'cure' for this systematic doubling of counts would be to halve $Q$
before finding the P-value based on $\nu$ degrees of freedom. In our case according to your Comment,
we have sporadic multiple counts that we might view as inflating $Q$ by
a factor of $c = 840/701 \approx 1.2.$ We could use an adjusted statistic
$Q^\prime = Q/c = 82.334/1.12 = 73.51,$ and the corresponding P-value would again be very nearly 0 (7.02 e-11).
1 - pchisq(73.51, 12)
[1] 7.021073e-11

An objection to this method of adjustment is that our multiple counts are not
uniformly spread across the table. For example, there are no multiple counts
for Treatment 5, while Treatment 1 has $504 - 434 = 70$ multiple counts.
Its strong point is that it does not interfere with overall proportions of counts.
Adjust treatment counts by "exclusion."  Sometimes when crucial data values are missing,
one uses 'imputation' to estimate them. Here we have the opposite problem, we
have unwanted extra counts. We might seek a rational way to exclude them.
rowSums(TABLE)
[1] 504 117 111  63  45

Scaling method: A simple way to make an adjustment would be just to scale each row. However, due to rounding, the adjusted row total may be slightly incorrect
(below for Treatment 1, 436 instead of 435).
t1.over = TABLE[1,];  t1.over;  sum(t1.over)
[1] 396  44  10  54
[1] 504
t1.adj = round(t1.over*435/504); t1.adj; sum(t1.adj)
[1] 342  38   9  47
[1] 436

Then re-scale other the other rows with overcounts.
Random method: Alternatively, we might imagine original researchers looking at an individual in Treatment 1 and
trying to decide whether to classify it as phenotype A or B. Unable to decide
for sure, they just count it as both. So we are left with counts
t1.over = c(369, 44, 10, 54) totaling 504, while the number of subjects
in Treatment 1 is known to be 435.
We could use the sample function in R
to 'decide' for them on a probability basis.
t1.s = sample(1:4, 435, repl=T, prob=TABLE[1,])
table(t1.s)
t1.s
  1   2   3   4
346  40   8  41
t1.radj = as.numeric(as.matrix(table(t1.s)));  t1.radj
[1] 346  40   8  41
sum(t1.radj)
[1] 435

This random method of exclusion might be done repeatedly in a simulation to see
how much P-values vary.
Comment: Thank you for the ideas! My concern is that the double counting in this case is not an unknown or intermediate but is truly both. Without replicates I'm not sure how to handle that fact. I'm not convinced that any of your suggested adjustments is exactly right for my particular case. I'll have to keep thinking about it and get back to you but I'm not a statistician... I'm a cookbook user.
Comment: The difficulties with the data are probably why the the researchers did not do a statistical analysis of their own. Unclear what you mean by 'not an unknown or intermediate'. I'm curious to know their 'conclusions' that you hope to verify in retrospect with your own analysis. An analysis using my last method repeatedly, akin to a bootstrap, may be the the most defensible possibility. If you are looking for a 'tried-and-true' cookbook method, I doubt there is one. I suspect your only entirely satisfactory course would be to re-do the experiment using a sound design.
Comment: They made gene expression reporter constructs deleting putative gene regulatory elements. They scored whether in each individual animal the reporter was expressed in each of 4 cell types (1 where it is normally expressed a 3 where it isn't), and claim that 2 of their constructs each show ectopic expression in 1 cell type each and the other 3 don't. I don't think they ruled out general mis-regulation rather than cell type-specific regulation because reporter incorporation is stochastic so the control reporter doesn't even show up in all the cells it "should".
Comment: It isn't central enough to my work to re-do it at this time but I wanted to know how seriously to take it as either way it would be useful context for my project. Thanks so much for thinking about it.
Comment: Basing further work on these data may make sense. Although the data are somewhat messy, they seem to give a strong hint treatments and phenotypes aren't indep. You can look at 'Pearson residuals' of a chi-squared test to spot the cells with the greatest contribution to the total $Q.$ If you understand _how_ treatments might influence phenotype distribution, you can check to see if the the large contributions occur in cells that make some sense according to your understanding. So informed, if you eventually investigate further, you can design an experiment that might confirm your suspicions.


prove that $\Lambda(S)$ is closed for every closed set $S \subset X$

Question: Let $X$ and $Y$ be Banach spaces and let $\Lambda : X \to Y$ be a bounded linear operator. Is it true that if $\Lambda$ is surjective and hence open, then $\Lambda(S)$ is closed for every closed set $S \subset X$ ?
$\textbf{Is the following a correct answer? if yes I wondered why we need the openness of $\Lambda$?}$
We have to show that  any sequence $(x_n,y_n)$ in $\mathcal{G}(\Lambda (S)) = \{(x,y) : x\in S \ \ \& \ \ \ \Lambda x=y \in Y\}$ converges to a point $(x,y)$ in $\mathcal{G}(\Lambda (S))$.
Since $S$ is a closed subset of $X$, it is complete. So let's take any cauchy sequence $(x_n, \Lambda x_n)$ in $\mathcal{G}(\Lambda (S))$. Since $S\times Y$ is Banach, and $\Lambda$ is surjective we have that $(x_n, \Lambda x_n) \to (x,y)$ in $S\times Y$ for some $y$. It remains to show that $\Lambda x=y$.
We have that by completeness of $S$, that $x_n \to x$ in $S$, and by the continuity of $\Lambda$, $\Lambda x_n \to \Lambda x$,  also we have that by definition of convergence $\Lambda x_n \to y$. So choose $N$ s.t. for all $n\geq N$, $\|\Lambda x_n - \Lambda x\|<\frac{\epsilon}{2}$ and $\|\Lambda x_n - y\|<\frac{\epsilon}{2}$. So
$$\|\Lambda x-y\|\le\|\Lambda x_n - \Lambda x\|+\|\Lambda x_n - y\|<\frac{\epsilon}{2}+\frac{\epsilon}{2} = \epsilon$$
so since $\mathcal{G}(\Lambda (S))$ is closed for any point in $S$,  then $\Lambda (S)$ is also closed.
Comment: How does closedness of $\mathcal G(\Lambda(S))$ imply closedness of $\Lambda (S)$?
Comment: so may be I got it wrong. I thought that when  the graph is closed then the image of the operator as is part of the graph is also closed.  Can you provide me an example (also is possible geometrically) that this is no always true?
Comment: That is what I have done in my answer. $\mathcal G(\Lambda(S))$ is closed in that example but $\Lambda (S)$ is not closed.
Answer: Let $X=\mathbb  R^{2}$ and $Y=\mathbb R$. Let $\Lambda (x,y)=x$ and $S=\{(x,y)\in \mathbb R^{2}: xy=1\}$. Then the hypothesis is satisfied and $\Lambda (S)=\mathbb R \setminus \{0\}$ which is not closed.
Comment: @stat It appears that you are not happy with the answer. Do you want more details?


How to preform an activity at the android lock screen?

Question: I would like to perform an activity at the key guard screen, but the only way I can find to do so is to disable the key guard, do my activity, then re enable the key guard.  This would be ok if it didn't cause the key guard screen to flash off and on (disable makes it disappear then re enable brings it back).  I am wondering how I can perform an activity with it still being displayed?  Kinda like how when the alarm goes off it shows a pop-up which is displayed over the key-guard screen?
Answer: You can check out Alarm clock source on Android git, maybe that will help, also Music player does something similar to what you mentioned. So my advice is download snapshots of Music Player app and AlarmClock app from android git, and search those files for KeyGuardManager.
https://android.googlesource.com/platform/packages/apps/Music/+/master
https://android.googlesource.com/platform/packages/apps/AlarmClock/+/master


What interval is from G♭ to A♯ (same octave)?

Question: It is a number 2 interval, but being 4 semitones, it is beyond an augmented second. So, what it is?
Comment: related: https://music.stackexchange.com/questions/15430/are-doubly-augmented-and-doubly-diminished-intervals-practical
Answer: The interval from any G (flat / sharp / neutral) to any A (flat / sharp / neutral) (in the same octave) is always a second. In your case, since the G is flat and the A is sharp, you have a doubly augmented second.
Of course, this interval is sonically equivalent to a major third.
Comment: "Of course, this interval is sonically equivalent to a major third." This is only true in equal temperament.
Comment: Has anyone ever personally seen this used? I have not. I'd be curious to see an example. I cannot think of a theoretical justification for using it, either. I'd be interested in an explanation for that as well.
Comment: @fferri: On [this site](https://harmonia.illiacsoftware.com/support?article=206) I've found `++2` but I don't know how common and universally accepted it is.
Comment: @HeatherS. See the question that Tim H posted as a comment to the original post: https://music.stackexchange.com/questions/15430/are-doubly-augmented-and-doubly-diminished-intervals-practical
Comment: is there a symbol for that? `AA2` perhaps?
Comment: @fferri: In practice it occurs very rarely, and I haven't yet seen a symbol for it.
Comment: @HeatherS. In less tonal (or maybe bi-tonal) music voice leading may provide a reasonable explanation. However, I agree it is contrived.
Comment: @MattL.  The guys at that site may have riffed on the standard "+" symbol for augmented interval :-) .  I'll note that Dolmetsch does not list any double-anything symbol.
Comment: @HeatherS. I found a surprising example of this kind of enharmonic relations in surprisingly early music: have a look at bar 75 and 102 of violin sonata K 481 (Mozart).


Dockerized react app doesn't work on localhost

Question: I have the following Dockerfile for the server:
FROM node:18.13.0
WORKDIR /app
COPY package*.json .
RUN npm install
COPY . .
EXPOSE 8080
CMD [ "npm", "run", "start"]

and for the client:
FROM node:18.13.0
WORKDIR /app
COPY package*.json .
RUN npm install
COPY . .
EXPOSE 3000
CMD ["npm", "start"]

and the docker-compose.yml
version: "3.8"

services:
  server:
    build: ./server
    container_name: e_commerce_server_container
    ports:
      - 8080:8080
    volumes:
      - ./server:/app
      - /app/node_modules

  client:
    build: ./client
    container_name: e_commerce_client_container
    ports:
      - 3000:3000
    volumes:
      - ./client:/app
      - /app/node_modules
    # stdin_open: true
    # tty: true

when I run docker-compose up --build and try to access the server on localhost:8080 is working, but for localhost:3000 to see my react app is not working.
What should I do? I've tried a lot of methods but noting happened.
Comment: "not working" in what way? blank page? connection refused?
Comment: I get the "This page isn't working localhost didn't send any data.
ERR_EMPTY_RESPONSE", and in the terminal looks something like
e_commerce_client_container  |   vite v2.9.15 dev server running at: Local: http://localhost:3000/
Answer: I fixed the issue. It looks like it was from vite, I should add the "vite --host 0.0.0.0" and now it works


knockout select tag value mysticism

Question: Knockout don't create value attribute if it's text representing equal option text. So selected value mystically transformed to the string type:
<select data-bind="foreach: opts, value: val">
    <option data-bind="text: name, value: id"></option>
</select>

var vm = {
    opts: [{id: 1, name: 'a'}, {id: 2, name: '2'}],
    val: ko.observable(1)
};
vm.val.subscribe(function (x) {
    console.log(x, typeof x);
});
ko.applyBindings(vm);

test it in jsFiddle
How to can I fix this behavior?
Answer: The options binding will preserve your types. For example:
<select data-bind="value: val, options: opts, optionsText: 'name', optionsValue: 'id'"></select>

Outputs:
2 number
1 number

Comment: Thank you. But unfortunatly, I can't use this method, because I have complex logic of disabling and styling options.

knockout select tag value mysticism


Universal formula for direction angle of a vector for any quadrant

Question: I was studying how to find the angle between the (positive) $X$-axis and a position vector for a program I'm making. And I came to the conclusion that there is no universal formula for that. If I use the following formula: $$\theta = \tan^{-1}\left(\frac yx\right)$$ I have to add a particular offset depending on the quadrant where the vector is.
So I was wondering if there was another formula, another way to think about this problem, where the formula wouldn't have to change depending on the quadrant. It would be the universal formula for finding the angle. Google search didn't do any good.
Also is there a universal formula (in the sense mentioned above) for finding the angle between any two position vectors without having to offset the angle in any way?
I'm sorry for my bad English and my poor knowledge in Mathematics.
Answer: You are clearly seeking for the two-argument inverse tangent function $\left(x,y\right)\mapsto\arctan\left(x,y\right)$.
Its main difference from the plain old $\tan^{-1}$ is precisely that it takes the quadrant of the point $\left(x,y\right)$ in $\mathbb{R}^2$ into account, which is lost when the usual one-argument arctangent function version is used like you first intended due to the identities $$\frac xy=\frac{-x}{-y}$$ and $$\frac{-x}y=\frac x{-y}$$ which arise when the usual one-argument inverse tangent function is used like follows: $$\arctan\left(x/y\right).$$
To mitigate the problem, instead use the two-argument version $\arctan\left(x,y\right)$, which in most languages is known as arctan2, atan2 or something similar.
Comment: A word of warning about how to read this answer: most people use $x$ as the coordinate in the direction of the $x$ axis, so that the vector with $x=1$ and $y=0$ should have angle $0$ with the $x$ axis. In most implementations, `atan2(1,0)` will produce $\frac\pi2,$ a right angle. To get the angle with the $x$ axis, in most implementations you give the $y$ coordinate first and the $x$ coordinate second; `atan2(0,1)` produces the value $0,$ as desired. I usually document this as: the angle is `atan2(y,x)`.
Comment: Thank you @dbanet Is there any way to universalize this function like the one above, because it seems that this function depends on the quadrant too: cos(theta)= dotProduct(a, b) / ||a||  * ||b|| (to find angle between to vectors a,b)?
Comment: Excuse me @user3071028, I'm not following how does the angle between vectors depend on their position relative to the basis. Which function are you trying to build?
Comment: I don't know how to explain this, but I used this function to calculate angle between some unit vector(that represented direction of a line) and (1, 0) in order to rotate text to have the same rotation as the line. And when testing, the text was rotated correctly only for lines that pointed in certain range of directions, and for the others i had to add offset to the angle to get the right angle. So that made me think that this had something to do with the quadrant the vectors are in. Sorry if this makes no sense.
Comment: @user3071028 $(1,0)$ is the unit vector on the $x$ axis. So if there is some other vector $(a,b)$ and you need to know the angle $\theta$ such that the vector $(1,0)$, when rotated by $\theta$ in positive (anticlockwise) direction, lands on the direction of $(a,b)$, then $\theta$ is just the angle between the positive direction of $x$ axis and $(a,b)$. Hence, use $\theta=\arctan(b,a)$.
Answer: There are many, but here is one.
If $x=y=0$ then $\theta$ is undefined.
Then
\begin{equation}
\theta=\begin{cases}
       \frac{y}{|y|}\arccos\left(\frac{x}{\sqrt{x^2+y^2}}\right)\text{ if }y\ne 0\\
       \frac{\pi}{2}\left(\frac{x}{|x|}-1\right)\text{ if }x\ne0\text{ and }y=0
       \end{cases}
\end{equation}
This will give a result $-\pi<\theta\le \pi$.
Answer: A formula for the angle of a vector from 0 to 2pi.
f(x,y)=pi()-pi()/2*(1+sign(x))*(1-sign(y^2))
     -pi()/4*(2+sign(x))*sign(y)

     -sgn(x*y)*atan((abs(x)-abs(y))/(abs(x)+abs(y)))

The formula gives the angle for any value of x and y.
For x=y=0 the result is undefined.


Testing parallel trends in staggered difference-in-differences

Question: When creating the time dummies, what is the value for the control group? Always $0$? Also, I standardized the time dimension ($t=1$ in the treatment year, $t=0$ is the year before the treatment, $t=-1$ is two years before the treatment, $t=2$ is one year after the treatment, etc.). But again: how do I use the standardized time dimension for the control group? Stata also does not seem to accept that the time dimension goes to negative values. Is there a way to work around this but still capture time $0$ as the base year and then $2$ years before and then $2$ years after (i.e., leads and lags)?
Comment: Welcome. How many time periods do you have? Do you want to saturate the model with a full series of lead and lag indicators, or just two periods before and two periods after?
Comment: Hi Thomas, thank you for reaching out! And sorry for the brevity of the question, I initially wanted to write it under a post referring to a similar topic, was not aware that was not allowed. Anyway, to your questions; even though my data set includes time horizon 2000-2019,  I believe I read somewhere that too many time periods can weaker the model? hence I was thinking about including "only" 2-3 leads and 2-3 lags.
Comment: How long is treatment in place? Or should I say, when does treatment start for the early-adopter entities?
Comment: Earliest treatment is in year 2004 then in 2010, 2014, 2015 (4 treatment years in total)
Answer:
When creating the time dummies, what is the value for the control group? Always 0?

Yes.
The timing of the intervention isn't standardized. In fact, the onset of treatment is staggered over time. That is, some entities start early, while others start late. Some treatments may even reverse. In this setting, the "generalized" difference-in-differences estimator must be used. The treatment dummy used to estimate your treatment effect should 'turn on' (i.e., switch from 0 to 1) if a unit is treated and only during the precise policy adoption years, 0 otherwise. Units never espousing the treatment should be coded as 0 for the entire observation period. To be clear, it should equal 0 from 2000–2019.

But again: how do I use the standardized time dimension for the control group?

You don't.
Instantiating a variable delineating pre- versus post-treatment is inappropriate in this setting. The term "post-treatment" is defined differently across units. Because of this, we cannot assume control units would have received treatment in a particular year.

Stata also does not seem to accept that the time dimension goes to negative values.

It appears you created a variable denoting each unit's time to event, where each unit is $t$ periods relative to the last pre-treatment year (i.e., $t = 0$). I can see how this breeds confusion especially if you have a subset of non-adopter units. In my opinion, I only recommend this approach in settings where all entities enter into treatment. The variable denoting the relative periods is allowed to enter the model as a series of time dummies reflecting the observed onset of treatment. Stata cannot handle negative values assigned to a categorical variable. Adding a numeric constant to the variable is a cheap trick to get around it. Suppose I add 10 to the sequence of integers in the first line. The last pre-treatment year is 10.
<code>-3 -2 -1  0  1  2  3
 7  8  9 10 11 12 13
</code>
We can't normalize the time dimension around some event where a suitable counterfactual exists. Oftentimes we don't know when the subset of non-adopters would have entered into treatment.
A couple of visuals might help. The data frame that follows shows 3 units observed over 20 years. Appended is a variable $R_{it}$ which delineates the relative periods before and after treatment. The variable $T_{it}$ is the treatment dummy. Note how it doesn't demarcate a specific treatment group. It simply 'turns on' (i.e., switches from 0 to 1) once a particular treated entity enters into treatment, 0 otherwise. As indicated in your post, treatment starts when $t = 1$, and the 'time to event' varies across units. The year of exposure is as follows:

Unit 1 adopts in 2011
Unit 2 adopts in 2012
Unit 3 adopts in 2013

Note how all units undergo treatment. The "treatment effect" is identified solely based upon variation in treatment timing. To estimate leads and lags of treatment, software is certainly capable of 'dummying out' the individual relative period dummies. Again, I only recommend standardizing the time dimension in this way in settings where all entities eventually become treated.
$$
\begin{array}{ccc}
unit & time & R_{it} & T_{it} \\
\hline
 1 & 2000 & -K       & 0 \\
 1 & \vdots & \vdots & \vdots \\
 1 & 2008 & -2       & 0 \\
 1 & 2009 & -1       & 0 \\
 1 & 2010 & \ \ \ 0  & 0 \\
 1 & 2011 & \ \ \ 1  & 1 \\
 1 & 2012 & \ \ \ 2  & 1 \\
 1 & 2013 & \ \ \ 3  & 1 \\
 1 & 2014 & \ \ \ 4  & 1 \\
 1 & 2015 & \ \ \ 5  & 1 \\
 1 & \vdots & \vdots & \vdots \\
 1 & 2019 & \ \ \ L  & 1 \\
\hline
 2 & 2000 & -K       & 0 \\
 2 & \vdots & \vdots & \vdots \\
 2 & 2008 & -3       & 0 \\
 2 & 2009 & -2       & 0 \\
 2 & 2010 & -1       & 0 \\
 2 & 2011 & \ \ \ 0  & 0 \\
 2 & 2012 & \ \ \ 1  & 1 \\
 2 & 2013 & \ \ \ 2  & 1 \\
 2 & 2014 & \ \ \ 3  & 1 \\
 2 & 2015 & \ \ \ 4  & 1 \\
 2 & \vdots & \vdots & \vdots \\
 2 & 2019 & \ \ \ L  & 1 \\
\hline
 3 & 2000 & -K       & 0 \\
 3 & \vdots & \vdots & \vdots \\
 3 & 2008 & -4       & 0 \\
 3 & 2009 & -3       & 0 \\
 3 & 2010 & -2       & 0 \\
 3 & 2011 & -1       & 0 \\
 3 & 2012 & \ \ \ 0  & 0 \\
 3 & 2013 & \ \ \ 1  & 1 \\
 3 & 2014 & \ \ \ 2  & 1 \\
 3 & 2015 & \ \ \ 3  & 1 \\
 3 & \vdots & \vdots & \vdots \\
 3 & 2019 & \ \ \ L  & 1 \\
\end{array}
$$
The previous data frame allows for any number of $K$ leads and $L$ lags. The number of relative pre- and post-periods will vary across units. For instance, the late-adopters will invariably have more relative pre-policy periods than those treated much earlier.
But suppose you have a subset of non-adopters. What is their relative time to treatment? By "non-adopters" I mean units never exposed to treatment in any time period (i.e., control units). In short, we can't impute their time to exposure because the intervention is switching on, and possibly off, at different times within different entities. In my opinion, there's no computationally easy method to estimate lead and/or lag indicators in settings where the timing of treatment isn't well-defined and where a subset of units were left untreated.
The next example is a staggered design, but now the treatment only affects a subset of units. In other words, some units never receive treatment. The precise start date for the non-adopters is largely unknown. In scenarios where we want to instantiate a lead or a lag, then I recommend instantiating the relative lead and/or lag indicators manually.
In the example data frame that follows we have 3 units observed over 20 years. Here, unit 1 was precluded from receiving treatment. This is the "always 0" control unit(s) mentioned earlier. Note the column of values under $T_{it}$; unit 1 equals 0 throughout the entire observation period. The treatment histories are as follows:

Unit 1 is not eligible for the treatment (i.e., control unit)
Unit 2 adopts in 2012
Unit 3 adopts in 2013

Now suppose you want to estimate some leads and lags. The indicator $d^{+1}_{it}$ is the immediate effect of treatment. It is the first adoption year. Depending upon the discipline, some may label $d^{0}_{it}$ as the immediate effect of treatment, with each subsequent period referred to as lagged effects. It doesn't really matter which period is labeled as the initial adoption year. Just be consistent! I also binned the final lag (i.e., $d^{+\bar{3}}_{it}$). It simply switches on in the third period after treatment and stays on. You don't have to do this in practice, but you should think about how to model the endpoint. You could also estimate separate effects in all post-periods. It's up to you!
$$
\begin{array}{ccc}
unit & time & T_{it} & d^{-2}_{it} & d^{-1}_{it} & d^{0}_{it} & d^{+1}_{it} & d^{+2}_{it} & d^{+\bar{3}}_{it} \\
\hline
 1 & 2000 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 1 & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots\\
 1 & 2008 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 1 & 2009 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 1 & 2010 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 1 & 2011 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 1 & 2012 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 1 & 2013 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 1 & 2014 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 1 & 2015 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 1 & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\
 1 & 2019 & 0 & 0 & 0 & 0 & 0 & & 0 \\
\hline
 2 & 2000 & 0        & 0 & 0 & 0 & 0 & 0 & 0\\
 2 & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\
 2 & 2008 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 2 & 2009 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\
 2 & 2010 & 0 & 0 & 1 & 0 & 0 & 0 & 0 \\
 2 & 2011 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\
 2 & 2012 & 1 & 0 & 0 & 0 & 1 & 0 & 0 \\
 2 & 2013 & 1 & 0 & 0 & 0 & 0 & 1 & 0 \\
 2 & 2014 & 1 & 0 & 0 & 0 & 0 & 0 & 1 \\
 2 & 2015 & 1 & 0 & 0 & 0 & 0 & 0 & 1 \\
 2 & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\
 2 & 2019 & 1 & 0 & 0 & 0 & 0 & 0 & 1 \\
\hline
 3 & 2000 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 3 & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\
 3 & 2008 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
 3 & 2009 & 0  & 0 & 0 & 0 & 0 & 0 & 0 \\
 3 & 2010 & 0  & 1 & 0 & 0 & 0 & 0 & 0 \\
 3 & 2011 & 0  & 0 & 1 & 0 & 0 & 0 & 0 \\
 3 & 2012 & 0  & 0 & 0 & 1 & 0 & 0 & 0 \\
 3 & 2013 & 1  & 0 & 0 & 0 & 1 & 0 & 0 \\
 3 & 2014 & 1  & 0 & 0 & 0 & 0 & 1 & 0 \\
 3 & 2015 & 1  & 0 & 0 & 0 & 0 & 0 & 1 \\
 3 & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\
 3 & 2019 & 1  & 0 & 0 & 0 & 0 & 0 & 1 \\
\end{array}
$$
As an aside, it's also quite common to observe the individual post-treatment effects expressed as follows: $d^{-1}_{it}, d^{-2}_{it}, d^{-3}_{it},..., d^{-L}_{it}$, where the negative integers represent the lags. Again, just keep your notation consistent!
Remember you could proceed in many ways. You could estimate a finite number of leads and/or lags, or trace out the full dynamics of exposure by saturating the model. If the treatment is transient, then incorporating a full series of time indicators is very demanding. Moreover, some units may opt out of the treatment over time. As you move farther and farther away from the first adoption year, you may have less and less data to estimate your lags. This isn't usually a problem in practice, but you may find that the fifth, sixth, and seventh lags are less precisely estimated.
Comment: Thank you so much Thomas for taking the time! Your answer addressed exactly the practical use of the leads and lags that I was not able to find anywhere else. I will definitely follow your comments
Comment: No problem. If this helped please consider giving it a check. If not you can always follow up with me here.


Generating a matrix so that correlations amongst columns are close to pre-defined values

Question: I want to generate a matrix of values such that the correlations (e.g., Pearsons correlations) are close to a pre-defined set of values (e.g., corrs = [0.5, 0.7, -0.3, 0.9, 0.2]). For simplicity, we can assume we're only concerned with the correlation of pairs of columns.
For example, I may want a matrix with 4 columns and 100 rows, and the correlations between columns should be close to the following:
CORR(c1, c2) = corrs[0]
CORR(c1, c3) = corrs[1]
CORR(c1, c4) = corrs[2]
CORR(c2, c3) = corrs[3]
CORR(c2, c4) = corrs[4]
CORR(c3, c4) = corrs[5]

Depending on the parameters specified (matrix size, correlations), a solution may not exist, but for reasonable parameters, it seems like some algorithm should be able to come up with a solution.
Is there any areas of research that can help me accomplish this?  It seems like something related to graphs and/or some iterative algorithm might be what I need, but I'm struggling with how to represent the problem mathematically or in a graph, and my knowledge of these areas is pretty limited.
Comment: It seems like a multivariate normal distribution (or multivariate Student's $t$ distribution, if you like) checks these boxes. Are there any additional requirements?
Comment: This might be what you're looking for http://www.econometricsbysimulation.com/2014/02/easily-generate-correlated-variables.html
Comment: Thanks @Sycorax, that makes sense. I just adjust my covariance matrix accordingly.  I'll try playing with this
Answer: Following @Sycorax's recommendation, I used a multivariate normal distribution to generate my desired dataset.
I set the means and variances for my columns to 0 and 1 respectively, thus the covariance matrix is populated with just the desired Pearson correlations (since cov(X,Y) = corr(X, Y) * std_dev_X * std_dev_Y).
Caveat to myself and others: Apparently the covariance matrix should be positive-semidefinite for proper sampling to occur, so the input set of correlations will need to be arranged or modified to create a positive-semidefinite covariance matrix.


How do SOMs reduce dimensionality of data?

Question: This is a problem with which I have been grappling with for days. From my research on self-organizing maps, I know that a common feature of self organizing maps is to reduce the dimensionality of data. For example, if you had a 3x3 SOM, and an input space consisting of 50 10-Dimensional vectors, the SOM would reduce this to 50 2-Dimensional vectors. If I am creating my own SOM, where is this data? Please excuse me if my question too vague or broad. The reference vectors that are attached to each neuron in the SOM are the same dimension as the input space. The input space itself does not get reduced in dimensionality. So where is the reduced-dimension data? In other words, what data structure in relation to the self organizing map contains this data? My only guess is that this data could be found in the location of each node in the self organizing map.
Thanks!
Answer: The SOM grid is a 2-d manifold or topological space onto which each observation in the 10-d space is mapped via its similarity with the prototypes (code book vectors) for each cell in the SOM grid.
The SOM grid is non-linear in the full dimensional space; the "grid" is warped to more-closely fit the input data during training. However, the key point in terms of dimension reduction is that distances can be measured in the topological space of the grid - i.e. the 2 dimensions - instead of the full $m$-dimensions. (Where $m$ is the number of variables.)
Simply, the SOM is a mapping of the $m$-dimensions onto the 2-d SOM grid.
Comment: So just for clarification...say that the 27th 10-Dimensional vector  in the input space just happened to match up with the reference vector in the SOM grid at location (2, 2) in the grid. So, would the reduced dimension data be (2, 2) in this case?
Comment: @diwgan32: Yes, your comment is correct.
Comment: basically you input your m-dimensional pattern, and the location where som grid is activated/most active gives you the 2-d coordinates.
Comment: +1. To clarify for the OP, the reduction to 2D is not stored anywhere, it's implicit in the fact that the prototypes have 2-D neighbors prototypes and the SOM algorithm adjusts the prototypes in 10-D space based on their neighbors in 2-D space.
Comment: So suppose I wanted to reduce the dimensions of the 50 10-D data. To do this, I would first train the SOM (by presenting each vector at random and finding the best matching unit). Then, after training, I would find the closest neuron in the SOM to each data piece by calculating the Euclidean distance from the original data piece to the neuron's reference vector. The location of the neuron in the SOM would give the the compressed data. However, since the vectors attached to each neuron are randomized in the beginning, wont the best matching unit that you find in the end also be random?
Comment: @diwgan32 No, the random presentation doesn't change the locations of the 50 data points in the 10-d space. The SOM learns the data. Once learning is finished, the assigned grid for each observation is the one with the prototype (or codebook vector) that each observation is most similar. The information about where the points are in 10-d space or to which prototype it projects is not randomised in any sense.
Comment: I would argue that, actually, the information about which prototype "wins" for a given point in the 10-d training space is, in fact, randomized, based on the initial values for the prototypes (often created at random), and the order of presentation of the training examples. If you want to use a SOM to do dimensionality reduction in the way suggested here (by using the topological location of an input vector's "winner") then you'd need to be careful to always use the same SOM when mapping different data vectors to lower-dimensional space.
Answer:
Consider your 2-dimensional SOM artificial neurons units as aiming to
have values equal to those of your high-dimensional data. It attains
this through the learning process- where a sample (a row of your data)
is taken from the data, compared for similarity with each of the units
on the map. The unit that comes closer in terms of similarity to the
sample becomes the winner of that sample. Then to effect the
"learning", the value on the unit is adjusted to be closer to that of
the sample it has just won. Units near this winner have their values
adjusted too, but with smaller amounts than that of the winner. That
adjustments of the units values make the learning to occur. The
process is repeated for all samples from the data. At the end of the
learning process, you have a learned SOM with units that come closer
to resembling your data values.
Note that your data values remain intact, they were only read and
assisted in conducting the learning process.
Now concentrate on the values carried by each unit at the end of the
learning process. Each unit may have won several samples from the data
and they are now "clustered" around it. That is, several samples from
your data can be comfortably represented by one unit of the SOM - this
brings in the dimensionality reduction idea! Your 10-dimensinal data
can now be visualized as 2-dimensional since similar data in the
original dataset can be respresented by one unit of the SOM.
For a deeper understanding check out here


Relatively fast approximations to the marginal likelihood?

Question: Let $\theta\in{\mathbb R}^d$ be a multidimensional parameters, where $d$ can be large (e.g. $d=100$ or more). What approximations can I use for the marginal likelihood:
$$\int f(x\mid \theta)\pi(\theta)d\theta.$$
The likelihood $f(x\mid \theta)$ does not seem to be log-concave, so, I am not confident of using a Laplace approximation. Also, the model is slightly complex, so I am trying to find general alternative methods that may work beyond simple linear regression models, and preferably without having to sample from the posterior. My main objective is to perform Bayesian model selection among a number of candidates.
Answer: Variational Inference is known to produce a lower bound on the log marginal likelihood: $$ \log p(x) = \log \int p(x|\theta) p(\theta) d\theta \ge \int q(\theta|x) \log \frac{p(x|\theta) p(\theta)}{q(\theta|x)} d\theta $$Where $q(\theta|x)$ is an arbitrary distribution supposed to approximate the posterior $p(\theta|x)$. The integral in the lower bound is likely to be intractable but can be estimated with Monte Carlo.
The gap between the log marginal likelihood and the lower bound is known to be equal to the KL divergence between $q(\theta|x)$ and $p(\theta|x)$. Since the lower bound is always smaller than the $\log p(x)$, one can maximize the lower bound w.r.t. $q$'s parameters to learn good posterior approximation.
However, unless the true posterior can be represented by $q(\theta|x)$, the gap will be non-zero. To tighten the lower bound further, one can use multiple samples (the so called IWAE lower bound):
$$
\log p(x) \ge \int \dots \int \prod_{m=1}^M q(\theta_m|x) \log \left( \frac{1}{M} \sum_{m=1}^M \frac{p(x, \theta_m)}{q(\theta_m|x)} \right) d\theta_1 \dots d\theta_M
$$
This lower bound obviously coincides with the previous one for $M = 1$. Moreover, the more samples $M$ you use, the (strictly) tighter the bound becomes. In particular, in the limit of infinite samples ($M = \infty$) the gap vanishes.
Finally, you can combine the multisample lower bound with randomized truncation to obtain an unbiased estimate of $\log p(x)$. For more details see the SUMO paper. In a related work Goda and Ishikawa propose an improved estimator that guarantees unbiasedness and finite variance under certain conditions.

An alternative approach is use coupled Markov Chain to perform Unbiased MCMC (this falls under posterior sampling category).


Convergence of $\sum_{n=1}^{\infty}\frac{1}{(a+n)^z}$ for $a>0,\operatorname{Re}z>1$

Question: I need to prove that for $a>0$ and $\operatorname{Re}z>1$, $\sum_{n=1}^{\infty}\frac{1}{(a+n)^z}$ converges.
Here's where I am.
The ratio test gives
$\lim |\frac{(a+n)^z}{(a+n+1)^z}|=\lim |\frac{a+n}{a+n+1}|^z=1$, which tells me nothing.
Hints only, please.
Comment: In general, $|k^z|\neq |k|^z$. The left-hand side is real, the right-hand side need not be.
Comment: Why not try the Cauchy condensation test initially? This approach is quite standard on Riemann zeta and can be used on Hurwitz zeta with some manipulation.
Comment: Your limit in question does not hold, since the limit as $n \to \infty$ is actually $1$. In general, for a rational function $P(n)/Q(n)$, when $\deg(P) = \deg(Q)$, the limit as $n\to \infty$ is the ratio of the leading coefficients of $P,Q$.
Comment: What makes you think this limit is zero? Anyway, do you know what is $|(a+n)^z|$?
Comment: Yes, the limit is actually 1. Edited.
Comment: I figure I'm not allowed to use the condensation test because it isn't in the textbook.
Answer: Hint:
$$
 \left| \frac{1}{(a+n)^z}\right| = \frac{1}{|(a+n)^z|} = \frac{1}{(a+n)^{\operatorname{Re}z}}
\le \frac{1}{n^{\operatorname{Re}z}} \, .
$$
Comment: ... and then see https://math.stackexchange.com/q/29450/42969.


Css font not changing correctly or aligning to the center

Question: I am trying to change the font-family of my tab and align the text to the center using "font-family" and "text-align" but it isn't working. Here is my code and the image. As you can see the font is not Roboto and the text "Chat" is not aligned properly.

              Chat

Answer: For the font, the font-family declaration works if you try an already-included font, like Arial or Times New Roman. I don't believe Roboto is a default font that you can just include.  Have you imported it from somewhere like Google Fonts?
For the text centering: I'm not entirely sure what you mean by "not aligned properly," but it is centered in your example.  Both the "Chat" text and the SVG icon are part of the centered element (the  tag), so it's those two elements together being centered within the box.  (The same distance extends from the "t" to the top and from the edge of the speech bubble to the bottom.)  Depending on how you actually want it to be aligned, you may need to target the text or image and align them individually.
Comment: by centered I mean within the tab. It's touching the bottom of the tab and not in the center. As in its touching the right hand side of the screen.
Comment: Thanks for clarifying - so the issue then is that `text-align` aligns elements on the horizontal axis.  Since your `div` has been rotated 90 degrees, `text-align` is still operating on what it "thinks" is the horizontal axis, but is actually the vertical axis visually.

The easiest way to fix this will likely be using flexbox.  Try adding the following three rules to `#olark_tab a`:

`display: flex;`
`align-items: center;`
`justify-content: center;`

You won't need the `text-align: center` rule anymore as `justify-content: center` will be doing that instead.
Comment: thanks for the response. After adding that three rules the icon and text shift down towards the bottom of the tab and not centered. Just to clarify again, I'm wanted for the icon and text to be centered all the way around the tab, so right, left, up and down.
Comment: Is `display: block` still included in the list of style rules?  If so, it may be overwriting `display: flex`.  There should be only one `display` rule set to flex.  And if by "centered all the way around" do you mean having the same space on each side? Check this Fiddle - is this more what you want to do? [https://jsfiddle.net/apLftmq8/](https://jsfiddle.net/apLftmq8/)
Comment: Yes that's exactly what I want
Comment: you were right, the block was overriding flex
Comment: Great, glad that's working.  If you still need help getting the spacing right, I've updated the Fiddle to show how to do that.  Only look at the CSS box and see the areas I've commented on.  I would recommend specifically removing any `width` or `height` rule and adding `padding` instead, which will give you a uniform space around your text.  This also shows the Roboto font import rule at the top.  [https://jsfiddle.net/6tb52f1a/](https://jsfiddle.net/6tb52f1a/)
Answer: The Roboto font is not supported by css and is only available through google. I have added the code to get the font from google and display it in on the tab.
HTML

              Chat

Answer: make your image smaller

div#olark_tab {
  position: fixed;
  right: 0;
  bottom: 50%;
  z-index: 5000;
}

#olark_tab div {
  width: 150px;
  margin-right: -59px;
  transform: rotate(-90deg);
}
/* changes */
#olark_tab img{
  width: 50%;/* or :75%; make it the much you want */
}
/* changes */
#olark_tab a {
  /*Edit these to change the look of your tab*/
  background-color: #DCDCDC;
  color: #1EAFE6;
  font-family: 'Roboto';
  font-style: normal;
  font-weight: bold;
  font-size: 20px;
  line-height: 20px;
  height: 20px;
  padding: 6px;
  display: block;
  text-decoration: none;
  text-align: center;
  width: auto;
  border-top-right-radius: 9px;
  border-top-left-radius: 9px;
  border-top-style: none;
  border-top-width: 0;
}

#olark_tab a:hover {
  background-color: white;
}

       Chat

Comment: adding that did not work, the text is still stuck to the bottom and not in the middle
Comment: man the problem was in your image, if you change the size it will be fine


Intervals when making a tone row - what to keep in mind?

Question: For writing 12 tone or serial music, one generally makes a tone row, and then uses that for constructing a melody. I want to know what would be the maximum range of intervals between any 2 consecutive notes of the tone row that will be feasible to sing/play?
For example, can I move from a B down by a major 7th to a C (if B and C were two successive notes of the tone row)? Will this jump be ok for a singer or any instrument (especially wind instruments) to play?
Comment: You're not locked to one octave when using the row. You can use B - C as a minor second. Rows are about pitch classes, you can get creative with the intervals.
Comment: One of the first twelve tone row pieces I learned at the university was Dallapiccola's _Die Sonne Kommt,_ and the very first interval is a major seventh, B down to C (in the transcription for high voice).  Not a problem with a little practice. https://vdocuments.mx/dallapiccola-die-sonne-kommt.html
Comment: General rule of thumb - keep it singable. Any really big jump needs a note after to be closer to the first note.
Comment: Following up on what Your Uncle Bob says, I've seen 12-tone serialist piano music change which hand plays successive tone row notes (e.g. right hand plays Note 6, left hand plays Note 7).
Comment: This is not a well posed question.  Feasible to sing or play is dependent on the instrument and/or vocalist.  Withe a 4 octave range there aren't too many limits.  And the difficulty in getting from one note to another is not, imo, dependent on the instrument as much as the instrumentalist.  Singing a Maj 7 should be east for a well trained singer.  I can think of some classic musicals that have that interval in solos.
Comment: Ok, that makes sense @Tim. Could you explain with an example please, to make it more clear?
Comment: So, @YourUncleBob, would the major 7th leap be alright to do? Also, I want to know what is the maximum interval between any 2 successive notes in general, when you write a melody.
Comment: Well, that is really a general instrument question, not specific to 12-tone music. And it not only depends on the instrument, but also on the abilities of the player. I don't think there's really a theoretical reason to limit the intervals in a melody; the listener may or may not consider consecutive notes to form a melody, depending on many details in the orchestration. Look at klangfarbenmelodie, where a melody is shared between instruments; whether or not that is one melody is quite intangible.
Comment: You should probably split this question in two: one about the technical aspects of playing large intervals (to which the answer will probably be different for every instrument) and one about what the largest interval is before a melody is no longer perceived as such.
Answer: As the comments suggest, there is no real maximum range between two pitches of the tone row.
This is because, more accurately stated, these are pitch classes, not just pitches. In other words, we are dealing with octave equivalence: if the next member of your tone row is a B, it can be a B in any octave. Thus, if you come from a C, this B could be a major seventh above it, a minor second below it, or any other B. In short, the octave equivalence makes these intervals largely moot.
As for the technical demands of vocalists and wind players, serial composers were pretty demanding. In the opening measures of the first movement of his Op. 21 Symphony, Webern asks one horn player to move a descending major fourteenth (a major seventh plus an octave), another horn player to move this same interval ascending, and then the clarinet to return to this interval in its descending form.
It seems that the intervallic demands you speak of are less the demands regarding intervallic leaps and more the demands of the ranges of the instruments themselves.


Can I still use an overfitted model with high test accuracy?

Question: Below is the training statistics output from training a Keras/TF model. You can see val_accuracy peaks at Epoch 4 with 0.6633. After that accuracy(train) continues to go up but val_accuracy becomes worse/lower. I generated two models: model_4 from 4 epoch runs, model_10 from 10 epoch runs. Then the test dataset is applied to them to compute test_accuracy. I got 0.7040 for model_4 and 0.7152 for model_10. Apparently from the training statistics, model_10 is overfitting the training set. However its test_accuracy doesn't deteriorate. Can I still choose model_10 for the production deployment? What is the implication of using an overfitted model?
<code>Epoch 1/10
# 703/703 [==============================] - 821s 1s/step - loss: 1.0799 - accuracy: 0.6451 - val_loss: 1.0686 - val_accuracy: 0.6306
# Epoch 2/10
# 703/703 [==============================] - 821s 1s/step - loss: 0.8158 - accuracy: 0.7277 - val_loss: 0.9982 - val_accuracy: 0.6488
# Epoch 3/10
# 703/703 [==============================] - 820s 1s/step - loss: 0.6541 - accuracy: 0.7798 - val_loss: 0.9794 - val_accuracy: 0.6616
Epoch 4/10
703/703 [==============================] - 821s 1s/step - loss: 0.5162 - accuracy: 0.8340 - val_loss: 0.9962 - val_accuracy: 0.6633
Epoch 5/10
703/703 [==============================] - 821s 1s/step - loss: 0.3926 - accuracy: 0.8840 - val_loss: 1.0285 - val_accuracy: 0.6587
Epoch 6/10
703/703 [==============================] - 821s 1s/step - loss: 0.2804 - accuracy: 0.9298 - val_loss: 1.0956 - val_accuracy: 0.6616
Epoch 7/10
703/703 [==============================] - 821s 1s/step - loss: 0.1864 - accuracy: 0.9629 - val_loss: 1.1609 - val_accuracy: 0.6584
Epoch 8/10
703/703 [==============================] - 820s 1s/step - loss: 0.1155 - accuracy: 0.9825 - val_loss: 1.2688 - val_accuracy: 0.6502
Epoch 9/10
703/703 [==============================] - 820s 1s/step - loss: 0.0698 - accuracy: 0.9923 - val_loss: 1.3471 - val_accuracy: 0.6493
Epoch 10/10
703/703 [==============================] - 820s 1s/step - loss: 0.0445 - accuracy: 0.9952 - val_loss: 1.4566 - val_accuracy: 0.6488

```
</code>
Answer: That means your test set mimics the training set better than your validation set. Considering the size of your training dataset (i.e. 703) this is possible. And, that's not guaranteed in production. What if you were using your test set as validation, and your validation set as test? In that case, the situation would be the opposite, and you'd have chosen the model with 4 epochs (ignoring the other epochs for the sake of simplicity).
The implication is that overfitting means high variance. So, in production, if your model is tested with a lot of samples, model 10 will have more variance in its decisions than model 4. Considering the gap between training and validation performances, both cases have some degree of overfitting, but situation in model 10 seems to be more serious.
Apart from choosing amongst these two, you may also use cross-validation to either select the best epoch, or generate $k$ different models and average/vote their predictions. Considering the size of your data, this approach seems possible and will increase the stability of your predictions.


Multinomial-dirichlet with fractional counts

Question: Suppose a lepidopterologist wants to estimate the relative proportions of three different species of butterfly. They go out into the field and count $N$ butterflies and record the number of each species $(N_A,N_B,N_C)$. This is a Bayesian lepidopterologist, however, and so they want to use the Dirichlet-Multinomial conjugate pair to obtain a Dirichlet posterior on the probabilities $(p_A,p_B,p_C)$ that an observed butterfly is of each species. If the counts are integer and the prior hyperparameters of the Dirichlet are $(\alpha_A,\alpha_B,\alpha_C)$, then the posterior hyperparameters are simply $(\alpha_A+N_A,\alpha_B+N_B,\alpha_C+N_C)$.
However, the lepidopterologist is not sure of the species of some of the butterflies and so assigns a probability $(p_A^i,p_B^i,p_C^i)$ to each individual butterfly $i$ that it is of each species. This probability is calculated quantitatively and under the assumption that all species are equally likely. These probabilities capture genuine uncertainty in the species determination of a butterly, perhaps because some of members of species A can look like members of species B. These probabilities are also unbiased, i.e. if we observed an infinite number of butterflies with $p_A^i = 0.54$, then 54% of those butterflies would be in species A.
Naively, the posterior hyperparameters would be of the same form but with $N_A$ now being the sum of the probabilities that each butterfly is of species $A$, etc.
Is this correct? If incorrect, are there limits where this is a good approximation?
TLDR: Can we generalise the dirichlet-multinomial conjugate prior-likelihood to fractional count data?
Edit for clarity and to add example data:
We observe $N=4$ butterflies and record that their probability of being a member of each species is:
Butterfly 1 - $p_A = 0.34,p_B = 0.42,p_C = 0.24$
Butterfly 2 - $p_A = 0.14,p_B = 0.23,p_C = 0.63$
Butterfly 3 - $p_A = 0.97,p_B = 0.01,p_C = 0.02$
Butterfly 4 - $p_A = 0.00,p_B = 0.67,p_C = 0.33$
Comment: ]@diagonalisable did you get to a best solution for this? I have a similar problem in my hands. In my case is a bit more complicate as they are likelihoods rather than probabilities for the categorical uncertainty on obdservations.
Comment: How excitly does the data look like? Could you give us example? Hard-coding the guesses as data to be passed to the model does not sound like the best approach.
Comment: The data would be of the form $(p_A^i,p_B^i,p_C^i)$ for each butterfly $i$, with $p_A^i+p_B^i+p_C^i = 1$.
Comment: So you know only the probabilities and total count?
Comment: Yes. We know the total number $N$ and the probabilities for each butterfly.
Comment: How did you end up solving this problem?
Answer: I would change the problem slightly:
$i$ = butterfly observation
$j$ = butterfly class (A,B,C)
$X_{i,j}$ = subjective probability that a butterfly is in class $j$ that is recorded by the observer $i$
$\rho$ = vector of true proportion of butterflies in the observation area of each class
$\alpha_i$ vector of dirichlet parameters for butterfly $i$
Assume that $X_i \sim dirichlet(\alpha_i)$ so that $\alpha_i$ describes the underlying distribution of the recorded observations.  Each butterfly might have a different $\alpha_i$ describing the difficulty of classifying that particular butterfly.  In other words, if multiple people tried to classify that same butterfly and they all recorded probabilities of classes then their observations would form a dirichlet distribution.  We only see one set of probabilities per butterfly.
Now, to simplify, assume that all the $\alpha_i$ are coming from the same underlying distribution which is centered on the $\rho$.
As a Bayesian, you can put a prior on $\alpha$ and then update using MCMC $X_i \sim dirichlet(\alpha)$
As a frequentist, you can get the dirichlet distribution estimates directly.
$\bar{X_j} = \alpha_j / \sum \alpha_j$
$s^2_{X_1} = \frac{z_1 (1 - z_1)}{1 + \sum \alpha_j}$ where $z_1 = \alpha_1 / \sum \alpha_j$
Note:  you only need the second equation for one of the classes $j$.
Comment: Thanks for the answer... however is kinda oversimplying the problem  as stated and I cannot apply it to my own that is similar to the one posted. After looking for a solution around I guess the answer is that there is no a convenient conjugate prior in this scenario. Then perhaps what remains to asked that is not asked in the original question, is how to address this in approximate yet efficient way instead of the "brute force" MCMC.
Comment: I think the "approximate, yet efficient" way to address this is in the frequentist equations I put at the end of the answer.  I have used these equations in a very similar situation where I asked experts to quantify class probabilities and then I needed to combine across experts.  The only difference is that you truly are showing each expert a different butterfly while in my case, I showed each expert the same object and then elicited the class probabilities.
Comment: The notation above doesn't seem entirely correct. Given observations from each expert $(X_{i}\ \text{for}\ i=1,...,N )$, how would one uncover the parameters $\alpha_j$ needed to fit the final Dirichlet distribution?
Comment: I added some clarifying notation.  Thanks for the comment.


Change the order of a column

Question: I am struggling to order column in my dataset. The way I want to reorder can be shown in the following example:
I have the following data:
dd = data.frame("AssignedA"=1:2, "AverageA"=1, "AssignedB"=1, "AverageB"=1,
  "AssignedC"=1, "AverageC"=1, "ValueA"=1, "ValueB"=1, "ValueC"=1)

For an easy overview, I would like the columns to be ordered in the following way:

Assigned A - Assigned B - Assigned C - Average A - Average B - Average C-  Value A - Value B - Value C

How to do this? I couldn't find the solution in questions posted earlier.
Answer: Just keep things simple and use seq to create the order of the columns. First, (and you should do this in your question), create some dummy data:
dd = data.frame("AssignedA"=1:2, "AverageA"=1, "AssignedB"=1, "AverageB"=1,
  "AssignedC"=1, "AverageC"=1, "ValueA"=1, "ValueB"=1, "ValueC"=1)

then just reorder as usual:
dd[,c(seq(1,6,2), seq(2,6,2), 7:9)]

Here, the command in the [] specifies the order:
R> c(seq(1,6,2), seq(2,6,2), 7:9)
[1] 1 3 5 2 4 6 7 8 9

Comment: That really is simple.. Works perfectly!

Change the order of a column


database having only one table with one column and one value size > over 200mb

Question: I have been playing with database, I imported over million columns, played with functions at the end i selected only one value, deleted the rest of it and my database is the size over 200mb. I am doing it in sqlite3. How to reduce the size of it?
Answer: Your database is probably still reserving the space from your previous records.
This is essentially the same question:
Why does clearing an SQLite database not reduce its size?
The accepted answer:
When an object (table, index, trigger, or view) is dropped from the database, it leaves behind empty space. This empty space will be reused the next time new information is added to the database. But in the meantime, the database file might be larger than strictly necessary. Also, frequent inserts, updates, and deletes can cause the information in the database to become fragmented - scrattered out all across the database file rather than clustered together in one place.
The VACUUM command cleans the main database by copying its contents to a temporary database file and reloading the original database file from the copy. This eliminates free pages, aligns table data to be contiguous, and otherwise cleans up the database file structure.
Edit: you may want to research the pragma command 'auto_vacuum' if you expect to be doing this regularly. It will keep your file size down but has some pros and cons. In a production environment It is best to reserve more space than you need, as this reduces the risk of running out of disk space on the server.


Duvida orientação a objeto + ORM java

Question: Estou querendo armazenar os dados do meu programa via ORM e estou separando bonitinho cada objeto.
Meu objeto Usuario por exemplo, tem seus atributos basicos e também é composto por outros objetos como Endereco,Contato e Acesso.
Cada objeto tem um atributo ID, para identificação e associação...
Minha duvida é a seguinte, no metodo construtor de cada classe(Usuario,Endereco,Contato e Acesso) Eu devo passar o id correspondente ou apenas os demais atributos? Porque eu acho, que de repente o hibernate possa atribuir o ID correto.
Public Class Cliente
   private long Id;
   private String Nome;

   Public Cliente(Long id,String Nome){}
   'ou
   Public Cliente(String Nome){}
   '?

Answer: Endereço, Contato e Acesso são classes? Porque se sim, muito provavelmente temos um pequeno problema aqui. Sim, pequeno, sem ironias.
Essas informações são agregadas à um usuário – elas pertencem a ele e são compiladas conceitualmente como (meta) dados – em outras palavras, sem necessidade de projetá-las de classes para objetos; apenas as defina como variáveis de um usuário.
Sobre a herança, o id é associativo. Um usuário é uma coisa, um cliente é outra. Um usuário pode ser um cliente, bem como um representante ou administrador. O id(entificador) compartilhado entre as entidades (usuario e cliente) deve, contextualmente, ser o mesmo – um cliente é um usuário, mas não necessariamente o contrário.
Por fim, transformando isso em um idioma técnico:
+-------+---------+---------+
| #     | Usuário | Cliente |
+-------+---------+---------+
| Id    | 1       | 1       |
+-------+---------+---------+

Esse id vai pertencer sempre ao usuário, sem exceções. Lembre-se: clientes são usuários; administradores são usuários; banidos são usuários. Independente da circunstância, organismos vivos são usuários.
Para finalizar, você deve definir as características em comum de todos os usuários na dita cuja classe; nos filhos (administrador, cliente, representante, etc.) você adiciona suas particularidades MAS "pega emprestado" o id do pai.
Answer: O Hibernate Trata ids relacionais como Objeto entao vc deve declarar com O Objeto Endereco ,Contato e Acesso dentro da classe Cliente se a sua tabela cliente tiver os campos idendereco, idcontato e idacesso e eles devem estar indexados com chaves estrangeiras.

Duvida orientação a objeto + ORM java


Why does this factoring method only work for quadratics of the form $x^2 + bx + c$?

Question: $x^2-7x+10$
Because the first item is $1x^2$ (the quadratic term has coefficient $1$), we can use the shortcut. The second sign is a plus sign, so we want the sum of $10$'s factors to be $-7$; $10=(-2)(-5)$
$x^2-7x+10=(x-2)(x-5)$
If the last item has a lot of factors, we can use a shortcut, provided the first term is $1x^2$ (the quadratic term has coefficient $1$)
Please explain why you can only use the shortcut if the first term is $1x^2$
Comment: What "shortcut" are you referring to?
Comment: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comment: It may help to try an example where you know what the factored answer is but the coefficient of $x^2$ isn't 1. Take two binomials like $(x+3)(3x+5)$ and multiply them together, then see what your shortcut suggests.
Comment: If you have an coefficient for #x^2$ other than $1$, the rule changes and guessing the correct values becomes difficult.
Answer: In cases where the coefficient of $x^2$ isn't $1$, you can still convert the expression into the standard form so that you can apply your formula.
For example, let's say that you want to factorize $2x^2-10x+12$
You just have to factor out the coefficient of $x^2$ to get $2(x^2-5x+6)$
Now, the sum of $6$'s factors should be $-5$; $6=(-2)(-3)$
$2x^2-10x+12=2(x-2)(x-3)$
Now, I'll explain the reason why this works.
When you have a general quadratic expression $x^2+bx+c,\ $you can factorize it into something in the form $(x+\alpha)(x+\beta)$.
$(x+\alpha)(x+\beta)=x^2+\alpha x+\beta x+\alpha\beta=x^2+(\alpha+\beta)x+\alpha\beta$
Comparing $x^2+(\alpha+\beta)x+\alpha\beta$ and $x^2+bx+c$, we see that $(\alpha+\beta)=b$ and $\alpha\beta=c$
$\alpha$ and $\beta$ are just factors of $c$, so the problem boils down to finding two factors of $c$, such that their sum $\alpha+\beta$ equals $b$.
Now if $x^2$ didn't have a coefficient of $1$, then we would have to compare the expressions $x^2+(\alpha+\beta)x+\alpha\beta$ and $ax^2+bx+c$.
As you can see, no matter what value we choose for $\alpha$ and $\beta$ (the factors of $c$), the two expressions cannot be made the same (because $x^2$ is always unequal to $ax^2$ when $a\ne1$).


is there a way to overwrite the views Sort criteria?

Question: i have overwritten a field's output, and want to use it as the Sort criteria. so i must overwrite the Sort criteria?  is there a way to overwrite the views module Sort criteria?
the sort criteria is:
table votingapi_vote, the field content_id,which is $node->nid, the vaule=1. i want to to sort it by the number counts of the vaule. desc. HOW TO do it?
if i want to alter the  sort criteria in the template.php, is there a way to get it?
Answer: Sure, just use hook_views_query_alter().
You can intercept the view's query object and make any changes you need to it, including (but not limited to) adding/removing sort criteria.
Better WHERE's for your Drupal forums with hook_views_query_alter and Using hook_views_query_alter to change your views sort order give some examples of how to alter a view query.
Comment: i don't know how to use this hook,could you give me an example. thank you.
Comment: Alex, i have updated the question. could you tell me how to use the hook_views_query_alter(). thank you.
Comment: Just an FYI, none of your links are working...


Determine if the collection {$(a_n : n \in N) : |a_n| = 1$ for all n} is countable.

Question: Determine if the collection {$(a_n : n \in N) : |a_n| = 1$ for all n} is countable.
Comment: Where are the elements $a_n$ from? If they're from $\mathbb{N}$, then $a_n = 1$ for all $n$, and there's exactly one such sequence. If they're from $\mathbb{R}$, there are a lot more sequences, corresponding to choices of $\pm 1$. If they're from $\mathbb{C}$, there are vastly more sequences.
Comment: I think it is from R, the question did not really specify
Answer: If $a_n \in \mathbb{C}$, then the set $\{ a_n \}$ may not be countable.  Take the set of all irrational roots of unity, that is, let your set be $\{e^{2 \pi i n \alpha}: \alpha \notin \mathbb{Q}\}$.
Answer: If $a_n \in \mathbb{R}$ for every $n$, then the mapping $(a_n)\mapsto\{n\in\mathbb N\mid a_n=1\}$ is a bijection between the set of interest and the set $2^\mathbb N$ of subsets of $\mathbb N$. One knows that, for every set $A$, the sets $A$ and $2^A$ are not equipotent, hence the set of interest is uncountable.
Answer: If $a_n \in \mathbb{R}$ for every $n$, then this set is the collection of sequences whose terms are $\pm 1$. This can be corresponded with the collection of sequences in $0$ and $1$, which can then (via binary expansion) be corresponded with the unit interval.


how do you paste only plain text on react-quill?

Question: I using React quill as a custom text editor in my web application and I cannot figure out how to make it only paste plain text when users try to paste. Formatting causes a lot of issues with my code.
How can i override the paste event and access the clipboard text to paste just plain text?
Please help!!
  class Editor extends React.Component {
      constructor(props) {
        super(props)
        this.quillRef = null;      // Quill instance
        this.reactQuillRef = null; // ReactQuill component
        this.titleRef = React.createRef();
        this.languageRef = React.createRef();
        this.state = {
          title: "",
          language: "",
        };
      }

  modules = {
    toolbar: [
      ['bold', 'italic', 'underline'],
      ['clean']
    ],
  }

  componentDidMount() {
    this.attachQuillRefs()
  }

  componentDidUpdate() {
    this.attachQuillRefs()
  }

  validate() {
    //validate text
  }

  keyDown = (event) => {
    //here is where i tried to override paste event
    event.preventDefault();
    console.log(event);
    if(event.keyCode === 86){
        console.log(event.clip);
    }
  }

  onFieldChange = (event) => {
    const { target: { name, value } } = event;
    this.setState({ [name]: value });
  }

  attachQuillRefs = () => {
    if (typeof this.reactQuillRef.getEditor !== 'function') return;
    this.quillRef = this.reactQuillRef.getEditor();
  }

  addReading = () => {
    send text

  }

  render() {
    return (

        <input name="title" ref={this.titleRef} type="text" placeholder="Title" onChange={this.onFieldChange} />
        <select name="language" id="classLanguage" ref={this.languageRef} onChange={this.onFieldChange} className="modalInput">
          {languages.map(language => (
            <option key={language.value} value={language.value}>
              {language.display}
            </option>
          ))}
        </select>
        <ReactQuill
          ref={(el) => { this.reactQuillRef = el }}
          modules={this.modules}
          onKeyDown={(e) => this.keyDown(e)}
          placeholder={this.props.placeholder}
          theme={'snow'} />
        <button onClick={this.addReading}>Add Reading</button>

    )
  }
}

Comment: I don't think this is possible out of the box
Comment: oh man, do you have any other rich editors for react that might offer this functionality. at this point i am just thinking about using sanitize html to clean the html tags before submission.
Comment: Did you manage to find a workaround?
Answer: Hi i didn't want to paste any image type and i fixed like this:
const handlePaste = (e: ClipboardEvent) => {
    const data = e.clipboardData?.items[0]

    if(typeof data === 'undefined') return
    if (data.type === 'image/png' || data.type === 'image/jpeg' || data.type === 'image/jpg') {

      e.preventDefault();
      toast.error('Lütfen resim yapıştırmayınız.');
    }
  };

  React.useEffect(() => {
    const quill = quillRef.current.getEditor();
    quill.root.addEventListener('paste', handlePaste);

    return () => {
      quill.root.removeEventListener('paste', handlePaste);
    };
  }, []);

maybe you will try this code for only text/plain mime type


Moment Generating Function of a nonlinear transformation of an exponential random variable

Question: Let $\tau$ be an exponential random variable, with parameter $\lambda$.  Let
$$
V = \delta^\tau
$$
where $0 < \delta <1$.  Sorry if this notation seems strange, but it is what I am using, I will get confused if I switch it around!  What I want is the moment generating function of $V$.  The density function of $\tau$ is $f(\tau) = \lambda \exp(-\lambda \tau)$.  The change of variable to $V$ is
$$
\delta^\tau = V \\
\tau \log \delta = \log V \\
\tau = \frac{\log V}{\log \delta}
$$
And $\frac{\partial \tau }{\partial V } = \frac{1}{V \log \delta}$.  So the density of $V$ is
$$
g(v) = f(\frac{\log v}{\log \delta}) \frac{\partial \tau }{\partial V }\\
 =  \lambda \exp(-\lambda \frac{\log v}{\log \delta})\frac{1}{v \log \delta}
$$
The moment generating function of $V$ is therefore
$$
MGF_V(t) = \int_0^1 \exp(t v) \lambda \exp(-\lambda \frac{\log v}{\log \delta})\frac{1}{v \log \delta} dv
$$
Integrating this is Mathematica gives me a seriously complicated function of Gamma functions:
$$
-\frac{\lambda}{\log \delta} (-t)^{\frac{\lambda}{\log \delta}} (\Gamma(-\frac{\lambda}{\log \delta})-\Gamma(\frac{-\lambda}{\log \delta} ,-t))
$$
where the second gamma is using Mathematica's implementation of the Incomplete Gamma function. I need to do further things with this function, including use it numerically in either Matlab or Mathematica, and have serious trouble working with it.  Compare this to the pleasantly simply moment generating function of $\tau$
$$
MGF_{\tau}(t) = \int_0^{\infty} \exp{(t \tau)} \lambda \exp(-\lambda \tau) d \tau = \frac{\lambda}{\lambda - t}
$$
To have a sense of what I'm doing, think of $\tau$ as wait time, and $\delta$ as a discount factor.  So if I have wait time 0 I get value 1, but if I have to wait two periods I get value $\delta^2$.  Everything that is going on with $V$ is completely pinned down by $\tau$, so I feel like I should be able to work in terms of $\tau$, but I know I can't just perform my change of variable after calculating the MGF. Can I?

For further information, there are a variety of ways to express the $MGF_V(t)$, depending on how you feed it to Mathematica.  One additional way is
$$
-\frac{\lambda}{\log \delta} ( - ExpIntegralE[1+\frac{\lambda}{\log \delta},-t]+(-t)^{\frac{\lambda}{\log \delta}} \Gamma[-A])
$$
where ExpIntegralE is Mathematica's implementation of the Exponential Integral Function:
$$ExpIntegralE[n,t] = \int \exp(-z t) \frac{1}{t^n} dt$$
I've done a lot of messing about with various forms of this expression, but none are really helpful.  I need, eventually, to take derivatives of this thing and or calculate it numerically.  Eventually I have a calculus of variations problem that uses it!  All these implementations (including Matlab's, which are annoyingly different that Mathematica's) seem to have real instability issues, at least the way I want to use them.
This is cross posted from Mathematics, didn't get much interest there.
Comment: (1) For the record, the correct formula for $\text{ExpIntegralE}(n,t)$ is $\int_1^\infty \exp(-t x) /x^n dx$. Differentiating it with respect to $t$ is easy, because it just produces another exponential integral. Differentiation with respect to $n$ is very messy. (2) It's not evident where the "instability issues" are: could you be more specific about that? (3) The Gamma function representation looks particularly nice and is straightforward to differentiate with respect to $t$.
Comment: Are you sure that you are calculating the definite integral (0 to infty), and not just the indefinite integral? I don't get the incomplete Gamma function in there.
Comment: I didn't specify the limits of integration, but for $\tau$ it is 0 to inf, and after the change of variable for $v$ it is from 0 to 1 (with a sign change), since the density is 0 elsewhere.  When I calculate the MGF at particular values, it does have the properties I expect, so I think it is correct.  The problems come when I want to invert it, take derivatives, and do all this other stuff later in my problem, and it becomes a nightmare.
Comment: You are right, I missed the fact that the limits become 0 to 1 after the variable change.
Comment: I don't know why Mathematica comes up with the forms it does.  If I simplify the density, letting $A=\frac{\lambda}{\log \delta}$, and reducing things like $Exp[-A Log[v]]= v^{-A}$, it will give the expression in the later part of my note.
Comment: The Legendre Transform of $\log MGF_V$ is the instantaneous cost function in an optimal control problem. So I need to differentiate it and invert it at a few steps, as well as integrate it over a path $v(t)$ to evaluate the cost of the path.  Doing this numerically in Matlab has been a huge headache because depending on the parameters ($\delta$,$\lambda$) numerically solving equations using them doesn't always give (what I believe to be) correct answers. I'm happy to be educated on alternate ways to express it!
Answer: Since $0 < \delta <1$, then $\log \delta <0$. So the density of $V$ is
$$g(v) = f\left(\frac{\log v}{\log \delta}\right)\cdot \left|\frac{\partial \tau }{\partial V }\right|
 =  \lambda \exp\left\{-\lambda \frac{\log v}{\log \delta}\right\}\frac{1}{v |\log \delta|} \\= \frac {\lambda}{|\log \delta|}\frac 1v\exp\left\{ \frac{\lambda}{|\log \delta|}\log v\right\}$$
Set $\alpha \equiv \frac{\lambda}{|\log \delta|}$. Then, manipulating,
$$g(v) = \alpha v^{\alpha-1}, \;\;v\in [0,1] $$
which is the density of a $\text{Beta}(\alpha,1)$ distribution.
The moment generating function is
$$MGF_V(\alpha,1,t) = 1  +\sum_{k=1}^{\infty} \left( \prod_{r=0}^{k-1} \frac{\alpha+r}{\alpha+r+1} \right) \frac{t^k}{k!}$$
and which, among other things provides a nice recursive formula
$$E[V^s] = \frac {\alpha +s-1}{\alpha+s}E[V^{s-1}]$$
Comment: +1 Very nice!  You have also incidentally provided another answer to http://stats.stackexchange.com/questions/114481 for the case $\lambda/\log(\delta)=-2$.  Note that the MGF is a [confluent hypergeometric function](http://en.wikipedia.org/wiki/Confluent_hypergeometric_function#Integral_representations), which makes its computation susceptible to a huge array of techniques.
Comment: Then perhaps, if you update your question in math.SE using the results from my answer, _and_ the "equation of the type..." you typed in your comment above, perhaps you will get some useful feedback there, adjusting perhaps also the tags of the question.
Comment: @whuber Thanks. Indeed, there lies the general case.
Comment: Thanks very much! I think this answer(and whuber's comment) will help a lot.  Other than typical equation solving approaches, any advice for solving an equation of the type: $\frac{MGF_V^{\prime}(t)}{MGF_V(t)} - s =0$ for $t$?  This was the sort of exercise that gives me trouble, numerically.  What I had been doing was using implementations of the $\Gamma$ function and then using either built in or home-cooked equation solving methods, having a lot of trouble as parameters and $s$ varied.
Comment: It appears that your interest in the MGF is not so much in its capacity as a generator of distribution moments (which happens when $t=0$), since you want to _solve_ for $t$. This apparently makes your problem much more mathematical than statistical, don't you think?
Comment: Well, certainly. As I said, I asked originally in math.stackexchange.com
Comment: Sure, I'll do that.


What is the easiest way to use python to read pureconfig config files created in scala?

Question: I have a scala config file created with the library pureconfig. I would like to write a wrapper in python to read and write this configfile (to give a better access to non scala users). Is the easiest way to parse the whole schema ?
My config file looks like:
tables = [
  {
    name = TABLE_1
    partition_strategy = "none"
    partitions = [
      {
        path = [
          "/shared/fall/raw/TABLE_1_2015.csv"
        ]
      }
    ]
  }
  {
    name = TABLE_2
    partition_strategy = "none"
    partitions = [
      {
        path = [
          "/shared/fall/raw_data/TABLE_2_2015.csv"
        ]
      }
    ]
  }
  {
    name = TABLE_3
    partition_strategy = "year"
    partition_column = "FLX_DIS_DTD"
    partitions = [
      {
        year = 2014
        path = [
          "/shared/fall/raw/2014_2016/PRODUCT/TABLE_3*2014.CSV",
          "/shared/fall/raw/2014_2016/PRODUCT/TABLE_3*2015.CSV",
        ]
      },
      {
        year = 2016
        path = [
          "/shared/fall/raw/2014_2016/PRODUCT/TABLE_3*2016.CSV",
          "/shared/fall/raw/2014_2016/PRODUCT/TABLE_3*2017.CSV",
        ]
      }

    ]
  }
]

Comment: It looks that config syntax is actually HOCON (https://github.com/lightbend/config/blob/master/HOCON.md). Perhaps you could try to use a hocon parser (e.g. https://github.com/chimpler/pyhocon)
Comment: Worked perfectly ! Thx !
Answer: According to @mjjaniec: This syntax appears to be HOCON. You can use a Hocon parser like PyHocon
This package also provides a conversion tool to convert from HOCON to the JSON, .properties and YAML formats.
usage: pyhocon [-h] [-i INPUT] [-o OUTPUT] [-f FORMAT] [-n INDENT] [-v]

optional arguments:
  -h, --help                 show this help message and exit
  -i INPUT, --input INPUT    input file
  -o OUTPUT, --output OUTPUT output file
  -c, --compact              compact format
  -f FORMAT, --format FORMAT output format: json, properties, yaml or hocon
  -n INDENT, --indent INDENT indentation step (default is 2)
  -v, --verbosity            increase output verbosity


What is a more efficient solution for this linear algebra problem?

Question:
In the above question , one way of approach is definitely multiplying the matrices and comparing the respective elements then solving the equations. But I am not able to figure out a more efficient solution?
Comment: Please write an *INFORMATIVE* title.  Yours is so vague it could apply to innumerable questions.
Answer: Your matrix equation can be written as $\vec{c} \times \vec{b} = \vec{a} - \vec{c}$, using the cross product.
The left-hand side is orthogonal to $\vec{b}$ (property of cross products), and $\vec{a}$ is orthogonal to $\vec{b}$, so dotting the equation with $\vec{b}$ says that $0 = \vec{b} \cdot \vec{c}$. So (B) is true.
Rewriting our equation as $\vec{c} + (\vec{c} \times \vec{b})= \vec{a}$ gives a decomposition of $\vec{a}$ as a sum of two orthogonal vectors. By the Pythagorean theorem then, $\lVert\vec{c}\rVert^2 + \lVert \vec{c} \times \vec{b}\rVert^2 = \lVert \vec a\rVert^2 = 11$, so $\lVert\vec{c}\rVert^2 \leq 11$ and so $\lVert\vec{c}\rVert \leq \sqrt{11}$. So (D) is true.
If $\vec{a}\cdot \vec{c} = 0$, then dotting both sides of the equation with $\vec{c}$ then gives that $\lVert\vec{c}\rVert^2=0$, so $\vec{c}=0$, but then $\vec{0} = \vec{c}\times \vec{b} = \vec{a} - \vec{c} = \vec{a}$, which is false because $\vec{a}\neq 0$. So (A) is false.
The condition $\vec a \cdot \vec b = 0$ expands to $3 + b_2 - b_3 = 0$, so $|b_3 - b_2| = 3$, and since $b_2b_3 > 0$, we must have one of these numbers have absolute value larger than $3$. But then $b_2^2 + b_3^2 > 9$, so $\lVert \vec b \rVert=\sqrt{1 + b_2^2 + b_3^2} > \sqrt{10}$, and so (C) is true.
Comment: You mean (C) is true for the last one? The original statement is whether the norm of $b$ is greater than $\sqrt{10}$
Comment: Edited, thanks.


How to add a text Autocomplete Widget programmatically

Question: Trying to include a text Autocomplete Widget in a compound field (more here) using the snipped below.. but the field does not autocomplete. What am I doing wrong?
  $element['contact_name'] = array(
    '#title' => t('Name'),
    '#type' => 'textfield',
    '#default_value' => $defaults['contact_name'],
    '#weight' => 2,
    '#size' => '100',
    '#max_length' => '255',
    '#widget_type' => 'autocomplete_widgets_flddata',
    '#autocomplete_match' => 'contains',
    '#autocomplete_case' => '0',
    '#widget_module' => 'autocomplete_widgets',
  );

Comment: If you have firebug enabled, can you investigate the "Net" tab, to see what url is called to fetch the autocomple-data? And can you look if the response (by Drupal) is actual JSON autocomplete data?
Comment: please add a tag indicating your core version. I'm assuming it is 6 since you mention CCK.
Answer: Ok, so if you're trying to do it through hook_form_alter, you have to read the documentation on altering CCK fields.
The short version: the CCK fields are processed after the form is built. If you want to change anything in them, you have to add a callback to $form['#after_build'] (but do not replace the array, just add to it). Then implement the function, change values in $form and return the $form.


So'ton abbreviation

Question: Why do some sports media and broadcasters (eg Sky Sports) routinely abbreviate Southampton FC to So'ton when they don't abbreviate other, longer club names? For example, take the PL fixtures from this weekend: So'ton v Tottenham on the one hand but Norwich City v Crystal Palace on the other.
Comment: Do you have any images demonstrating this? I'd be surprised if they abbreviated "Southampton" to "So'ton" but not "Norwich City" to e.g. "Norwich".
Comment: Now added as requested
Answer: If this were something that happened consistently on all platform, it would be safe to assume that it is an official abbreviation that the team prefers to use. However, in the given case, and judging from the way you have framed the question, this is not something that happens consistently, instead differs from platform to platform. Therefore, it is safe to assume that is is a "design decision" by the people who are running the platform under consideration.

So’ton abbreviation


Erro ao usar Câmera no Android 7.1

Question: estou com problemas em uma rotina para tirar fotos, rotina simples que no Android 4.1 a 6.0 não tem problema mais no 7 ao tentar tirar uma foto para o app, tentei de tudo há uma semana tentando, procurando por este problema em fóruns, mais to quase desistindo porque não vejo mais onde eu posso procurar e por isto ver se alguém já passou por este problema.
O Erro é este
<code>java.lang.RuntimeException: android.os.TransactionTooLargeException: data parcel size 1387072 bytes at
android.app.ActivityThread$StopInfo.run(ActivityThread.java:3781)
at android.os.Handler.handleCallback(Handler.java:751)
</code>
o código para a foto é igual ao que encontramos em tutoriais
<code>    private void takePicture() {
    Log.i(TAG, "takePicture()");
    FirebaseCrash.log(TAG + " takePicture()");

    Intent takePictureIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);

    if (takePictureIntent.resolveActivity(getActivity().getPackageManager()) != null) {

        arquivoFoto = getOutputMediaFile();

        if (arquivoFoto != null) {
            //Uri fotoURI = Uri.fromFile(arquivoFoto);
            Uri fotoURI = FileProvider.getUriForFile(getActivity(),
                    getActivity().getApplicationContext().getPackageName() + ".provider",
                    arquivoFoto);
            takePictureIntent.putExtra(MediaStore.EXTRA_OUTPUT, fotoURI);
            startActivityForResult(takePictureIntent, PICK_FROM_CAMERA);
        }
    }
    FirebaseCrash.log(TAG + " takePicture() - pickImage()");
    FirebaseCrash.log(TAG + " takePicture() - Fim");
}

private File getOutputMediaFile(){
    Log.i(TAG, "getOutputMediaFile()");

    String appName = Helper.getAppName(getActivity());

    File mediaStorageDir = new File(Environment.getExternalStoragePublicDirectory(
            Environment.DIRECTORY_PICTURES), appName);
    Log.i(TAG, "getOutputMediaFile() - pasta " + mediaStorageDir.toString());

    if (!mediaStorageDir.exists()){
        if (!mediaStorageDir.mkdirs()){
            return null;
        }
    }

    String timeStamp = UUID.randomUUID().toString();
    File imagem = new File(mediaStorageDir.getPath() + File.separator +
            timeStamp + ".jpg");
    mCurrentPhotoPath = imagem.getAbsolutePath();
    Log.i(TAG, "getOutputMediaFile() -  mCurrentPhotoPath: " + mCurrentPhotoPath);
    return imagem;
}

@Override
public void onActivityResult(int requestCode, int resultCode, Intent data) {
    super.onActivityResult(requestCode, resultCode, data);
    Log.i(TAG, "onActivityResult()");

    FirebaseCrash.log(TAG + " onActivityResult()");

    if (requestCode == PICK_FROM_CAMERA && resultCode == RESULT_OK) {
        salvarFoto();
    }
}
</code>
Answer: O erro ocorria quando usava a câmera mais ele não estava ali, a partir do Android 7 eles começaram a olhar o bundle, então onde tinha onSaveInstanceState eu comentei tudo.
<code>/*public void onSaveInstanceState(Bundle outState) {
   FirebaseCrash.log(TAG + " onSaveInstanceState()");

   outState.putSerializable("listFotos", (Serializable) fotos);
   super.onSaveInstanceState(outState);
}*/
</code>
minha activity tinha vários fragments e em dois deles e o da foto eu guardava dados ali.
Apos comentar o código os problema de Crash, TransactionTooLargeException: data parcel size.... não mais ocorreram.
Obrigado ao Igor Morais pela força.

Erro ao usar Câmera no Android 7.1


How to discover the URL for my organization in Dynamics 365 CRM (online) using Web Api

Question: I cannot connect Dynamics CRM online to discover organization URL using a Web API. The sample code from Microsoft is not working.
I am developing a console application, and want to connect to CRM-online using Web API(we are not using Organization Service since we would like to move to Web API), In the application, I need to discover all available organizations and then query the data from each organization. I am using the example from Microsoft, but it didn't work as expected. Here is sample code.
I followed the Quick Start guide in the Azure AD. using MSAL.NET.
Active Directory Dotnet Core Daemon
I tried both global discovery URL and data center discovery URL, same fault.
Here are my code using MASL.NET.
static void Main(string[] args)
{

    IConfidentialClientApplication app = ConfidentialClientApplicationBuilder.Create("3f4b24d8-61b4-47df-8efc-1232a72c8817")
        .WithClientSecret("of+Ozx_ARRX?ONj+QCzWXf84eTQABA17")
        .WithAuthority("https://login.microsoftonline.com/common/oauth2/authorize", false)
        //.WithAuthority("https://login.microsoftonline.com/3a984a19-7f55-4ea3-a422-2d8771067f87/oauth2/authorize", false)
        .Build();

    var authResult = app.AcquireTokenForClient(new String[] { "https://disco.crm5.dynamics.com/.default" }).ExecuteAsync().Result;
    //var authResult = app.AcquireTokenForClient(new String[] { "https://globaldisco.crm.dynamics.com/.default" }).ExecuteAsync().Result;

    //var authResult = app.AcquireTokenForClient(new String[] { "https://crm525842.crm5.dynamics.com/.default" }).ExecuteAsync().Result;

    HttpClient client = new HttpClient();
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", authResult.AccessToken);
    client.Timeout = new TimeSpan(0, 2, 0);
    client.BaseAddress = new Uri("https://disco.crm5.dynamics.com/");
    //client.BaseAddress = new Uri("https://crm525842.api.crm5.dynamics.com");

    HttpResponseMessage response = client.GetAsync("api/discovery/v9.1/Instances()", HttpCompletionOption.ResponseHeadersRead).Result;
    //HttpResponseMessage response = client.GetAsync("api/data/v9.1/WhoAmI()", HttpCompletionOption.ResponseHeadersRead).Result;

    if (response.IsSuccessStatusCode)
    {
       //Get the response content.
        string result = response.Content.ReadAsStringAsync().Result;
    }
    else
    {
        //401 occurred
        throw new Exception(response.ReasonPhrase);
    }
}

I can get the access token, I expect this work but the response is:
{StatusCode: 401, ReasonPhrase: 'Unauthorized', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  X-CrmServer: HK2CRMWOSDISW11
  Strict-Transport-Security: max-age=31536000; includeSubDomains
  Date: Mon, 05 Aug 2019 05:47:12 GMT
  Set-Cookie: crmf5cookie=!DTRY0s4VpQxjLnMqwoaZ9AdJzXkdU6Vfto/+0KmA1KJeDxXIK15WxYQUdDi6BwLXgIrzskdvtk0u7A==;secure; path=/
  Server: Microsoft-IIS/8.5
  WWW-Authenticate: Bearer error=invalid_token, error_description=Error during token validation!. Ticket id feef1832-8a12-4d3d-a426-099176943f35, authorization_uri=https://login.microsoftonline.com/common/oauth2/authorize, resource_id=https://disco.crm5.dynamics.com/
  Content-Length: 1293
  Content-Type: text/html
}}

Answer: I know this post is old, but I experience this exact issue and it has taken me more than 3 day of reading countless articles, samples and answers to finally get my request to be successful. Here is my code that gets an access code and then uses the token to make a request to gets the contacts from Dynamics CRM.
I believe the main difference betrween my code and the OP's is that I use OAuth v2 as my authority URL. I am also using PublicClientAppkicationBuilder vs the Confidential version. I also set the Tenant to force the login to be within the same organisation.
private static HttpClient _client = new HttpClient
{
    BaseAddress = new Uri("https://<your org>.crm11.dynamics.com")
};

public static async Task Main(string[] args)
{
    _client.SetClientAuthentication(await Authenticator.GetAccessTokenAsync());

    var whoami = await _client.GetJsonAsync("api/data/v9.1/WhoAmI");
    Console.WriteLine(whoami);

    var contacts = await _client.GetJsonAsync("api/data/v9.1/contacts?$select=fullname");
    Console.WriteLine(contacts);
}

public static void SetClientAuthentication(this HttpClient client, string accessToken)
{
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
}

public static class Authenticator
{
    public static async Task<string> GetAccessTokenAsync()
    {
        var scopes = new string[] { "https://<your org>.crm11.dynamics.com/.default" };
        var app = PublicClientApplicationBuilder.Create("<your client id>")
            .WithAuthority("https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize", false)
            .WithTenantId("<your tenant id>")
            .WithDefaultRedirectUri()
            .Build();

        AuthenticationResult result;

        try
        {
            var accounts = await app.GetAccountsAsync();
            result = await app.AcquireTokenSilent(scopes, accounts.FirstOrDefault()).ExecuteAsync();
        }
        catch (MsalUiRequiredException)
        {
            result = await app.AcquireTokenInteractive(scopes).ExecuteAsync();
        }

        return result?.AccessToken;
    }
}

Required App registration ids
Required API permissions for Dynamics CRM
Add HTTP localhost as a Mobile/App redirect URL in the Authentication blade
Set public client to true in the Manifest
UPDATE - 26/11/2020
I found <your org> a couple of ways. Both are done from the Dynamics 365 Home Page (https://home.dynamics.com/).

You should have a couple apps either under the My Apps or Pinned Apps headings even if they are just the default ones provided by Dynamics 365. If you hover over one of these apps, the URL will point to a CRM address. The first part is your organisation.

Hovering over an app will display the organisation in the URL

Look for the HTTP request to /apps using the developer tools. The response will contain the list of apps which contain the organisation in the naming convention.

HTTP Request to /apps will have a response containing your organisation
N.B. Your orgnaisation will not always start with "org" but will appear in the URL in the aforementioned places.
Comment: What is organization uri here ? Microsoft docs for web api , but there is no direction how to get [Organization Uri]
https://learn.microsoft.com/en-us/powerapps/developer/common-data-service/webapi/associate-disassociate-entities-using-web-api
Comment: @AdeelShekhani - After re-reading the question and my answer, I completely agree, I didn't actually say where the  part of my example actually was and focused more of how I built up the request to get a working access token. Sorry I didn't notice this sooner, I have updated my answer.
Comment: @AdeelShekhani, In the OP's case, "disco.crm5.dynamics.com" will be their Organization URL if they have located it correctly. So if they are still getting an unauthorized response, I have address how I got a successful response in my original answer. Check my update on how to locate your own Organisation URL.
Comment: Thanks for the info @ConorEyers. Would you mind posting the full URL for the /apps request you mentioned?