How to generate uniform distributed samples with given auto-correlation function

Question: As I mentioned in the question title, I want to generate specific uniformly distributed samples.
I need them to model a real world scenario. For my real data, I estimated a function, which approximates the auto-correlation function by a e-function. I also modeled a distribution model for my real data.
I can generate samples, which follow my distribution model with the inverse transform sampling.
If i think correct, instead of "standard" uniform distributed samples, which i feed into the inverse transform sampling, i need to adapt those uniform distributed samples, so that their auto-correlation function follows my estimated e-function.
Now the question is, if i am correct, how to adapt my uniform distributed samples, that they correspond to my requirements.
It would be great if, somebody could help.
EDIT 1:
Here is a plot of the auto-correlation function of my real data (different scenarios) and the modeled function (green). The ACF of my uniform distributed samples should follow the green (respectively blue) line.

EDIT 2:
Thank you so much so far for the answers. To make my problem a bit clearer, i added a illustration of the problem. For a simulation, i need to generate samples, which follow my distribution model (with inverse transform sampling a easy task) AND have the same auto-correlation behavior like the real data.
My idea was, that i need to adapt the uniformly distributed samples, which i feed into the inverse transform sampling, so that their ACF follows my model.
The problem is, that i don't know if this idea is correct and that i don't know how to adapt the uniformly distributed samples.

EDIT 3:
Following the approach from jblood94, the solution is based on a reordering of the output samples after the inverse transform sampling. So, the underlying distribution stays the same AND the variables follow the desired acf.
I tried to translate the R code from jblood94 to matlab:
clear;
close all;
clc;

x = gamrnd(0.9, 1, 1e6, 1);
alpha = [1, sort(betarnd(1, 3, 1, 50), 'descend')];

tic
xx = acf_reorder(x, alpha);
toc

figure;
autocorr(xx, 'NumLags', length(alpha)-1);
hold on;
plot((1:length(alpha))-1, alpha);

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function xx = acf_reorder(x, alpha)

tol = 1e-5;
maxIter = 10;
n = length(x);
xx = sort(x);
y = randn(n,1);
[w0, w, alpha1] = deal(alpha);
m = length(alpha);
m1 = 1:(m-1);
tol10 = tol/10;
iter = 0;
x = xx(tiedrank(filter(w, 1, y)));
SSE0 = inf;

acfInit = autocorr(x, 'NumLags', m-1)';
SSE = sum((acfInit(1:m)-alpha).^2);

while SSE > tol
    if(SSE < SSE0)
        SSE0 = SSE;
        w = w0;
        iter = iter + 1;
        if(iter == maxIter)
            break;
        end
    else
        break;
    end

    w1 = w0;
    a = 0;
    sse0 = inf;

    while (max(abs(alpha1 - a)) > tol10)
        a = [1, arrayfun(@(x) sum(head(w1, -x).*tail(w1, -x)), m1)/sum(w1.^2)];

        sse = sum((a - alpha1).^2);
        if (sse < sse0)
            sse0 = sse;
            w0 = w1;
        else
            % w0 failed to converge; try optim
            disp('w0 failed to converge; try optim');
            w1 = exp(fminsearch(@fun, log(w0), [], m1));

            a = [1, arrayfun(@(x) sum(head(w1, -x).*tail(w1, -x)), m1)/sum(w1.^2)];
            if (sum((a - alpha1).^2) < sse0)
                w0 = w1;
                break;
            end
        end

        w1 = (w1.*alpha1./a + w1)/2;
    end

    x = xx(tiedrank(filter(w0, 1, y)));
    acf = autocorr(x, 'NumLags', m-1);
    alpha1 = (alpha1.*alpha./acf(1:m) + alpha1)/2;

    SSE = sum((acf(1:m)-alpha).^2);
end

xx = xx(tiedrank(filter(w, 1, y)));

end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function output = fun(ww, m1)

ww = exp(ww);
output = sum([1, arrayfun(@(x) sum(head(ww, -x).*tail(ww, -x)), m1)/sum(ww.^2)]);
end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function dataOut = head(dataIn, num)

dataIn = dataIn(:);

if num>0
    dataOut = dataIn(1:num, :);
elseif num<0
    dataOut = dataIn(1:(length(dataIn)+num));
end

end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function dataOut = tail(dataIn, num)

dataIn = dataIn(:);

if num>0
    dataOut = dataIn((length(dataIn) - num + 1):length(dataIn),:);
elseif num<0
    dataOut = dataIn((-num+1):length(dataIn),:);
end

end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

This yields SOMETIMES plots like this, and sometimes it's trapped in the inner while loop (I need to fix this). As you can see, the plot shows, that the result is not ideal.

EDIT 4:
Maybe I didn't formulate my problem correctly at first: I want to generate a time series that follows a defined distribution (in my case approximately Half-Gaussian) and whose autocorrelation function follows a given autocorrelation function (e.g. up to lag 30). So I have two properties that my time series must fulfil.
I tried the following approaches (without success):
1)

generation of uniformly distributed samples
invese transform sampling with desired distribution
Reordering the samples to match the ACF
Failed: it takes far too long for my time series (1e6 samples) + acf could not be reached.

generation of normally distributed samples with desired autocorrelation (with the approach from jblood94)
invese transform sampling with uniform distribution (based on this solution)
invese transform sampling with desired distribution
Failed: the autocorrelation of the transformed samples does not correspond to the desired Acf, which was correct before the transformation.

I think approach 2 might work. However, it must be prevented that the autocorrelation is changed by the transformation. Or the initial autocorrelation must be adjusted in such a way that it matches the desired Acf again, at least after the transformation.
Now the question is how the initial autocorrelation needs to be adjusted. Or is the approach correct? First generate normally distributed samples with autocorrelation, transform them into normally distributed samples, and then transform them into my desired distribution?
In addition, so far I have only managed to change individual lags of the normal distribution, but not that lag 1-30, for example, follow my function.
EDIT 5:
I tested approach 2 again. It seems, that the change in the autocorrelation is dependent from the acf itself. I generated samples, which follow now my distribution and my acf. I will post plots later.
This approach seems to work for me, but I can't guarantee, that it works in general.
Comment: Related literature Luc Devroye, Gérard Letac (2015) [Copulas with prescribed correlation matrix](https://link.springer.com/chapter/10.1007/978-3-319-18585-9_25)
Comment: Greetings! It may be helpful to show specifically what you have tried so far. This question is fairly vague and doesn't give much for people to work with I'm afraid.
Comment: I vote to reopen.  The answer linked to in the close vote does not answer this question.
Comment: Hi! So far i do not have a entry point for my problem with the correlated uniform distributed samples. All references, which i found so far only explain how to generate random samples with a single correlation-coefficent, but not with a auto-correlation function.
For my auto-correlation i have 30 lags with the corresponding value of my e-function which approximates the auto-correlation-function.
Comment: @JarleTufto the nature of the question was a bit changed due to the answers giving more straightforward approaches to generating correlated variables. But your answer explains nicely why this question is not a duplicate and how finding the Gaussian coppula with a matching correlation structure for the uniform variables is still a difficult task and seperate from the linked duplicate.
Comment: This Q/A may help: https://stats.stackexchange.com/questions/176722/creating-random-variable-with-certain-auto-correlation-in-r/606314#606314
Comment: You can try simulating an Ornstein-Uhlenbeck process and use a memoryless transformation.
Comment: See the update to my answer. The function was breaking prematurely in some cases. It has been updated to fix the behavior.
Answer: Letting $U_t=\Phi(X_t)$ where $X_t$ is a zero-mean and unit-variance stationary Gaussian process with autocorrelation function $\rho_h$ and $\Phi$ is the standard normal cdf, it follows that each $U_t$ is marginally uniformly distibuted.  It follows that the relation between $\rho_h$ and the autocovariance function of $U_t$ is
\begin{align}
\gamma_h&=\operatorname{Cov}(U_t,U_{t+h})
\\&=E(U_tU_{t+h})-E(U_t)E(U_{t+h})
\\&=E(\Phi(X_t)\Phi(X_{t+h}))-1/4
\\&=P(Z_1\le X_t\cap Z_2\le X_{t+h})-1/4
\\&=P\left(\frac{Z_1-X_1}{\sqrt{2}}\le0\cap \frac{Z_2-X_{t+h}}{\sqrt{2}}\le 0\right)-1/4
\\&=\Phi_2\left(\begin{bmatrix}0\\0\end{bmatrix};\frac{\rho_h}2\right)-1/4.
\end{align}
Here $Z_1$ and $Z_2$ are independent standard normal random variables, and $\Phi_2$ is the cdf of the standard bivariate normal distribution with correlation $\rho_h/2$.
Using this result, the above relation simplifies to
$$
\gamma_h = \frac1{2\pi}\operatorname{arcsin}\frac{\rho_h}2.
$$
Solving this equation, we find that the autocorrelation of $X_t$ needed to achieve a target autocovariance $\gamma_h$ of $U_t$ is
$$
\rho_h = 2\sin(2\pi \gamma_h).
$$
Even if the target autocovariance function $\gamma_h$ is positive semi-definite, the above construction may not be feasible.  For example, if the target autocovariance function is that of a MA(1)-process with a unit root,
$$
\gamma_h=\begin{cases}
1/12 &\text{for }h=0 \\
1/24 &\text{for }h=1 \\
0 &\text{for }h>1
\end{cases},
$$
this would imply that
$$
\rho_h=\begin{cases}
1 &\text{for }h=0 \\
0.518 &\text{for }h=1 \\
0 &\text{for }h>1
\end{cases}
$$
which is not a positive semi-definite autocorrelation function.
Comment: Hi Jarle Tufto, i edited my post, so that my problem should be clearer. As i understand you correct, its not feasible to generate uniformly distributed samples with a desired ACF or at least a approximation to the ACF of my real data?
Comment: @Christian My answer shows that how you can generate uniformly distributed $U_t$ with a given autocovariance function $\gamma_h$.  The construction may or may not exists since $\rho_h$ needs to positive semi-definite.  But in your EDIT2 you seem to want to further thansform $U_t$ into something that matches your empirical distribution function (via the inverse cdf method).  This additional transformation will also further change the autocovariance function so you need to further tweak the autocorrelation function of $X_t$ to achieve your goal (maybe via numerical integration and root finding).
Comment: Could you provide an example code for your approach?
Comment: @Christian In general, you would need to use `mvtnorm::rmvnorm` to simulate $X_t$ with a given autocovariance function $\rho_h$.  The `toeplitz` function can be used to construct the required covariance matrix from a vector of autocorrelations.  If $X_t$ is a moving average process ($\rho_h=0$ for lags $h>q$), then `arima.sim` would be much more efficient.
Answer: Here's a pragmatic and easy approach with room to expand and establish proofs, with the focus on the main problem: how do you generate a correlated uniform sample?
Let $U_1, U_2$ be uniformly distributed and independent on the unit interval. Let $V_1 = U_1$. For a desired correlation $d$, let $G_{1,2}$ be yet another uniformly distributed random variable.
Let $$V_2 = \left\{ \begin{array}{ccc} U_1 & \text{if} & G_{1,2} < d \\
U_2 & \text{if} &G_{1,2} \ge d\end{array} \right.$$
I claim that:

One can show that $V_2$ is indeed uniformly distributed, with a correlation of $d$ with $V_1$
By way of induction, one can set an arbitrary sequence of uniform random variables and create a new sample having a desired covariance structure.

As usual, this site is always most compelled by code, so I can show a couple cases of the spherical AR-1 auto correlation, where the correlation between observations with a lag of 1 is set to $d$, but otherwise it's relatively straightforward to use any structure you want.
do.one <- function(n,N,d) {
  u <- matrix(runif(n*N), N, n)
  g <- matrix(runif((n-1)*N/2), N, n-1)

  v <- u
  for( i in 2:n) {
    v[g[, i-1] < d,i] <- v[g[, i-1] < d, i-1]
  }

  acf <- sapply(2:n, function(i) cor(v[,i], v[, 1]))
  acf
}

set.seed(123)
ds <- c(0,0.25, 0.5, 0.95)
acfs <- sapply(ds, do.one, n=10, N=1000)
matplot(acfs,type='l', xlab='Lags')
legend('topright', title = 'AR-1 spherical correlation', legend = ds, lty=1:4, col=1:4)

Comment: Interesting. Basically a sort of random walk where you stay in the same place with probability $d$ and transition to a random position in the unit interval with probability $1-d$.
Comment: You might extend your answer to explain that your method gives ACF $\rho_h=d^{|h|}$. Should a more general ACF be required, one can increase the lag, for example with lag 2: $U_k=\begin{cases}U_{k-1}&\text{with probability $d_1$}\\ U_{k-2}&\text{with probability $d_2$}\\ \mathrm{Unif}(0,1)& \text{otherwise}\end{cases}$
Comment: What's also interesting is that the Gaussian version of your technique is indistinguishable from a standard AR(1) based on mean, variance and ACF alone, while the actual time series is very different.
Comment: It is also worth noting that some ACF profiles may not be achievable with the above approach. For example one cannot have $\rho_1>0$ and $\rho_2<\rho_1^2$. This can be easily overcome by allowing some of the lags to have terms $1-U_{k-\ell}$ rather than $U_{k-\ell}$. For example, the case I just mentioned can be obtained with: $U_k = \begin{cases} U_{k-1} & \text{with probability $d_1$}\\ 1 - U_{k-2} & \text{with probability $d_2$}\\ \mathrm{Unif}(0,1) & \mathrm{otherwise} \end{cases} $.
Comment: @LucaCiti and AdamO: Out of interest I posted [another answer](https://stats.stackexchange.com/a/606716/77222) spelling out some of details of what you propose.
Answer: If I understand correctly your problem, you need to simulate random variables which marginally follow an uniform distribution while the joint distribution is a multivariate uniform distribution with some correlation between your marginals.
Formally:
$X_i\sim U(0,1), \forall_i, i = 1,2,...,K$ , where $X\sim U(a_i, b_i, Cor)$.
In the method I am firstly proposing, your uniform random variables will be uniformly distributed between 0 and 1. However, I also provide a "Normal to Anything" kind of approach.
One trick is to use generate a Multivariate normal distribution, specifying a correlation matrix. Let's say $\Sigma$ is your auto-correlation structure for your $K$ random variables.
$$X \sim MVN(\mu, \Sigma)$$
$\mu$ is a vector of means of size $K$, while $\Sigma$ is a $K*K$ matrix
Then you have to transform your quantiles to probabilities from any normal distribution you want, for one $X$:
$$CDF_{Normal}(X_{i}, \mu_i, \Sigma_{ii}) \sim U(0,1)$$
Now if you assume that your variables are not uniformly distributed between a common support $[a,b], \forall K$ but a variable-dependent support instead. You just need to convert your probabilities obtained previously with an inverse uniform distribution specifying a and b.  For a random variable,
$$\theta^{-1}(CDF_{Normal}(X_{i}, \mu_i, \Sigma_{ii}), a_i, b_i)$$
It is worth noting that you will respect globally your correlation structure, while under-estimate it a little bit. Some force-brute methods should be used to correct your final correlation structure.
I can provide script to illustrate the method.
Hope this helps.

set.seed(1234)

#Example function to generate auto-correlation matrix
autocorr.mat <- function(p = 100, rho = 0.9) {
  mat <- diag(p)
  autocorr <- sapply(seq_len(p), function(i) rho^i)
  mat[lower.tri(mat)] <- autocorr[unlist(sapply(seq.int(p-1, 1), function(i) seq_len(i)))]
  mat[upper.tri(mat)] <- autocorr[unlist(sapply(seq_len(p - 1), function(i) seq.int(i, 1)))]
  return(mat)
}

#Simulate normal data with some autocorrelation structure
autocorrelated.normal = MASS::mvrnorm(1000, mu= rep(0,100), autocorr.mat())

#Let's see the heatmap
heatmap(cor(autocorrelated.normal))

#If you are interested in simulating random uniform between 0 and 1 with your auto-correlation structure
pnorm(autocorrelated.normal)

#If you are interested in simulating random uniform between q and p, here [-10; 10]

qunif(pnorm(autocorrelated.normal), -10,10)

A brief comparison of the respective heatmaps give global concordance of initial correlation structure

heatmap(cor(autocorrelated.normal))
heatmap(cor(pnorm(autocorrelated.normal)))
heatmap(cor(qunif(pnorm(autocorrelated.normal), -10,10)))

Comment: More specifically, it is a Gaussian coppula. The sidenote is very important *"It is worth noting that you will respect globally your correlation structure, while under-estimate it a little bit."*. The correlation after transformation will not be the same.
Comment: +1 this is essentially a copula approach https://en.wikipedia.org/wiki/Copula_(probability_theory)
Comment: @Christian If you are interested only in generating uniform RVs so my first solution is the way to go. However, if you are interested to simulate others RVs from others distributions, you will need to correct your correlation structure. I can provide additional details if you are interested.
Comment: Yes indeed, intuitive and easy to implement for this kind of problem
Comment: Hi Mangnier Loïc, i need to generate random samples for a simulation. The samples should have the probability distribution of my model AND should have the same ACF like my real data (the ACF is modeled by an e-function, which approximates the ACF of my real data quite well). Currently i generate the samples by feeding in uniformly distributed samples in a inverse transfom sampling. my generated samples follow now my desired distribution, BUT do not have the desired ACF.
Comment: i am not sure, if its correct, that i simply adapt the uniformly distributed input-samples. But if so, then i am looking for an solution on how to modify the uniform distributed samples as mentioned above.
Comment: Hi Mangnier. Additional details would be great. Currently i try the reordering-approach from jblood94. But another (faster) solution would be great.
Comment: Could you provide additional info? It would be great!
Comment: @Christian Do you mean R code ?
Comment: Yes, or literature. So far every solution i tried failed. I created a timeseries with the inverse transform sampling from my desired Distribution. But this Series of pseudo-random values is not correlated. Every approach to add a autocorrelation to my timeseries either changed the underlying Distribution or needs way to much Time (Reordering of timeseries)
Comment: Matlab is also welcome, as i Need to translate the r-Code to matlab, because my Simulator is implemented in matlab.
Comment: @Christian, I have provided R code to illustrate the method.
Comment: Ok, thanks. But I think i stated my problem wrong: I need one timeseries which has a desired acf (and follow a desired distribution). I think your approach solves the problem, with multiple time series, which are correlated with each other.
Answer: You can try implying AR(p) process coefficients $\phi_i$ from the given ACF $r(p)$. You could apply Yule Walker equations:

form a vector $r$ of ACF for lags $p$: $1, r_1, r_2,\dots, r_p$
construct a correlation matrix $R$ as described in the link above from $r_p$, e.g. the third row would be $(r_2,r_1,1,p_1,\dots,r_{p-2})$
calculate $\phi=R^{-1}r$

Use these coefficients to produce autocorrelated samples
Comment: This wouldn't generate samples that are uniform.
Comment: @AdamO easy to transform them to uniform using CDF
Answer: This problem can be approached by first generating samples from the desired distribution and then reordering them to match the desired autocorrelation function.
The R code below demonstrates an approach based on this answer that can be modified for any desired ACF and distribution. The example generates $n=10^6$ samples from $\text{Gamma}(0.9,1)$ whose ACF follows 50 random samples from $\text{Beta}(1,3)$, sorted descending.
The process is as follows.

Generate the desired number of samples, $X=\{x_1,...,x_n\}$, from the target distribution. Set $\alpha_0$ equal to the desired ACF. Initialize the target ACF, $\alpha$, as the desired ACF.
Find a set of weights that, when passed to filter along with $n$ random normal variates, results in a series, $Y$, with $ACF=\alpha$ (see the answer linked above).
Reorder $X$ so that its rank ordering matches that of $Y$. If $X$ is normally distributed, the resulting series should have the desired ACF; however, the more $X$ deviates from normality, the more the ACF will deviate from $\alpha_0$ (the example below has a target distribution of $\text{Gamma}(0.9,1)$, which is very "non-normal"). Update the target ACF, $\alpha$, according to $\alpha'=\frac{\alpha}{2}\Big(\frac{\alpha_0}{ACF}+1\Big)$ and repeat steps 1-3 until the ACF of the reordered $X$ converges.

The function that performs the reordering (it works only for positive values for alpha):
acf.reorder <- function(x, alpha) {
  tol <- 1e-5
  maxIter <- 10L
  n <- length(x)
  xx <- sort(x)
  y <- rnorm(n)
  w0 <- w <- alpha1 <- alpha
  m <- length(alpha)
  i1 <- sequence((m - 1):1)
  i2 <- sequence((m - 1):1, 2:m)
  i3 <- cumsum((m - 1):1)
  tol10 <- tol/10
  iter <- 0L
  x <- xx[rank(filter(y, w, circular = TRUE))]
  SSE0 <- Inf
  f <- function(ww) {
    sum((c(1, diff(c(0, cumsum(ww[i1]*(ww[i2]))[i3]))/sum(ww^2)) - alpha1)^2)
  }
  ACF <- function(x) acf(x, lag.max = m - 1, plot = FALSE)$acf[1:m]

  while ((SSE <- sum((ACF(x) - alpha)^2)) > tol) {
    if (SSE < SSE0) {
      SSE0 <- SSE
      w <- w0
    }
    if ((iter <- iter + 1L) == maxIter) break
    w1 <- w0
    a <- 0
    sse0 <- Inf

    while (max(abs(alpha1 - a)) > tol10) {
      a <- c(1, diff(c(0, cumsum(w1[i1]*(w1[i2]))[i3]))/sum(w1^2))

      if ((sse <- sum((a - alpha1)^2)) < sse0) {
        sse0 <- sse
        w0 <- w1
      } else {
        # w0 failed to converge; try optim
        w1 <- optim(w0, f, method = "L-BFGS-B")$par
        a <- c(1, diff(c(0, cumsum(w1[i1]*(w1[i2]))[i3]))/sum(w1^2))
        if (sum((a - alpha1)^2) < sse0) w0 <- w1
        break
      }

      w1 <- (w1*alpha1/a + w1)/2
    }

    x <- xx[rank(filter(y, w0, circular = TRUE))]
    alpha1 <- (alpha1*alpha/ACF(x) + alpha1)/2
  }

  xx[rank(filter(y, w, circular = TRUE))]
}

Generate samples from the target distribution and specify the desired ACF:
set.seed(1960841256)
x <- rgamma(1e6, 0.9, 1)
alpha <- c(1, sort(rbeta(50, 1, 3), TRUE))

Reorder x and plot its ACF against alpha:
system.time(x <- acf.reorder(x, alpha))
#>    user  system elapsed
#>    7.13    0.41    7.53
acf(x, lag.max = length(alpha) - 1)
lines(seq_along(alpha) - 1, alpha, col = "green")

The resulting ACF is a good match with the target, and since x has simply been reordered, it is known to have the desired distribution.

Update: Worst of 100 iterations
The original code was breaking too early for some cases as noted by the OP in the updated question. The above code has been modified to correct the behavior. I also removed the restriction that the ACF be strictly positive.
I ran the same procedure 100 times, each with a new random vector for x and alpha. The longest-running iteration took 7.36 seconds, and the worst performing iteration (by maximum absolute difference of the achieved vs. desired ACF) had a maximum absolute error of 0.056. It is plotted below.
Comment: Hi jblodd94. Thanks for your answer. this seems to be the solution of my problem. After the sorting, the samples still follow the initial distribution AND also follow the desired acf. Do you have literature for your approach?

I translated your R code to matlab, but till now i did not manage to get your algorithm working.
Comment: Good to hear you got it working in Matlab. Sorry, no literature to cite here. The approach was just off the top of my head without any literature search, so it is possible someone else has published something similar.
Comment: +1 clever solution. But gadzooks! Using `optim` in a `while` statement might be a CPU smasher if $n$ is big and $\alpha$ is high. I wonder if you can't just calculate the fraction $p$ of the sample $U = U_1, \ldots U_n$ that should be replaced with the order statistics...
Comment: As an aside, I'm fairly confident this approach can be extended to work with mixed-sign autocorrelation values. Your "real data" showed strictly positive values, so I didn't bother.
Comment: @AdamO, it's actually `optim` in a nested `while` :). The break conditions seem to keep it manageable. I tried to demonstrate with a hard example (1M samples from a very non-normal distribution and a not-very-smooth ACF) and it took ~1 min. I just added the timing.
Comment: as I need the samples quite fast for a simulation, I sure need to adapt the algorithm. first I need to get my matlab translation running
Comment: @Christian you should consider modifying the approach as follows. For a random uniform, 0,1 sample, $U=U_1, \ldots, U_n$, let the order statistics be $U_{(1)}, \ldots, U_{(n)}$. Generate $V_i = \rho U_{(i)} + (1-\rho) U_i$. You can show that 1: $V$ is indeed random uniform and 2: $\rho$ gives the desired lagged acfs.
Comment: AdamO: Do you have literature for this? and stays the acf of the uniform samples the same after the inverse transorm sampling? I think Jarle Tufto mentioned, that the due to the inverse transform sampling of the uniform samples, the acf will also change.
Comment: @AdamO, how would one select $\rho$? Using `alpha` from my example, I tried the following: `u <- runif(1e6); v <- filter(sort(u), alpha, circular = TRUE) + filter(u, 1 - alpha, circular = TRUE); acf(v, lag.max = length(alpha) - 1)`, but the resulting ACF does not match `alpha`.
Comment: @Christian, See the updated function. I was able to vectorize the `sapply` in the `optim`. It sped up the code by a factor of 8. If I can get the suggestion by @AdamO to work, it may remove the need for the inner `while` loop (with the `optim`), which would lead to a ~1 second speedup.
Comment: As a approximation to my problem, I will use your approach from this [question](https://stats.stackexchange.com/questions/176722/creating-random-variable-with-certain-auto-correlation-in-r/606314#606314). As my core distribution is nearly gaussian, this should be a good approximation.
Comment: Got it. That approach forms the core of this answer. The `a <- c(1, diff(c(0, cumsum(w1[i1]*(w1[i2]))[i3]))/sum(w1^2))` line above is a vectorized version (much faster) of `a <- c(1, sapply(n, function(i) sum(head(w, -i)*tail(w, -i)))/sum(w^2))` from the linked answer.
Comment: Unfortunately the simplified approach does not work. As my distribution is not nearly gaussian but nearly HALF-gaussian. Therefor the 'reordering' completely messes up my desired ecdf. Also I got strange behavior with the weights, when I put in my desired acf `a*exp(b*x) + c*exp(d*x)`
Comment: What values for `a,b,c,d`?
Comment: `a = 0.741, b = -1.247, c = 0.259, d = -0.007139` Somehow it doesn't converge
Comment: It does seem to be converging, just not at the desired ACF for `length(alpha) > 12`. It's unclear if it is converging at a local minimum or if the approach fails for some ACFs, or if the desired ACF is not actually achievable.
Answer: The following spells out the details of the approach proposed in the other answer by @AdamO and in its comments by @LucaCiti.
For $i=1,2,\dots,\infty$, let $|\phi_i|$ denote the probability that $U_t$ takes a value identical to either $U_{t-i}$ or $1 - U_{t-i}$ and let these two possibilities be determined by the sign of $\phi_i$.  Let the remaining fraction
$$
\phi_0=1-\sum_{i=1}^\infty |\phi_i|
$$
denote the probability that $U_t$ takes a uniformly distributed value independent of the history of the process.  Clearly, we must have
$$
0\le \phi_0\le 1. \tag{1}
$$
and
$$
-1\le \phi_i\le 1 \tag{2}
$$
for $i=1,2,\dots,\infty$.
Letting $V_t=U_t-\frac12$ denote the mean-centered process, and using the law of total expectation, we have
\begin{align}
E(V_t|V_{t-1},V_{t-2},\dots)
&=|\phi_1|\operatorname{sgn}(\phi_1) V_{t-1} + |\phi_2|\operatorname{sgn}(\phi_2) V_{t-2} + \dots
\\&=\phi_1 V_{t-1} + \phi_2 V_{t-2} + \dots.
\end{align}
Thus it is immedeately clear that $\phi_1,\phi_2,\dots$ are the coefficients in the  $\operatorname{AR}(\infty)$ representation of the model.
Unlike ordinary Gaussian ARMA models, constraint (1) and (2) implies that not all positive semi-definite autocovariance functions are possible via this construction, however. For example, if the target autocovariance function is that of a MA(1) model with MA polynomial $1-\theta B$, the infinite AR polynomial would equal
$$
\frac1{1-\theta B}=1+\theta B+\theta^2 B^2+\dots,
$$
and we would have
$$
\sum_{i=1}^\infty|\phi_i|=\sum_{i=1}^\infty |\theta^i|=\sum_{i=1}^\infty |\theta|^i = \frac{|\theta|}{1-|\theta|}.
$$
Combined with (1) this limits possible values of $\theta$ to
$$
|\theta|\le \frac12
$$
and the correlation at lag 1 to
$$
-\frac25\le \rho_1=\frac{\theta}{1+\theta^2}\le\frac25
$$
In contrast, via the copula described in my other answer the correlation at lag 1 is limited to $|\rho_1|<0.4825837$ only .  Semipositive definiteness in itself limits the same correlation to $|\rho_1|\le 1/2$.
Answer: Uniform distributed samples are the set of samples in which every element is distributed uniformly, if we place further constraints on this, it ceases to be uniform distributed samples. But anyway, if you meanе something else, we can easily upgrade my answer.
Let us just to generate samples with given autocorrelation function. Our idea is put all needed constraints on samples and let scipy.optimize to do everything for us.
While the key concept is simple, optimization and numerical problems can be arbitrary hard in real application and it can be necessary to adjust scipy solvers parameters here and/or optimize some computations.
The code implements this idea:
import numpy as np
import scipy.optimize as sco
import matplotlib.pyplot as plt

def cov(x,y):
    return np.sum(x*y)/len(x)

def cor(x,y):
    return cov(x,y)/(np.std(x)*np.std(y))

#define our autocorrelation
def autoMoment(x, moment, n):
    out = []
    for i in range(n):
        out = [moment(x,x)]
        for i in range(1,n):
            out.append(moment(x[i:],x[:-i]))
    return np.array(out)

#define generation of function for scipy.optimize
def generateWithSpecificAutomoment(moment, values):
    def f(x):
        out = autoMoment(x, moment, len(values))
        return np.sum((np.array(out) - np.array(values))**2)
    return f

desiredAutocorr = [1,0.3,0.1,-0.23,0.03,0.07,-0.03]

initial = np.random.randn(50) #starting point for the solver
solution = sco.minimize(generateWithSpecificAutomoment(cor, desiredAutocorr), initial)

print(solution.success)
out = solution.x #samples obtained after optimization

#check autocorrelation on generated samples
realAutocorr = autoMoment(out, cor, len(desiredAutocorr))

#compare desired and real result
plt.title("Autocorrelations")
plt.plot(desiredAutocorr, linewidth=7, label="desired")
plt.plot(realAutocorr, label="real")
plt.legend()
plt.show()

Generated picture:
Comment: "... if we place further constraints on this, it ceases to be uniform distributed samples." I think this statement needs to be qualified in some ways. For instance, you may set U1, U2 uniformly distributed. You can set U1=U2 or U1=-U2 to create a perfect counter example.
Comment: Hi Kirill Erofeev, i need to generate random samples for a simulation. The samples should have the probability distribution of my model AND should have the same ACF like my real data (the ACF is modeled by an e-function, which approximates the ACF of my real data quite well). Currently i generate the samples by feeding in uniformly distributed samples in a inverse transfom sampling. my generated samples follow now my desired distribution, BUT do not have the desired ACF.
Comment: i am not sure, if its correct, that i simply adapt the uniformly distributed input-samples. But if so, then i am looking for an solution on how to modify the uniform distributed samples as mentioned above.
Comment: But with your approach i can solve at least the problem with the ACF of the input samples to my inverse transform sampling. Thank you!!
Answer: You can try 1d-OU Process:
$dX_t=-\alpha X_tdt+\sqrt{2\alpha}dW_t$
The autocorrelation function of $X_t$ is $\text{exp}(-\alpha t)$, where $dW_t\sim \mathcal{N}(0, \sqrt{dt})$
Assume that your exponential ACF fitting parameter is $0.08$ and $dt=0.1$, then the SDE is $dX_t = -0.08dX_t+0.4dW_t$.
Use memoryless transformations on simulation results $Y$ to get the results you need:
$Y = U^{-1}[\varPhi(X)] = a+\varPhi(X)\cdot (b-a)$
where $\varPhi(\cdot)$ is the CDF of the standard normal distribution with zero mean and unit variance; $U(a,b)$ is the CDF of the uniform distribution with lower bound $a$ and upper bound $b$.
ACF verification:

Histogram of $Y$:
$a=1, b=3$ in this case

import numpy as np
import matplotlib.pyplot as plt
from statsmodels.tsa.stattools import ccf, acf
import scipy.stats as stats
import seaborn as sns

def oup(alpha, dt, num_steps, x0):
    sigma = np.sqrt(2*alpha)
    dW = np.sqrt(dt) * np.random.normal(size=num_steps)
    X = np.zeros(num_steps + 1)
    X[0] = x0
    for i in range(1, num_steps + 1):
        dX = - X[i-1] * alpha * dt + sigma * dW[i-1]
        X[i] = X[i-1] + dX
    return X

a = 0.08
dt = 0.1
T = 1e6
num_steps = int(T / dt)
lower = 1
upper = 3

X = oup(a, dt, num_steps, 0)
Y = lower + stats.norm.cdf(X, 0, 1) * (upper - lower)

num = int(30 / dt)
t = np.linspace(0, 30, num + 1)
acf_original = np.exp(-a*t)
X_acf = acf(X, nlags = num, fft = True)
Y_acf = acf(Y, nlags = num, fft = True)

plt.figure(figsize = (4, 3), dpi = 120)
plt.plot(t, acf_original, label = 'Original ACF', color = 'black')
plt.plot(t, X_acf, label = r'$X$', color = 'r', linestyle = '--')
plt.plot(t, Y_acf, label = r'$Y$', color = 'b', linestyle = '-')
plt.ylim(-0.1, 1.1)
plt.ylabel('ACF')
plt.xlabel('Lag [s]')
plt.legend()
plt.show()

sns.histplot(samples, stat = 'density', bins = 100)


The simplest nontrivial (unstable) integral cohomology operation

Question: By an integral cohomology operation I mean a natural transformation $H^i(X, \mathbb{Z}) \times H^j(X, \mathbb{Z}) \times ... \to H^k(X, \mathbb{Z})$, where we restrict $X$ to some nice category of topological spaces such that integral cohomology $H^n(-, \mathbb{Z})$ is represented by the Eilenberg-MacLane spaces $K(\mathbb{Z}, n)$. The Yoneda lemma shows that such operations are in natural bijection with elements of $H^k(K(\mathbb{Z}, i) \times K(\mathbb{Z}, j) \times ... , \mathbb{Z})$.
Altogether, these cohomology operations determine a (multisorted) Lawvere theory. There is an obvious subtheory of this theory generated by zero, negation, addition, the cup product, and composition. Based on the table in this paper, it looks like the simplest integral cohomology operation not in this obvious subtheory is a cohomology operation $H^3(X, \mathbb{Z}) \to H^8(X, \mathbb{Z})$ coming from the generator of $H^8(K(\mathbb{Z}, 3), \mathbb{Z}) \cong \mathbb{Z}/3\mathbb{Z}$, but I don't know enough algebraic topology to extract an explicit description of this cohomology operation from the paper.
So: what is this cohomology operation? Where does it come from? What can you do with it? (I know there are some cohomology operations coming from the $\text{Tor}$ terms in the Künneth formula; is this one of them?)
Comment: follow-up: http://math.stackexchange.com/questions/429976/why-massey-cube-of-an-odd-element-lies-in-3-torsion
Comment: (different) follow-up @ MO: [From calculation of $H^\bullet(K(\pi;n))$ to explicit construction of operations](http://mathoverflow.net/q/135303/1556)
Comment: (somewhat) related: http://mathoverflow.net/questions/6377/why-does-one-think-to-steenrod-squares-and-powers
Answer: Suppose $\alpha$ is an integral $3$-cocycle on a space $X$. Then $2\alpha\cup\alpha$ is a coboundary, because the cup product is graded-commutative, and there exists a $5$-cochain $\beta$ such that $\mathrm d\beta=2\alpha\cup\alpha$. One can easily check that $\alpha\cup\beta+\beta\cup\alpha$ is an $8$-cocycle. I think that the class of this cocycle does not depend on the choice of $\beta$, so we get a well defined mapping $H^3\to H^8$. This map is natural because there is a natural proof that the cup product is graded-commutative, so we $\beta$ depends naturally on $\alpha$.
This gives an operation :-)
Comment: This operation is presumably the same as $H^3(X,\mathbb{Z}) \to H^3(X,\mathbb{Z}/3) \xrightarrow{P^1} H^7(X,\mathbb{Z}/3) \to H^8(X,\mathbb{Z})$, where $P^1$ is the first Steenrod operation, and the first and last maps are from the coefficient sequence for $0\to \mathbb{Z}\to \mathbb{Z}\to \mathbb{Z}/3\to 0$.
Comment: @user17786, if $x$, $y$, $z$ are classes such that $xy=0=yz$, then the product $xyz$ is zero for «two reasons». This observation is used to construct a secondary operation called the Massey triple product (this is explained more or less in Wikipedia) I started with this and tried to make it work better than usual in our case (triple products have an indeterminancy, but it disappears in out case magically)
Comment: This is so obvious that it had to be done ages ago :-) Something very similar was studied by Kraines [here](http://www.ams.org/journals/tran/1966-124-03/S0002-9947-1966-0202136-1/home.html) He shows that his is zero over the rationals, and non-trivial over odd primes (it is then given by Bocksteins and reduced powers) He does not say anything about integralcoefficients, afaict.
Comment: Oh, you're right. (What anti-commutativity actually implies is that the definition doesn't depend on choice of $\beta$.)
Comment: Interesting! Does this Massey product line of reasoning suggest a reason why this operation ought to land in $3$-torsion?
Comment: @QiaochuYuan I suspect it's something like Jacobi identity. (Notice that the operation is $P(\alpha)=[\alpha,\beta]$ and $d\beta=[\alpha,\alpha]$, where $[-,-]$ is the supercommutator...)
Comment: $\alpha\beta=-\beta\alpha$, so your operation seems to be zero, no?
Comment: That equality holds in *cohomology*, not on the level of (singular) cocycles. Notice that $\beta$ is not even a cocycle.
Comment: (I'm upvoting this, but the question why this operation is nontrivial still remains...)
Comment: @mariano how does this definition occur to you?
Comment: If this is non-zero, the example should be on $K(Z,3)$, mapping the identity mapping in $H^3(K(Z,3))$ to a non-zero element of $H^8(K(Z,3))$. This should work already on sufficiently good approximations to the E-M space, like $(S^3)^n/\Sigma_n$ (with $n=3$, probably)...
Comment: @GrigoryM, triple products are indeed closely related to the Jacobi identity; this is explained in McCleary's book spectral sequences, iirc.
Comment: @CharlesRezk, that's the sort of thing Kraines gets for Massey powers $\langle a,\dots,a\rangle$, in fact.


Android vector drawables vrs png

Question: Having doubts about this.
I know that vector drawables have the scaling advantage. But before i start making all my new icons in this way, is there any good reason to stick with the good old Pngs?
I would really appreciate a good advice on this.
Answer:
But before i start making all my new icons in this way, is there any
  good reason to stick with the good old Pngs?

if providing only one version of your drawable instead than 3 or 4 scaled versions is not enough a good reason, then you can stick with drawables. Be aware that VectorDrawable are supported from lollipop onwards. For the platforms between ICS and Lollipop, the gradle plugin (1.5.0) creates a raster  versions, of your vector drawable. You can specify the density you want to export adding
 generatedDensities = ['mdpi', 'hdpi', 'xhdpi', 'xxhdpi']

to the defaultConfig section of your build.gradle.
Edit
As point out by @Huteri, the support library v23.2 makes it easier to use it, adding support-vector-drawable and animated-vector-drawable. Have a look here
Comment: On android support library 23.2 and Gradle 2.0 we don't need this anymore, http://android-developers.blogspot.co.id/2016/02/android-support-library-232.html
Comment: I've been using it for weeks, and seems great.

Android vector drawables vrs png


Is my hypothesis correct?

Question: $$\left| \left|(a^2) - 25\right|-b\right| + b = 0$$
You have to prove that $b<0$ and $b=0$ at the same time
I have no problem to prove that $b$ can be $0$ the thing that I need help with is $b<0$. My thought there is a bit complicated and some people I've asked have said that's it's not valid but here it is. Oh and by the way sorry that I don't write pure mathematics I am a rookie just a 16 year old kid  so my thoughts will more on like the in the form of words(I know that it's not 100% maths but these are my thoughts). Sorry if it sounds silly.So here it starts.
Let's break the whole thing down into 2 parts
1)$\left| \left|(a^2) - 25\right|-b\right|$
2)$+ b$
As it's well know in the subtraction of two numbers in order to be $0$ (considering that they belong to Real)numbers they should be opposite for ex.
$4-4=0$ or $a-a=0$
That means that one of them should be negative and another positive AND THEY SHOULD BE REPRESENTED BY THE SAME SYMBOL BUT OTHER SIGN
Let's hypothesize that $b$ is actually smaller that $0(b<0)$. Each number consists from it's number sign(+,-) and an number sign and a symbol(1,2,3,4,5....)(Again I am talking about Real numbers). Let's call the symbol "$y$" because of it's randomness so b should be equal with (- "$y$")
The thing now changes from :
$\left|\left|(a^2) - 25\right|-b\right| + b$  to :
$\left|\left|(a^2) - 25\right|-(- y)\right| + (-y) $
since $b$ is negative it should be represented as ($- y$) because of it's negativity
So it proves out that :
$\left|\left|(a^2) - 25\right| + y\right| - y$
That guides us to think that the negative part on this is the second part actually because the second part ( $- y$) is negative the first one should be the positive one a fact that can be verified from the fact that $\left|\left|(a^2) - 25\right| + y\right|$ is a positive number since $|-4|=|4|=4$ or $|y|=|-y|=y$
Because the 1st part and the 2nd part are represented by the same symbol but not sign we supposed that
$\left|\left|(a^2) - 25\right| + b\right| = + y$ like the positive part
and
$+ b = - y$ the negative part
this means that
$b + b \Rightarrow + y - y = 0 \Rightarrow y=y$ something that is valid so
I hypothesized something and it turned out that my result is something valid this must mean that my hypothesis is correct
Comment: Sorry...what are the assumptions and what are you trying to prove?  It is not true that no $b$ exists...indeed $(a,b)=(\pm 5,0)$ both work.
Comment: You should **first** tell us what you're tryng to do: to solve an equation? To prove an identity? What is $\;a,b\;$ ...This is too much reading to not know what is going on.
Comment: Sorry for not being quite understood the question is for the real numbers a,b it's true that | |(a^2) - 25|-b| + b = 0 you have to prove that b<0 and equal at the same time I am analyzing the half of the question the thing that b<0
Comment: "I hypothesized something and it turned out that my result is something valid this must mean that my hypothesis is correct"  WHOA!!!!  **!!!!NOT!!!!** TRUE!.  Very, very, very, **VERY** not true!!! Hypothesis:  I have a giant carnivorous dragon in my garage.  Conclusion:  If I had a giant carnivorous dragon in my garage then I it would eat all my rats.  Observation: I have no rats in my garage.  THEREFORE it must be true I have a giant carnivorous dragon in my garage.  (Not valid.  Doesn't follow.  Is not true.)
Comment: Could you try to make it clearer?  What are $a,b$?  Integers?  Real numbers?  Do you have a solution in which $b<0$?  I'd like to see that.
Comment: "You have to prove that b<0 and b=0 at the same time" which is impossible.  So you have to prove this equation can never be true.
Comment: Well sorry again for the misunderstanding all this shit a re on one symbol but I dont know it cuz  I don't know how to type it from my keyboard.What I want to say with b<0 and b=0 at the same time can be expressed with one symbol I will try to find it
Comment: @lulu as I foretold  a,b are real numbers.Regarding the solution is that big shit I wrote it is maybe not correct as fleablood says but I need  your opinions guys
Comment: @fleablood I found the symbol it's ≤ damn it xD
Comment: @George This is why it is advisable to read *first* the general, easy directions of this site to write mathematics in a readable, nice way.
Comment: @Joanpemo well this problem was quite a pain for me for like 2 days  I wanted to get it off me :P
Comment: Your reasoning isn't bad.  In fact it's good but there are a few point.  $\le$ means $b < 0$ **OR** $b = 0$.  It doesn't mean both.
Comment: Welcome to Math StackExchange. When you write mathematics please use the MathJax environment. You can find a basic tutorial on it
[here](http://meta.math.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference). Also while writing a question you can see a yellow box to the right of the page with indications on how to ask and how to format a question. It is very important that you know what is it exactly that you want to ask and state it precisely.
Comment: @fleablood thanks for the little ≤ thing that's one of the most stupid things I've ever said lol i think it was because I am clumsy as a person :P anyways thanks for the solution as well
Comment: This equation *will* be true whenever $a^2 - 15 = 0$ (which is when $a = \pm 5) and $b \le 0$.
Comment: @Anguepa alright next time I decide I have to post something I'll make sure I do it correctly (clumsy me :P) Sorry for any misunderstanding guys
Answer: $||a^2 - 25| - b| + b = 0 \implies$
$||a^2 - 25| - b| = -b$.
But $||a^2 - 25| - b| \ge 0$ because absolute values are non negative.
So $-b \ge 0$ so $b \le 0$  (i.e.  $b$ is negative or 0.)
And that's it.  We are done.  We know if the equation is true $b \le 0$.
But we don't know if there is more restrictions we can have and we don't know if the equation itself is even possible.
====
I want to go further and see if there is anything more we can deduce.
Now $|a^2 - 25| \ge 0$ and $b \le 0$ so $|a^2 - 25| - b \ge 0$.
So $||a^2 - 25| - b| = |a^2 - 25| - b$.
So $-b = ||a^2 - 25| -b| = |a^2 - 25| -b$
So $|a^2 - 25| = 0$
So $a^2 = 25$ and $a = \pm 5$.
$b \le 0$ and can be any such value.
You can not prove it is BOTH $b < 0$ and $b = 0$.  That would be a contradiction but $b < 0$ OR $b = 0$.  This are individually both possible. and any possible value that is either less than or equal to 0 is acceptable.
Comment: can't we? if $b = -goo; goo > 0$ then $||a^2 - 25| - b|  + b = |0 - b| + b = |goo| - goo = goo - goo = 0$.
Comment: No...  If $b<0$ then the left hand is clearly $|a^2-25|-b +b=0\implies a=\pm 5\implies 2b=0$.
Comment: Never mind, I'm just confused.  You have it right.
Comment: Um... the equation wasn't $||a^2 - b| - b| +b = 0$.  It was $||a^2 - 25| - b| + b = 0$.
Comment: That was a typo.  But in addition to the typo, I was just wrong.  Been a long day.
Comment: If we had $||a^2 - b| - b| + b = 0$ we'd have $0 \ge b =-||a^2 - b|- b| = -|a^2 -b| +b$ so $ 0 = a^2 - b$ and $ b = a^2 \ge 0$ so $b \le 0$ and $b \ge 0$ so $b = 0 = a^2$.  But that wasn't the equation.
Comment: meh... we all have days like that.  Sign errors.  They were a pain in the second grade.  They are a pain forty years later.
Comment: Between sign errors and off-by-one errors it's a miracle that I ever get anything right.
Answer: If $b>0$, the number
$$
\bigl|\,|a^2-25|-b\bigr|+b
$$
is strictly positive. So if you want it to be $0$, you need $b\le0$.
Suppose $b\le0$ and $\bigl|\,|a^2-25|-b\bigr|=-b$. Then squaring gives
$$
|a^2-25|^2-2b|a^2-25|+b^2=b^2
$$
so either $|a^2-25|=0$ or $|a^2-25|-2b=0$. In the latter case, $2b=|a^2-25|\ge0$, which forces $b=0$ and $|a^2-25|=0$.
Thus the equality holds for $a=\pm5$ and $b\le0$.

Is my hypothesis correct?


Why does the intercept column in model.matrix replace the first factor?

Question: I'm trying to convert my factor column to dummy variables:
str(cards$pointsBin)
# Factor w/ 5 levels ".lte100",".lte150",..: 3 2 3 1 4 4 2 2 4 4 ...

labels <- model.matrix(~ pointsBin, data=cards)

head(labels)

#     (Intercept) pointsBin.lte150 pointsBin.lte200 pointsBin.lte250 pointsBin.lte300
# 741           1                0                0                0                0
# 407           1                1                0                0                0
# 676           1                0                0                1                0
# 697           1                1                0                0                0
# 422           1                0                1                0                0
# 300           1                0                1                0                0

There is no column for the first value of my factor (".lte100"), which is what the first row should be categorized as. How do I get this data back? And what does the Intercept column that seems to be all 1's mean?
Comment: When you have "K" dummy variables then your resulting model will have a.) the intercept term (which is a column of ones) and b.) "K-1" additional columns. The reason is because otherwise the columns of the resulting matrix would not be linearly independent (and, as a result, you wouldn't be able to do **OLS**).
Comment: Why 'not meaningful'? It's the same model with the same goodness of fit, just parameterized in a different way.
Comment: @digitgopher: When you run a regression and end up with a model like this: $\hat{y} = \beta_{0} + \beta_{1}*x_{1}$, you're technically ending up with a model like this: $\hat{y} = \beta_{0}*x_{0} + \beta_{1}*x_{1}$, where this new term $x_{0}$ is always equal to "1" (hence the column of ones). If you were to eliminate this column of ones when running a regular regression, you'd end up with a *biased* model since you'd, in effect, be forcing every single model through the origin.
Comment: @SteveS: In fact R's so friendly that if you try remove the intercept  `- 1` when you have a single categorical predictor represented as a factor (as in this question), it'll assume you don't really mean that & switch to using sum-to-zero coding; which is of course just a different parametrization. Too friendly, if you ask me.
Comment: @SteveS: Thanks. I should have checked: it switches to cell-means coding. It doesn't do what you might expect, which is fit the forced-through-the-origin model you quite rightly warn against (it will do that though, when the column's of numeric type).
Comment: Basically, the idea is that you have one outcome which acts as the baseline with which the others are to be compared. Then the coefficients can be interpreted with respect to this baseline. You *could* create a matrix with only your "K" factors and no intercept by explicitly removing the intercept term in your formula (so your formula would look something like: **y ~ . - 1**) but it won't lead to a particularly meaningful model.
Comment: Basically, **R** is really Statistician-friendly in that goes out of its way to fix the user's errors (like by automatically adding an intercept term). In fact, you could even pass in a matrix with three *identical* columns and instead of throwing back an error (which you would normally be getting if you tried doing linear regression manually via linear algebra, or whatever) **R** will automatically drop the two redundant columns and continue on with its calculations.
Comment: @SteveS Of course the coefficients are not the same since it is a different parameterization, but the column space of the model matrices is the same and hence fitted values and residuals are also the same.
Comment: Check: http://stats.stackexchange.com/questions/16921/how-to-understand-degrees-of-freedom
Comment: @SteveS I wasn't trying to make some "point" - I just tried to point out that removing the intercept and keeping all dummy variables (as illustrated my RUser4512's answer) leads to an equivalent model that is just as good as the model with an intercept and all but one of the dummy variables. As I mentioned, the column span of the model matrices is identical. And a model with all the dummy variables (and no intercept term) is quite meaningful: the coefficients are the estimated means for each factor level. That's not a "bad model".
Comment: @SteveS Thank you, the -1 trick was what I was looking for. However, I think I am missing something elementary - why is the intercept all 1's?
Comment: @Wolfgang: Do the two models share the same coefficients? Nope. Do they return the same standard errors? Nuh uh. But oh, they share the same goodness of fit so they *must* be the same, right?? No, they are not.
Comment: @Wolfgang: You are absolutely missing the point--you're *going out of your way* to make a point which--in the context of the given question--is entirely irrelevant. If you tell a student that both approaches are the same do you know what you'll get?? **Bad Models**. Period.
Comment: @Scortchi, I'm not exactly sure what you mean--my **R** doesn't switch to sum to zero coding. Have an example in mind?
Answer: Consider the following:
require(mlbench)

data(HouseVotes84, package = "mlbench")
head(HouseVotes84)

labels <- model.matrix(~ V1, data=HouseVotes84)
head(labels)

labels1 <- model.matrix(~ V1+1, data=HouseVotes84)
head(labels1)

labels0 <- model.matrix(~ V1+0, data=HouseVotes84)
head(labels0)

labels_1 <- model.matrix(~ V1-1, data=HouseVotes84)
head(labels_1)

The first two commands are identical. The last two commands specifies not to produce the intercept and keeps the two dummy variables produced.
Comment: @digitgopher: See http://www.ats.ucla.edu/stat/r/library/contrast_coding.htm for an explanation of reference-level coding & other coding schemes.
Comment: Thank you - this explains the code I wanted. But it doesn't explain why or help me understand.
Answer: In statistics, when we have a factor variable with $k$ levels, we need to convert it to $k - 1$ indicator variables. We choose one level as the baseline, and then have an indicator variable for each of the remaining levels.
First let me explain why this isn't throwing away any information.
Say there are levels A, B, C, and we have $I_B$ and $I_C$, the indicators for being in level B and C. An individual is in level A if and only if $I_B = 0$ (not in B) and $I_C = 0$ (not in C). So we still have kept track of the individuals in level A. This works for any number of levels.
Now as you note we could code the same information with three indicator variables $I_A$, $I_B$ and $I_C$. The reason why we don't is known as multicollinearity. In short, the columns of the regression matrix won't be linearly independent. This means the matrix $X^T X$ is not invertible, so we can't perform linear regression, as we need to calculate this inverse to calculate the regression estimates $\hat{\beta} = \left(X^T X\right)^{-1}X^T y$.
To take a really simple example, say we have two individuals, one in each of two levels A and B, and an intercept term in our regression.
The regression matrix $X$ will be
\begin{equation}
X =
\begin{bmatrix}
1 & 1 & 0 \\
1 & 0 & 1 \\
\end{bmatrix}
\end{equation}
Then
\begin{align}
X^T X & =
\begin{bmatrix}
1 & 1\\
1 & 0\\
0 & 1\\
\end{bmatrix}
\begin{bmatrix}
1 & 1 & 0 \\
1 & 0 & 1 \\
\end{bmatrix}\\
& =
\begin{bmatrix}
2 & 1 & 1 \\
1 & 1 & 0 \\
1 & 0 & 1\\
\end{bmatrix}
\end{align}
You can then see that adding the second and third columns of $X^T X$ gives the first column, so the inverse can't be computed.
This applies in general the Wikipedia page on multicollinearity gives more of an explanation.

If there is an exact linear relationship (perfect multicollinearity) among the independent variables, at least one of the columns of $X$ is a linear combination of the others, and so the rank of $X$ (and therefore of $X^T X$) is less than $k+1$, and the matrix $X^T X$ will not be invertible.

It is ok to include an indicator for all levels of the factor if you don't include the intercept term, and this encodes the exact same information. (The intercept term minus all other indicators gives the indicator for the baseline factor). But if you have more than one factor in the model this runs into the multicollinearity problem, so you can only include all levels if there is only one factor variable and no intercept term in the model.


Loading of assets via Libraries API

Question: I'm writing a new module which has a dependency on an external library. In Drupal 7 I would use hook_libraries_info giving the name of the library the required assets and Drupal would search the valid locations, libraries directories, to load these assets when required. Drupal 8 introduces *.libraries.yml and after reading https://www.drupal.org/node/1996238 it seems that module developers must now provide a path, relative or absolute, to the required assets.
In my case I will be expecting site builders to download the external library - where should libraries be located in Drupal 8?
Comment: I think the best answer is going to vary depending on frontend library (JS/CSS) versus backend library (PHP).
Answer: hook_libraries_info is not in core, it is from a contrib module:
Libraries API
This module provides the external libraries. It will be ported to drupal 8 and can be used the same way as in drupal 7.
In drupal 8 the libraries, that are defined in *.libraries.yml are seperate and should not be confused with the external libraries, despite their identical name. They are mainly for js and css packages, that are specific to one module, but can also be used by other modules. They can be a replacement for the external libraries, but do not provide the full functionality.
In the long run composer will probably be the way module developers will go for downloading external libraries. But for now this is an ongoing discussion how this can be implemented.
Your reference to https://www.drupal.org/node/1996238 is a seperate issue about the replacement of hook_library_info() in drupal 8 core. This must not be confused with the external libraries defined in hook_libraries_info().

Loading of assets via Libraries API


CollectionView inside of StackView

Question: I have an UICollectionView inside of an UIStackView. The UICollectionViewCell has an UIImageView which is pinned to the cell.
What I'm trying to do here:

Showing the pictures which are picked by the user on the UICollectionView in a line.
The UICollectionView would be horizontally scrollable (Thinking displaying pictures like on Twitter's posting screen.)
The UICollectionView shows up only when the user adds a photo.
Using the UIImagePickerController to pick up photos.

Problems I got:

UICollectionView shows anything even after the user picks a photo to add.
I don't even know it's the problem of code or storyBoard.

What I tried so far. (All attempts had failed.):

Separate the UICollectionView and the UIStackView.
Set the fixed width and height of the UICollectionView, UICollectionViewCell and UIImageView.
Set a specific image of the UIImageView.

Here are my code and storyBoard.
ImageCell.swift
import UIKit

class ImageCell: UICollectionViewCell {
  @IBOutlet weak var cellImage: UIImageView!

  // Avoiding showing cached image
  override func prepareForReuse() {
    cellImage.image = nil
  }
}

ViewController.swift
import UIKit

class ViewController: UIViewController, UINavigationControllerDelegate, UIImagePickerControllerDelegate, UITextFieldDelegate, UITextViewDelegate {

@IBOutlet weak var albumTitle: UITextField!
@IBOutlet weak var pageTitle: UITextField!
@IBOutlet weak var date: UITextField!
@IBOutlet weak var location: UITextField!
@IBOutlet weak var bodyText: UITextView!
@IBOutlet weak var imageCollection: UICollectionView!
@IBOutlet weak var scrollView: UIScrollView!
@IBOutlet weak var frameStack: UIStackView!

var imageStore: ImageStore!
var images = NSMutableArray()
var selectedImage: UIImage?

override func viewDidLoad() {
  super.viewDidLoad()

  // UITextView
  self.bodyText.delegate = self

  // UITextFields
  self.albumTitle.delegate = self
  self.pageTitle.delegate = self
  self.date.delegate = self
  self.location.delegate = self

  // Create close button above the textView keyboard
  // Tool bar
  let closeBar = UIToolbar(frame: CGRect(x: 0, y: 0, width: 320, height: 40))
  closeBar.barStyle = UIBarStyle.default  // Style
  closeBar.sizeToFit()  // Size change depends on screen size
  // Spacer
  let spacer = UIBarButtonItem(barButtonSystemItem: UIBarButtonSystemItem.flexibleSpace, target: self, action: nil)
  // Close Botton
  let closeButton = UIBarButtonItem(barButtonSystemItem: UIBarButtonSystemItem.done, target: self, action: #selector(CreateVC.closeButtonTapped))

  closeBar.items = [spacer, closeButton]
  bodyText.inputAccessoryView = closeBar
}

  override func viewDidLayoutSubviews() {
    scrollView.contentSize = frameStack.bounds.size
    scrollView.flashScrollIndicators()
  }

  // Enable textView to close
  @objc func closeButtonTapped() {
    self.view.endEditing(true)
  }

  // Enable return key of UITextFields
  func textFieldShouldReturn(_ textField: UITextField) -> Bool {
    albumTitle.resignFirstResponder()
    pageTitle.resignFirstResponder()
    date.resignFirstResponder()
    location.resignFirstResponder()
    return true
  }

  // Open up device's camera
  @IBAction func takePicture(_ sender: UIButton) {
    let imagePicker = UIImagePickerController()
    if UIImagePickerController.isSourceTypeAvailable(.camera) {
      imagePicker.sourceType = .camera
    } else {
      imagePicker.sourceType = .photoLibrary
    }
    imagePicker.delegate = self

    // Place image picker on the screen
    present(imagePicker, animated: true, completion: nil)
  }

  // Open up device's camera
  @IBAction func pickPicture(_ sender: UIButton) {
    let imagePicker = UIImagePickerController()
    imagePicker.sourceType = .photoLibrary
    imagePicker.delegate = self
    // Place image picker on the screen
    present(imagePicker, animated: true, completion: nil)
  }

  func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [String : Any]) {
    // Get picked image from info directory
    let image = info[UIImagePickerControllerOriginalImage] as! UIImage

    // Put that image on the screen in the image view
    selectedImage = image
    images.add(selectedImage!)

    imageCollection.reloadData()

    // Take image picker off the screen
    dismiss(animated: true, completion: nil)
  }

}

extension CreateVC: UICollectionViewDelegate, UICollectionViewDataSource {

  func collectionView(_ collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {

    return images.count
  }

  // Assign cells contents
  func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
    let cell = collectionView.dequeueReusableCell(withReuseIdentifier: "Cell", for: indexPath) as! ImageCell

    cell.cellImage.isHidden = true
    if let image = cell.cellImage.image {
      cell.cellImage.isHidden = false
      cell.cellImage.image = images[indexPath.row] as? UIImage
    }

    return cell
  }

}

The Picture of my storyboard. The cell seems weird.
I'm kind of new to iOS development and completely stuck here.
Further finding:
I found the cell size looks weird.
Storyboard says the thing like this.
Cell
Expected: width 0
Actual: width 120
Comment: [updated]Make sure you have set the proper constraint for imageView
Comment: I agree with Matthew, you may not have the cell constraints properly configured. Are it's subviews correctly set up?
Comment: When it comes to the cell, I'm only using UICollectionView's "Cell Size" property because it seems unable to set specific constraints for the cell directly. And I pinned the UIImageView to the cell.
Answer: I figured out the problem. It was the cell's constraints. Here's how I solved this problem.
override func viewDidLayoutSubviews() {
  let layout = UICollectionViewFlowLayout()
  layout.itemSize = CGSize(width: 120, height: 120)
  layout.minimumInteritemSpacing = 0
  layout.minimumLineSpacing = 10
  layout.sectionInset = UIEdgeInsets(top: 0, left: 0, bottom: 0, right: 0)
  layout.scrollDirection = .horizontal
  imageCollection.collectionViewLayout = layout
}

Set the specific height for the CollectionView on the Storyboard
Set the cell's constraints in viewDidLayoutSubviews() method like above

CollectionView inside of StackView


compute confidence interval for indepedent t test

Question: I'm actually reading Andy Feld book and i'm trying to better understand how statistics works and especially t test.
I have two independent groups and i would like to compute 95% confidence interval in order to plot it as error bar on a bar plot graphic in R.
First would like to now my formula is correct. I have computed the standard error for each of my group and, as each group have a size of 12, i think i should refer to a student distribution (with a df of 11) in order to compute the lower and the upper bound and multiply this value with the SE. Is it right?
Thank you for your help

@BruceET thank you very much for your long and detailed comment :)
Maybe my problem wasn't clear enough. I already made a Welsh t-test to compare my two mean. The t test indicates that there is no significant difference between them.
    Welch Two Sample t-test

data:  Anxiety by Group
t = -1.7612, df = 21.246, p-value = 0.0926
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -16.168163   1.334829
sample estimates:
     mean in group Picture mean in group Real Spiders
                  40.00000                   47.41667

What i want to do, is to plot the data and compute the 95% confidence interval as error bar. As you mantioned it, standard error of the mean seems to be a bad way to do it, so i would like to take good habits. :)
So, i want to know wich confidence interval i should plot. For me (but maybe i'm wrong, tis what i want to know), it doesn't make sense to plot the interval given by the Welsh T-test as it is the confidence interval for the distrbution of the mean of the difference between the two groups.
If i understand correclty, i should compute the confidence interval from the distribution of the mean for each of my group.
So for me it is something like this :
lower_bound_real <- - 1.7612 * sd(Real) / sqrt(length(Real))
upper_bound_real <- + 1.7612 * sd(Real) / sqrt(length(Real))

lower_bound_picture <- - 1.7612 * sd(Picture) / sqrt(length(Picture))
lower_bound_picture <- + 1.7612 * sd(Picture) / sqrt(length(Picture))

Is it right?
Also, if it is, i would loke to know how to plot a different error bar for each og the group on a same plot.
Concerning the "Notches", i'm not sure what it is? Is this the "error bar" of the boxplot? Also, it tell smothing about the difference of the medians, but not the means right?
Thank you one again
Comment: Whose CI and SE? CI and SE of means, different of means or t-value?
Answer: If you are comparing two independent samples, then you need a two-sample t test.  Here are descriptive statistics for two samples of size 12, sampled in R from normal populations.
set.seed(123)  # for reproducibility
x1 = round(rnorm(12, 70, 3),2)
x2 = round(rnorm(12, 74, 3),2)
summary(x1); sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  66.20   68.58   70.30   70.58   71.95   75.15
[1] 2.777619
summary(x2); sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  68.10   71.59   72.97   73.36   75.27   79.36
[1] 2.963682

Boxplots illustrate the sample values overlap, but perhaps
not by enough to keep $\bar X_1 = 70.58$ from being significantly different from $\bar X_2 = 73.36.$
(Sample means are shown as solid red dots.)
boxplot(x1, x2, col="skyblue2")
points(1:2, c(mean(x1), mean(x2)), pch=19, col="red")

If you want to test the null hypothesis $H_0: \mu_1 = \mu_2$
that the two population means are the same against the
alternative $H_a: \mu_1 \ne \mu_2$ that they are not, you
might use a pooled 2-sample t test. The P-value of this test is
$0.02684 < 0.05$ so the difference between the sample means
is significant at the 5% level. Notice that the confidence interval for the difference in means does not
include $0$ (its two endpoints have the same sign).
t.test(x1, x2, var.eq=T)

        Two Sample t-test

data:  x1 and x2
t = -2.3723, df = 22, p-value = 0.02684
alternative hypothesis:
    true difference in means is not equal to 0
95 percent confidence interval:
 -5.213393 -0.349940
sample estimates:
mean of x mean of y
 70.58250  73.36417

The $T$-statistic is computed as $T = (\bar X_1 - \bar X_2)/SE,$
where $SE = S_p\sqrt{1/12 + 1/12},$ and the pooled variance estimate
$S_p^2 = \frac{(n_1-1)S_1^2 + (n_2-1)S_2^2}{n_1 + n_2 - 2}
= (S_1^2 + S_2^2)/2$ combines information from the two sample standard deviations. (The second, simplified expression for $S_p^2$ is possible because this is a 'balanced design' with the two sample sizes equal.) This pooled estimate is based on the assumption that the two population variances are equal: $\sigma_1^2 = \sigma_2^2 = \sigma^2,$ so that $S_p^2$ estimates $\sigma^2.$
However, the standard error SE would not necessarily be appropriate for making
error bars in a bar graph. There is some disagreement how best to illustrate statistical significance in a bar graph. The relevant Wikipedia page, although very brief,  makes a couple of suggestions.
I know that such error-bar methods are often used, but overlap of error bars can occur even when means are significantly different. I wonder whether any error-bar method will be a useful graphical description of your data.
Notes:  (1) For larger datasets than yours, one graphical method for showing significant differences between samples is a 'notched' boxplot. Notches (in the sides of boxes) are nonparametric confidence intervals, calibrated to facilitate comparing two samples. If the notches don't overlap, the two sample medians differ significantly at the 5% level.
Especially for small sample sizes, the notches can extend beyond the boxes, giving a result that may be confusing. When notched boxplots are available, they are often superior to barcharts for describing numerical samples.
Here is a notched-boxplot comparison of two samples, each of size 30.
set.seed(123)
x1 = round(rnorm(30, 70, 3),2)
x2 = round(rnorm(30, 74, 3),2)
boxplot(x1, x2, col="skyblue2", notch=T)

(2) The pooled 2-sample t test assumes that the two samples are from populations with equal variances. In practice it is often difficult to know whether variances are equal. The Welch 2-sample t test uses a t statistic
that has approximately Student's t distribution, but typically with a reduced number of degrees of freedom.
Absent solid prior knowledge that population variances are equal, it is good practice to use the Welch test as the default two-sample t test.
Here are results from the  Welch test using the original two samples $(n_1 = n_2 = 12).$ Notice the absence of the parameter var.eq=T in t.test, and in the output, df = 21.908, instead of df = 22 in the pooled test shown above. (The reduction in degrees of freedom is very slight here because we simulated data from populations with equal variances.)
t.test(x1, x2)

        Welch Two Sample t-test

data:  x1 and x2
t = -2.3723, df = 21.908, p-value = 0.02688
alternative hypothesis:
   true difference in means is not equal to 0
95 percent confidence interval:
 -5.2139845 -0.3493488
sample estimates:
 mean of x mean of y
  70.58250  73.36417


Right way of testing for Exceptions

Question: I do TDD on a project with 80% simple logic and 20% complex logic. I find myself testing a lot if certaing methods throws an error and wonder about the right way to do it. I use NUnit and JustMock.
I have two ways of doing this. Using the ExpectedException attribute, and spesifying the type. Or writing as below. The pro with writing as below is that I can also assert the exception.message (if i have made a custom one), and if the test fail i get the exception.message shown too. But I wanted to check with others how you do it. So to sum up:

Is it normal to have a lot of testing for exceptions like this?
Is this the right way to do it:

Just to explain: A supplier offers certain contracts, a Department accepts one contract but can not have more than one contract with the same supplier (but of cource can have different contracts with different suppliers)

    [Test]
    public void Accepting_more_than_one_contract_from_supplier_throws_exception()
    {
        //Arrange
        var department = new Department(Guid.NewGuid(), "1234");
        var supplier = Mock.Create<Supplier>();
        var contract1 = Mock.Create<DeliveryContract>();
        var contract2 = Mock.Create<DeliveryContract>();
        var id = Guid.NewGuid();
        supplier.Arrange(x => x.Id).Returns(id);
        contract1.Arrange(x => x.Supplier).Returns(supplier);
        contract2.Arrange(x => x.Supplier).Returns(supplier);

        //Act
        department.AcceptContract(contract1);

        //Assert
        try
        {
            department.AcceptContract(contract2);
            Assert.Fail("Duplicate contract with supplier did not throw an exception");
        }
        catch (Exception ex)
        {
            Assert.AreEqual(typeof(ArgumentException),ex.GetType(),ex.Message);
        }
    }

Answer: You can use Assert.Throws method:
Assert.Throws<ArgumentException>(() => department.AcceptContract(contract2));

or
Assert.Throws<ArgumentException>(() => department.AcceptContract(contract2), "some message");

Comment: Is this a better way to do it than the above?
Comment: Yes, it's defined in NUnit. Don't you think it's better than the try-catch block?
Comment: yeah, i do think so, but so is the ExceptionAttribute ;) But I do think the try/catch block does show very much of the intensions of the test, and how it is supposed to work. In addition the ability to assert the exception message, which is even more unique. Have had some problems with asserting the wrong argumentexception i.e. if I check for an empty string in another field and that throws an ArgumentException and the test passes on the wrong terms.
Comment: Yeah, ExceptedException is an alternative. It's a matter of choice I guess.
Answer: As @Ufuk said. Use Assert.Throws<T> where T is an exception type.
If you also want to check the exception message use one of this:
T Assert.Throws<T>( TestDelegate code, string message );
T Assert.Throws<T>( TestDelegate code, string message,
        params object[] parms);

By the way the data you created for tests shouldn't be mocks but stubs.
By stubs I mean objects which only contains data for tests, in your case you provide behaviour  for objects - not data.
I'd recommend you use of NBuilder code.google.com/p/nbuilder/. You can create clear stubs by it. It also have great flexibility of creating objects, they can be create randomly or exactly prepared for your test case.
The usage in your case:

var supplierStub = Builder<Supplier>
                .With(supplier => supplier.Id = id)
                .Build();

var contractsStub = Builder<DeliveryContract>.CreateListOfSize(2)
                .All()
                    .With(contract => contract.Supplier = supplierStub)
                .Build();
Comment: It depends on class logic flow. In your example throwing exception isn't obvious and it's part of class logic, so it's reasonable to test it. Remember that tests could be good documentation of your code.

I'm testing code for exceptions mostly when I throw them directly in code (by throw) or when program could throw an exception in some cases.

For me good example is testing servis responsible for reading .xml file f.e. by LinqToXml. Good case to test is how the service will behave if one node is missing.
Comment: Thanks for the advice on NBuilder. Only one question, do you find yourself doing a lot of these kind of tests or is it bad code to test for exceptions like this?

Right way of testing for Exceptions


Needed: Complex SQL Query for Recategorizing Output

Question: I need help to create a SQL Query that outputs data in a specific format.
Let's assume I have two tables with data as shown:
Table 1:
Name
A
B
C

Table 2:
Name        Address          Category
A           123 One St.      Type1
A           123 One St.      Type1
A           123 One St.      Type1
A           123 One St.      Type2
B           222 Two St.      Type2
B           222 Two St.      Type2
C           77 Three St.     Type1
C           77 Three St.     Type3
C           77 Three St.     Type3

I need to generate output in the following format:
Name        Address        Type1   Type2  Type3
A           123 One St.    TRUE    TRUE   FALSE
B           222 Two St.    FALSE   TRUE   FALSE
C           77 Three St.   TRUE    FALSE  TRUE

I cannot seem to figure this out. Any help would be greatly appreciated!
My attempt only gets partway:
    select t1.Name, Category="Type1" as Type1, Category="Type2" as Type2,
           Category="Type3" as Type3
    from Table2 t1
    order by Name

This falls way short of the goal.
Comment: Have you [**tried anything**](http://meta.stackoverflow.com/questions/261592) so far?
Answer: Seems like you're looking for conditional aggregation and an inner join.
SELECT t2.name,
       t2.address,
       CASE
         WHEN count(CASE t2.category
                      WHEN 'Type1' THEN
                        1
                    END) > 0 THEN
           'TRUE'
         ELSE
           'FALSE'
       END type1,
       CASE
         WHEN count(CASE t2.category
                      WHEN 'Type2' THEN
                        1
                    END) > 0 THEN
           'TRUE'
         ELSE
           'FALSE'
       END type2,
       CASE
         WHEN count(CASE t2.category
                      WHEN 'Type3' THEN
                        1
                    END) > 0 THEN
           'TRUE'
         ELSE
           'FALSE'
       END type3
     FROM table1 t1
          INNER JOIN table2 t2
                     ON t2.name = t1.name
     GROUP BY t2.name,
              t2.address;

Comment: Spectacular answer. This hit the mark perfectly. I added:

`ORDER BY t2.name`

and this is spot on 100% working.


Same dates for publish and post

Question: I am using the published date module.
The posted date, and published date are the same for the nodes. Does anyone know any reason why that is so?
Comment: By default in Drupal node has two dates - created and changed. Can you provide screenshot to get know about what "published" date are you spoken about. THis date can be added by some module.
Comment: I am not familiar with this module, but i think that published date is equal to post date by default to make node published at once if you don't specified any custom published date. I advice you to check settings of that module - maybe you can setup any other date to be default value.
Comment: May you provide the link to the project page on Drupal.org for the module you are using? It can help with answering the question.
Answer: The publish date can change, like after unpublishing and republishing some content. The post date is rather fixed, it's the date the content is created...
Comment: @CrawleyObara: altering a core module? Really bad idea.
Comment: What I wanted was to have several dates on the same node: when the node was initially created and when the node was published. The simplest was was to edit the node.module file. Change the line ($variables['date']      = format_date ($node->created);
  $variables['name']      = theme('username', array('account' => $node));)to( $variables['date']      = format_date ($node->published_at);
  $variables['name']      = theme('username', array('account' => $node));"
Comment: Sk8erPeter I had no other option.

Same dates for publish and post


Clarifying the homomorphism's definition?

Question: So, I've heard a lot that homomorphism is a structure-preserving mapping between somewhat algebraic objects. Rigorous definition is:
$f(x \circ y) = (f x) \circ (f y)$.
And it's not clear enough for me why operation from the left (denoted as $\circ$) remains unchanged on the right? Does it mean that homomorphism suppose to work only with algebraic objects sharing the same operation? Or this definition is just not enough explicit, shadowing the fact that $\circ$ gets mapped as well to a, let say, $\bullet$ and those are different in general?
Intuitively it feels like $f(x \circ y) = (f x) \bullet (f y)$ would be more correct.
Comment: you are right. An homomorphism preserves the same kind of operations between algebraic structures, but these operations are not the same generally.
Comment: Of course, the domain and codomain of the homomorphism *could* be the same object, in which case the the operations on the left and right side are the same - and then we call the homomorphism an endomorphism.
Comment: Related, possibly helpful: https://math.stackexchange.com/questions/2039702/what-is-an-homomorphism-isomorphism-saying/2039715#2039715
Answer: And your intuition is correct. Usually, the homomorphism is between two algebraic structures with distinct operations.
Answer: Does it mean that homomorphism suppose to work only with algebraic objects sharing the same operation?
No. It means that, when working with algebraic objects, we usually share the same symbol for different operations.
Herstein's book (p. 67) explains this issue in the context of groups:

Definition. Let $G, G'$ be two groups; then the mapping $\varphi:G\to G'$ is a homomorphism if $\varphi(ab)=\varphi(a)\varphi(b)$ for all $a, b\in G$.
In this definition the product on the left side - in $\varphi (ab )$ - is that of $G$, while the product $\varphi(a)\varphi(b)$ is that of $G'$.

Analogously, for a general homomorphism $f(x \circ y) = (f x) \circ (f y)$, the symbol $\circ$ on the left stands for the operation in the domain, while the symbol $\circ$ on the right stands for the operation in the codomain.
Answer: Yes, you are right. Take, for instance, $A=B=\mathbb{R}$, and take the usual sum ($+$) on $A$ and the usual product ($\cdot$) on $B$. Then a homomorphism between the two structures $(A,+)$ and $(B,\cdot)$ is a function $f:A\longrightarrow B$ such that for each $x,y\in A$ we have
\begin{equation}
f(x+y)=f(x)\cdot f(y)
\end{equation}
An example of such function is the exponential, $f(x)=e^x$.
Answer: The theory of your algebraic structure introduces a language for expressing calculations. For example, the theory of a ring introduces symbols $0,1,+,-,\cdot$, and we have a language using those symbols to express calculation.
Usually, we express things in this language unless there is a compelling reason not to do so.
Among the things ring homomorphisms must satisfy is requires $f(x \cdot y) = f(x) \cdot f(y)$. When you read that equation, you should read both instances of $\cdot$ as referring to the operation given in the theory of a ring; in this sense, both sides refer to the "same operation".
What's different about the two sides is the interpretation of the operation:

In $f(x \cdot y)$, one interprets the operation in accordance with the domain of $f$
In $f(x) \cdot f(y)$, one interprets the operation in accordance with the codomain of $f$

The (correct!) idea you are expressing is that these interpretations may be completely different functions between sets.
IF you were in a situation where you really did have cause to express arithmetic in terms of names for the specific functions that serve as the interpretations, then if the interpretation of $\cdot$ in the domain is a function we call $\circ$, and on the codomain is a function we call $\bullet$, then you would indeed translate $f(x \cdot y) = f(x) \cdot f(y)$ into the expression $f(x \circ y) = f(x) \bullet f(y)$.
Answer: You're right. It's an abuse of notation. More formally, you should have something like the following.
Suppose $(G,\oplus_G)$ and $(H,\oplus_H)$ are groups. Then $f:G\to H$ is a homomorphism if and only if $$f(x \oplus_G y ) =f(x) \oplus_H f(y)$$ for all $x,y\in G$.
The subscript is often dropped and inferred from context -- that's the "abuse".


SQL Server: rows to columns

Question: sample datas
 date     | num | name | value
 01012012 |  1  |    A |  20
 01012012 |  1  |    B |  30
 01012012 |  2  |    C |  40

wish to  like this
 date     | num |  A |  B |  C
 01012012 |   1 | 20 | 30 | --
 01012012 |   2 | -- | -- | 40

name is not fixed can allow many names
Answer: How about something like
DECLARE @Table TABLE(
        date DATETIME,
        num INT,
        name VARCHAR(20),
        value FLOAT
)
INSERT INTO @Table SELECT '20121201',1,'A',20
INSERT INTO @Table SELECT '20121201',1,'B',30
INSERT INTO @Table SELECT '20121201',2,'C',40

SELECT  *
FROM    (
            SELECT  date,
                    num,
                    name,
                    value
            FROM    @Table
        ) t
PIVOT   (
            SUM(Value) FOR name IN ([A],[B],[C])
        ) p

SQL Fiddle DEMO
Answer: For dynamic columns you need to use dynamic SQL
SQL FIDDLE EXAMPLE
declare
    @cols nvarchar(max),
    @stmt nvarchar(max)

select @cols = isnull(@cols + ', ', '') + '[' + Name + ']' from table1

select @stmt = '
    select *
    from table1 as T
        pivot
        (
            max(T.value)
            for name in (' + @cols + ')
        ) as P'

exec sp_executesql  @stmt = @stmt

If you don't need a dynamic number of columns, you can use plain SQL like
select *
from table1 as T
    pivot
    (
        max(T.value)
        for name in ([A],[B],[C])
    ) as P

SQL Server: rows to columns


how i can declare a variable and access it inside 2 functions in python

Question: In Python, I am new with python, I don't know about it , started a week ago. I want to count how many times I am executing fuction1.
<code>       the_list = ["1","2","3"]

       for i in the_list:
           print(i)

           function1(i)

   def function1(the_list):
       the_list2 = ["a","b"]
       count = 0

        "''here I i am defining the count so the value is
       getting reset whever it is exiting for loop"""

        for j in the_list2:
           print(j)
           count +=1
       print(">>",count)
       #here i wanna count how manny times we are running this print statment?

   function()```
</code>
Answer: You need to define your counter as a global variable.  Honestly the better way to do this is to use Python Decorators and just decorate your function.  But essentially you are doing this.
<code>count = 0

def example():
    global count
    count+=1

def example2():
    global count
    count+=1

example()
example2()
print(count)
</code>
Comment: You are overcomplicating it.  Just add the counter to the print function.
Comment: Not everyone is smart as you Sir


Uniform convergence of $\sum (-1)^nf_n(x)$ on $[0,1]$ where $f_n(x)=x^n(1-x)$.

Question: Let $f_n(x) = x^n(1-x)$ and $\sum (-1)^nf_n(x)$.
I showed that this series point wise.
Case1) $x=1$
$$ f_n(1)=0 \, , \sum{(-1)^nf_n(1)} = 0 $$
Case2) $x\neq 1$
$$|f_n(x)| \leq |x|^n$$
since $$\sum |x|^n$$ converges, By abosultely convergese test $\sum (-1)f_n(x)$ converges.
I want to show that this series uniformly converges on $[0,1]$.
Answer: Hint. One may recall that
$$
\sum_{n=0}^Nx^n=\frac{1-x^{N+1}}{1-x}, \quad x\neq1,
$$ giving
$$
\sum_{n=0}^N(-1)^nx^n=\frac{1+(-1)^Nx^{N+1}}{1+x}, \quad 0\leq x\leq1,
$$ then

$$
\left|\sum_{n=0}^N(-1)^nx^n-\frac{1}{1+x}\right|\leq\frac{|x|^{N+1}}{1+x}\leq |x|^N, \quad 0\leq x<1
$$

may help to conclude.
Comment: Eventually, This problem reduce to show that $\sum (-1)^nx^n(1-x)$ uniformly converges on $(0,1)$. Your Answer is very useful. Thank you.


I want to insert data from a single form into multiple table both table contains relations

Question: I am new to php and trying to learn. I m also trying to make an web application "school management system".
I am getting problem while i am inserting records in student and parents table from the same form, the form is ok and its inserting record in student table but not in parents table.
in parents and student table student_id is common and in student table student_id contain primary key and in parents table it contain foreign key.
my code is given below:

<?php>
     if(isset($_POST['submit'])){
 $reg = $_POST['reg_no'];
 $s_name = $_POST['student_name'];
 $s_father = $_POST['student_father'];
 $p_last_name = $_POST['parent_lastname'];
 $s_birth = $_POST['student_birth'];
 $p_phone = $_POST['parent_phone'];
 $p_address = $_POST['parent_address'];
 $school_name = $_POST['school_name'];
 $batch = $_POST['session_batch'];
 //filtering variables
     $reg_no = mysql_real_escape_string($reg);
 $student_name = mysql_real_escape_string($s_name);
 $student_father = mysql_real_escape_string($s_father);
 $parent_last_name= mysql_real_escape_string($p_last_name);
 $student_birth = mysql_real_escape_string($s_birth);
 $parent_phone = mysql_real_escape_string($p_phone);
 $parent_address = mysql_real_escape_string($p_address);
 $school = mysql_real_escape_string($school_name);
 $batch = mysql_real_escape_string($batch);
     //connecting to db by including db file
     include_once('include/dbconnect.php');

        $db_select = mysql_select_db($server_db_name,$db_connect);
        if ($db_connect)
        {
$student_query = "INSERT INTO students (school_id, session_id, student_name,
    student_father, student_birthdate, registration_no) VALUES
    ('$school','$batch','$student_name','$student_father','$student_birth','$reg_no')";
    $s_query = mysql_query($student_query) or DIE ("error. while inserting records in
    student");
    /* here im trying to select student_id which is inserted above to insert data in
      parents table*/
     $id_query = mysql_query("SELECT * FROM students WHERE student_id = $student_name
     LIMIT 1") or DIE ("Could complete the id query");
     while ($id_result = mysql_fetch_array($id_query))
     { $s_id = $id_result['student_id'];
           $parent_query = "INSERT INTO parents (school_id, student_id, parent_name,
             parent_lastname, parent_phone, parent_address)
          VALUES('$school','$s_id','$student_father','$parent_last_name',
           '$parent_phone','$parent_address')";
        $p_query = mysql_query($parent_query);
            if (!$parent_query) { echo "error. while inserting records in student"; }
              }
            mysql_close($db_connect);
    header('location:admin.php?student');
    }
    else {
    echo "Error While Connecting to server";
    }
              }else {
            header('location:admin.php?error');
         }
         ?>

Comment: am i doing this in correct way?
Comment: the records are not inserting in parents table
Comment: i want the student_id also inserted into the parents table.
Comment: SELECT * FROM students WHERE student_id = $student_name ...  a u sure?
Comment: hm. nice check. i should changed it to student_name, let me chk now
Comment: same error "Could complete the id query" the query is not selecting the id. i tried SELECT * FROM students WHERE student_name = $student_name
Comment: I found my mistake i forget to add apostrophe with $student_name
Answer: find id u Last inserted,  use mysql_insert_id()
<?php
mysql_connect("localhost", "mysql_user", "mysql_password") or
    die("Could not connect: " . mysql_error());
mysql_select_db("mydb");

mysql_query("INSERT INTO mytable (product) values ('kossu')");
printf ("Last inserted record has id %d\n", mysql_insert_id());
?>

Answer: Like JOE LEE has mentioned in above snippet,
$generated_key=mysql_insert_id();

// this line of code actually returns the auto generated id, of course I am guessing you want the auto incremented value from the student table for the parent table.


Classic ASP in the Feature View

Question: Am I missing something here? I am trying to configure Classic ASP on Windows 2008 Server SP1 and all the documentation points to an ASP option in the features view of IIS7.
I've got IIS7 open, Classic ASP is enabled but for the life of me I can't find a ASP option in the Features view.
Answer: Are you sure you have ASP configured as a role on the server?  Go to Server Manager, Roles and verify that Application Development -> ASP is installed.
Comment: Yes. ASP == Installed
Answer: Installed Windows 2008 SP 2 and now ASP Feature is showing up.

Classic ASP in the Feature View


How do I append a suffix to ids in a HTML file using standard unix commands?

Question: Assume I have an HTML file like this:

       content of div a

 content of div b

 content of div c

I want to append a unique suffix (say, -suffix) to all ids, which would include attributes id="..." and selectors #..., and result in a file like this:

       content of div a

 content of div b

 content of div c

How do I accomplish this with standard unix shell tools like sed, grep, awk in a way that would cover as many situations as possible?
My attempt:
I came up with the following sed command:
sed -e 's/id="\([-_a-zA-Z0-9]*\)"/id="\1-suffix"/g;s/#\([-_a-zA-Z0-9]*\)/#\1-suffix/g' index.html

Which is actually two commands in one:

s/id="\([-_a-zA-Z0-9]*\)"/id="\1-suffix"/g - substitutes id attributes id="..."
s/#\(\[-_a-zA-Z0-9]*\)/#\1-suffix/g - substitutes id selectors #...

However it's far from perfect. First, it only supports double attribute values in double quotes id="..." and id values are limited in that they have to match [-_a-zA-Z0-9]*. Second, this clashes with hex colors, so a white color like #ffffff would get a suffix #ffffff-suffix; An id selector like #... should only get a suffix if an appropriate attribute id="..." exists.
What is the best way to accomplish this?
Comment: I suggest to use xmlstarlet.
Comment: To avoid the clash with hex colors you could try matching a the beginning of line, some space and then the id.  `s/^[ \t]*#\([-_a-zA-Z0-9]*\)/#\1-suffix/g`.  (Beware: I haven't tested this regex.)
Answer: There are a lot of cases in your file, as you mentionned with the colour problem
My approach would be to treat the file line by line using
cat inputfile.html | while read a; do
some code
echo "$a" >> outputfile.html
done

This being said, you may use
b=$(expr "$a" : "regex")

To precisely filter what you want to modify and only then, use some
sed

on $b to get what you want and push $b into $a


Selecting patterns in character sequence using regex

Question: I would need to select all the accounts were 3 (or more) consecutive characters are identical and/or include also digits in the name, for example
Account
aaa12
43qas
42134dfsdd
did

Output
Account
aaa12
43qas
42134dfsdd

I am considering of using regex for this: [a-zA-Z]{3,} , but I am not sure of the approach. Also, this does not include the and/or condition on the digits. I would be interested in both for selecting accounts with at least one of these:

repeated identical characters,
numbers in the name.
Comment: Your expected output contradicts your interest `numbers at the end`
Comment: unfortunately I made a mistake in writing it. I meant including numbers in the name, not at the end. Apologies
Answer: Can you try :
x = re.search([a-zA-Z]{3}|\d, string)

Answer: Give this a try
n = 3 #for 3 chars repeating
pat = f'([a-zA-Z])\\1{{{n-1}}}|(\\d)+' #need `{{` to pass a literal `{`
df_final = df[df.Account.str.findall(pat).astype(bool)]

Out[101]:
      Account
0       aaa12
1       43qas
2  42134dfsdd