What effect does data averaging have on the variogram?

Question: What effect does data averaging have on the variogram? To be specific, please see a simple example:
#Simulate a pure random walk data set, call this *PROCESS A*
n <- 1000  #number of data points
t <- 1:n #time
y <- cumsum(rnorm(n)) #data points

# Averaging at every 2 lags, call this *PROCESS B*
t2 <- apply(matrix(t, nrow=n/2, byrow=TRUE), 1, mean)
y2 <- apply(matrix(y, nrow=n/2, byrow=TRUE), 1, mean)

# Compute and plot the empirical variogram
require(geoR)
var <- as.geodata(data.frame(coords = t(rbind(t, rep(1, n))), data.col=y))
vario <- variog(var1)
var2 <- as.geodata(data.frame(coords = t(rbind(t2, rep(1, n/2))), data.col=y2))
vario2 <- variog(var2)
plot(vario$u, vario$v, type='b', pch=16, cex=.7)
points(vario2$u, vario2$v, col=2, type='b', pch=16, cex=.7)

Theoretically $\gamma(h) = \frac{1}{2} Var(y(t) - y(t+h))$ and for a linear variogram of a simple process without the nugget effect this is $\frac{1}{2}\sigma^2 |h|$ where $\sigma^2$ is the variance of the underlying process.  Thus, we expect (1) the two variogram are parallel and (2) the proportion of the nugget effect of PROCESS A to PROCESS B is 4/3. Compare the plots and the Math: (1) is confirmed, but (2) is not. Why? Is this a simple example of the 'change of support' problem in geostatistics?? Thanking you in advance.
Answer: It is difficult to say what you mean by "(2) is not [confirmed]," so let's begin by describing what is going on.
The code begins by generating realizations of $n$ independent standard normal variates $X_t,$ $t=1,2,\ldots, n$, and computing their cumulative sum $Y_t = \sum_{s=1}^t X_t$.  Therefore the variogram of $Y$ is
$$\gamma_Y(t, h) = \frac{1}{2}\mathbb{E}\left(\left(Y_{t+h} - Y_t\right)^2\right)
=  \frac{1}{2}\mathbb{E}\left(\left(\sum_{s=t+1}^{t+h}X_s\right)^2\right) = \frac{h}{2}$$
for $h=0, 1, 2, \ldots, n-t$ and $t=0, 1, 2, \ldots, n.$ Notice that $Y$ has no nugget for any value of $t$: $\gamma(t,0) = 0$ for all relevant $t$.
The code next generates data from a derived process y2 which I will call $Z$, where
$$Z_{t + 1/2} = \frac{Y_{t} + Y_{t+1}}{2} = Y_{t} + \frac{1}{2}X_{t+1}$$
for $t=1, 3, 5, ..., 2\lfloor\frac{n}{2}\rfloor - 1.$
The variogram of $Z$ therefore is
$$\gamma_Z(t+1/2, h) = \frac{1}{2}\mathbb{E}\left(\left(Z_{t+1/2+h} - Z_{t+1/2}\right)^2\right) \\
=  \frac{1}{2}\mathbb{E}\left(\left(\frac{1}{2}X_{t+1}+\sum_{s=t+2}^{t+h}X_s + \frac{1}{2}X_{t+h+1}\right)^2\right) = \frac{h - 1/2}{2}$$
for $h=2, 4, 6, \ldots$ and $t=1, 3, \ldots.$  When $h=0$ the variogram is obviously $0$.  Thus the second plot (of an estimate of $\gamma_Z$ based on the single realization of $Z$) should, on the average, be approximately $1/4$ lower than the first plot (of an estimate of $\gamma_Y$).  This difference is too small to detect reliably in the plots, which is why they look almost identical.  For instance, re-running the R script after executing
set.seed(17)
n <- 10^4

for reproducible results with a larger realization and computing
mean(vario$v - vario2$v)

results in an estimate $0.22$ for this difference of $1/4$.  More runs of the code (without resetting the seed) yield $0.43, 0.19, 0.26, 0.17, -0.01, 0.76, 0.23, 0.26, 0.08$.  The mean of all 10 iterations is $0.258$ with a standard error of $0.067$: significantly different from zero but indistinguishable from $1/4$, consistent with the foregoing calculations.
This is indeed a (very simple) example of change of support, which uses the same kinds of calculations to relate the variogram of a process $Y$ to the variogram of a related process obtained from the convolution of $Y$ with some (fixed, deterministic) kernel $K$.  In this case the kernel is
$$K(-1/2)=K(1/2)=1/2.$$
Comment: Any geostats text that covers change of support should explain this. One of the oldest (and still useful) is Journal & Huijbregts, *Mining Geostatistics*. Cressie obtains a general result for block averages in *Statistics for Spatial Data.*  The kernel convolution formulation of the concept of support appears in the introduction to Diggle & Ribeiro Jr., *Model-based Geostatistics.*
Comment: I'm unsure what proportion you are taking, but similar calculations will enable you to work out the nugget for any process linearly derived from $Y$. In order to discuss this topic well, we would need first to distinguish a true nugget from measurement error: the two will enter differently into the calculations. Diggle & Ribeiro Jr. (*op. cit.*) do a good job of distinguishing these terms.
Comment: Brilliant answer, thanks so much. Could you kindly provide some references on your last point, about the COSP, and kernel convolution
Comment: The variogram can never be negative: it is defined, after all, as the expectation of a squared quantities and squares are never negative. What happens is that the variogram of the convolution with a local kernel is itself not only lowered, but also averaged out locally in some sense. As the lag approaches $0$, the averaging prevents the variogram from becoming negative.
Answer: The variogram is a function of the correlation function, inter alia. The correlation function is affected by data averaging. Check out the Slutsky-Yule effect. This is an old result in time series analysis to the effect that if you do a moving average on white noise, you induce autocorrelations into the transformed series. Visually, the moving averaged white noise appears to the imaginative to have quasi-periodic effects. It is tempting to take a moving average of the data in the hope of allowing genuine features to appear, but Slutsky-Yule warns you that the averaged data will have features that are an artifact of the smooth. I would imagine that a similar thing would happen to the variogram of spatial data.


From metric to topological vector space

Question: Suppose that $E = C[0,1]$ and suppose we have a metric given by
$$d(f,g) = \int_0^1 \min(|f(x)-g(x)|,1)dx$$
Why is it that the topology defined by this metric makes $E$ into a topological vector space?
On the same topic, suppose that $\tau_p$ is defined by the seminorms $f \mapsto |f(x)|, x \in [0,1]$, why is the mapping $$\iota: (E, \tau_p) \rightarrow (E,d) \quad f \mapsto \iota(f)$$
is sequentially continous? To put it in other words, if a sequence in $\tau_p$ convergent it is $d$ convergent.
Also, why is $\iota$ not continuous?
Comment: This is not a metric on $C[0,1].$ consider $f(x)=x^2, g(x)=x^3.$ Then $d(f,g)=0$ while $f\not=g.$
Comment: This is not true, because for those $f(x),g(x)$ we have $\min (|f(x)-g(x)|) = f(x) - g(x)$ which does not integrate to zero on $[0,1]$.
Comment: what do you mean by $\min(|f(x) - g(x)|)$? I mean, it's just $\min(a) = a$
Comment: I see that i made a mistake in the definition. But I fixed it!
Comment: You should first convince yourself that convergence w.r.t. $d$ is the same as convergence in measure. Then, note that convergence w.r.t. $\tau_p$ is pointwise convergence. Using dominated convergence, you can show that this implies convergence w.r.t $d$ **for sequences**, so that $\iota$ is sequentially continuous.
Answer: The topology is generated by sets of the form
$$
U_{f,r}=\{g\in C[0,1]: d(\,f,g)<r\}=f+\{g\in C[0,1]: d(0,g)<r\}.
$$
Note that
$$
U_{0,r}+U_{0,r}\subset U_{0,2r}.
$$
It suffices to show that
$$
(\lambda,f)\longmapsto \lambda f,\quad\text{and}\quad
(f,g)\longmapsto f+g,
$$
are continuous. with respect to above topology.
a. If $f+g\in V$, $V\subset C[0,1]$ open, then $f+g\in U_{f+g,r}\subset U$, for some $r>0$, and
$$
U_{f,r/2}+U_{g,r/2}\subset U_{f+g,r}\subset V,
$$
and hence $(f,g)\mapsto f+g$ continuous.
b. Continuity of $(\lambda,f)\mapsto \lambda f$ is also straight-forward, starting first with the case $\lambda=0$.
Comment: And even if it were equivalent to the $L^2$ metric, this doesn't answer the question (why does this metric or the $L^2$ one make it into a topological vector space).

From metric to topological vector space


If $A$ and $B$ are $n\times n$ matrices, prove that $|(A^TB)|^2\leq|A^TA||B^TB|$; when is this an equality?

Question:
Let $A$ and $B$ be square $n$-matrices. Prove that $|(A^TB)|^2\leq|A^TA||B^TB|$. Also, under what circumstances are the left and the right side equal?

I've tried multiple times, both sides should definitely be positive and since $|A^TA|=|AA|$ and $|AA|=|A||A|$ and so on, I would only get that $|A||A||B||B|=|A||A||B||B|$, which is obviously wrong. The only problem I see is that I don't understand why $(A^TB)$ is in parentheses and $A^TA$ and $B^TB$ aren't.
Comment: This looks like the Cauchy Schwarz inequality to me.
Comment: What norm $|\cdot|$ are you using on these matrices?
Comment: Determinants. Also, I shall look into the Cauchy Schwarz inequality.
Comment: Determinants? If you are using the operator norm coming with the Euclidean norm, it is true by Cauchy-Schwarz.
Comment: @AlexOlssen Indeed, since $\det (A^TB)$ defines an inner product in the space of square matrix. It seems that the OP forgot the $\det$ or something like that.
Answer: If $|X|$ means the determinant of a matrix $X$ (as noted in your comment), then as you said, $|A^TB|^2=\left(|A||B|\right)^2=|A|^2|B|^2=|A^TA||B^TB|$. So, equality always holds. It's just a matter of taste to put $A^TB$ inside a pair of parantheses; I'm surprised that this could be a source of confusion.
Note, however, that as the others suggest, the original problem is perhaps to show that $\|A^TB\|^2\le\|A^TA\|\|B^TB\|$ for some matrix norm $\|\cdot\|$. Make sure that you haven't misunderstood the question.
Answer: Note, that if you're using the Frobenius norm then $|AB|_F\leq|A|_F|B|_F$.


Why is the dot product non-zero for 90° rotated vector in xy plane?

Question: Let's consider a vector:
$$
\vec V = (x\, y\, z)^T
$$
$\vec W$ is the vector obtained by rotating $\vec V$ about $z-$axis by $90^\circ$.
$$
\vec W = R_z\left(90^{\circ}\right)\left[\begin{array}{l}
x \\
y \\
z
\end{array}\right]=\left[\begin{array}{ccc}
\cos 90^{\circ} & -\sin 90^{\circ} & 0 \\
\sin 90^{\circ} & \cos 90^{\circ} & 0 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{l}
x \\
y \\
z
\end{array}\right]=\left[\begin{array}{ccc}
0 & -1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{l}
x \\
y \\
z
\end{array}\right]=\left[\begin{array}{l}
-y \\
x \\
z
\end{array}\right]
$$
But the dot product of $\vec V$ and $\vec W$ is clearly non zero for arbitrary $z$:
$$
\vec V \cdot \vec W = -xy + xy + z^2 \neq 0
$$
What's wrong with my reasoning?
Comment: What do you think happens if you take for instance  $v = (0,0,1)$?
Answer: If you take an arbitrary vector (in a space with 3 dimensions), and rotate it $90^\circ$ about an arbitrary axis, the result is usually not orthogonal to what you started with. In this particular case, consider the vector $[0,0,1]^T$, and think about what should happen as you rotate it.
You can even see this in your calculations: $\vec V \cdot \vec W = -xy + xy + z^2 = z^2$. That tells you that the clue is the $z$-coordinate of $\vec V$. It doesn't tell you yet whether it is your calculations that are wrong, or your conceptions, but it tells you exactly where the two are different, and thus why I and Hyperplane above tell you to use the exact same example vector.
Comment: What is the procedure to get the perpendicular vector to that then?
Comment: There isn't a "**the** perpendicular vector". There is a whole plane of them to choose from (for instance, if your initial vector is along the $z$ axis, you can choose any vector in the $xy$-plane). And it's even been proven (the so-called "hairy ball theorem") that in odd dimensions like here, there isn't really a "nice" consistent and continuous way of doing it, the way for instance $(x, y, z, w)\mapsto (y, -x, w, -z)$ will always work in 4 dimensions.
Comment: What a shame, math always embarasses me even for things that I have always thought are basic. Thank you for your answer


How are Random Forests not sensitive to outliers?

Question: I've read in a few sources, including this one, that Random Forests are not sensitive to outliers (in the way that Logistic Regression and other ML methods are, for example).
However, two pieces of intuition tell me otherwise:

Whenever a decision tree is constructed, all of the points must be classified.  This means that even outliers will get classified, and hence will affect the decision trees where they were selected during boosting.
Bootstrapping is a part of how a RandomForest does sub-sampling.  Bootstrapping is susceptible to outliers.

Is there any way to reconcile my intuition about its sensitivity to outliers, with sources that disagree?
Comment: Some statisticians obtain a tunnel vision on those inliers, that one can predict and understand. Cherish the outliers as 'known unknowns' and wonder if your business model is fragile towards them. Some outliers are fundamentally unpredictable, but their impact is very real ... a paraphrase of N. Taleb's, 'Black Swan'
Comment: The answer, below, is very good. The intuitive answer is that a decision tree works on splits and splits aren't sensitive to outliers: a split only has to fall anywhere between two groups of points to split them.
Comment: So I suppose if the `min_samples_leaf_node` is `1`, then it could be susceptible to outliers.
Comment: yes min_samples and bootstrap sample can completely remove the influence of 1b outliers in RF regression
Answer: Your intuition is correct. This answer merely illustrates it on an example.
It is indeed a common misconception that CART/RF are somehow robust to outliers.
To illustrate the lack of robustness of RF to the presence of a single outliers, we can (lightly) modify the code used in Soren Havelund Welling's answer above to show that a single 'y'-outliers suffices to completely sway the fitted RF model. For example, if we compute the mean prediction error of the uncontaminated observations as a function of the distance between the outlier and the rest of the data, we can see (image below) that introducing a single outlier (by replacing one of the original observations by an arbitrary value on the 'y'-space) suffices to pull the predictions of the RF model arbitrarily far away from the values they would have had if computed on the original (uncontaminated) data:
<code> library(forestFloor)
library(randomForest)
library(rgl)
set.seed(1)

X = data.frame(replicate(2,runif(2000)-.5))
y = -sqrt((X[,1])^4+(X[,2])^4)
X[1,]=c(0,0);
y2<-y
rg<-randomForest(X,y)   #RF model fitted without the outlier
outlier<-rel_prediction_error<-rep(NA,10)

for(i in 1:10){
    y2[1]=100*i+2
    rf=randomForest(X,y2)   #RF model fitted with the outlier
    rel_prediction_error[i]<-mean(abs(rf$predict[-1]-y2[-1]))/mean(abs(rg$predict[-1]-y[-1]))
    outlier[i]<-y2[1]
}
plot(outlier,rel_prediction_error,type='l',ylab="Mean prediction error (on the uncontaminated observations) \\\ relative to the fit on clean data",xlab="Distance of the outlier")
</code>

How far? In the example above, the single outlier has changed the fit so much that the mean prediction error (on the uncontaminated) observations is now 1-2 orders of magnitude bigger than it would have been, had the model been fitted on the uncontaminated data.
So it is not true that a single outlier cannot affect the RF fit.
Furthermore, as I point out elsewhere, outliers are much harder to deal with when there are potentially several of them (though they don't need to be a large proportion of the data for their effects to show up).
Of course, contaminated data can contain more than one outlier; to measure the impact of several outliers on the RF fit, compare the plot on the left obtained from the RF on the uncontaminated data to the plot on the right obtained by arbitrarily shifting 5% of the responses values (the code is below the answer).

Finally, in the regression context, it is important to point out that outliers can stand out from the bulk of the data in both the design and response space (1). In the specific context of RF, design outliers will affect the estimation of the hyper-parameters. However, this second effect is more manifest when the number of dimension is large.
What we observe here is a particular case of a more general result. The extreme sensitivity to outliers of multivariate data fitting methods based on convex loss functions has been rediscovered many times. See (2) for an illustration in the specific context of ML methods.
Edit.
Fortunately, while the base CART/RF algorithm is emphatically not robust to outliers, it is possible (and quiet easy) to modify the procedure to impart it robustness to "y"-outliers. I will now focus on regression RF's (since this is more specifically the object of the OP's question). More precisely, writing the splitting criterion for an arbitrary node $t$ as:
$$s^∗=\arg\max_{s} [p_L \text{var}(t_L(s))+p_R\text{var}(t_R(s))]$$
where $t_L$ and $t_R$ are emerging child nodes dependent on the choice of $s^∗$ ( $t_L$ and $t_R$ are implicit functions of $s$) and
 $p_L$ denotes the fraction of data that falls to the left child node $t_L$ and $p_R=1−p_L$ is the share of data in $t_R$. Then, one can impart "y"-space robustness to regression trees (and thus RF's) by replacing the variance functional used in the original definition by a robust alternative. This is in essence the approach used in (4) where the variance is replaced by a robust M-estimator of scale.

(1)  Unmasking Multivariate Outliers and Leverage Points.
Peter J. Rousseeuw and Bert C. van Zomeren
Journal of the American Statistical Association
Vol. 85, No. 411 (Sep., 1990), pp. 633-639
(2) Random classification noise defeats all convex potential boosters.   Philip M. Long and      Rocco A. Servedio (2008). http://dl.acm.org/citation.cfm?id=1390233
(3) C. Becker and U. Gather (1999). The Masking Breakdown Point of Multivariate Outlier Identification Rules.
(4) Galimberti, G., Pillati, M., & Soffritti, G. (2007). Robust regression trees based on M-estimators.
Statistica, LXVII, 173–190.

<code>    library(forestFloor)
    library(randomForest)
    library(rgl)
    set.seed(1)

    X<-data.frame(replicate(2,runif(2000)-.5))
    y<--sqrt((X[,1])^4+(X[,2])^4)
    Col<-fcol(X,1:2) #make colour pallete by x1 and x2
    #insert outlier2 and colour it black
    y2<-y;Col2<-Col
    y2[1:100]<-rnorm(100,200,1);    #outliers
    Col[1:100]="#000000FF" #black

    #plot training set
    plot3d(X[,1],X[,2],y,col=Col)
    rf=randomForest(X,y)    #RF on clean data
    rg=randomForest(X,y2)   #RF on contaminated data
    vec.plot(rg,X,1:2,col=Col,grid.lines=200)
    mean(abs(rf$predict[-c(1:100)]-y[-c(1:100)]))
    mean(abs(rg$predict[-c(1:100)]-y2[-c(1:100)]))
</code>
Comment: I have added my earlier comments to my answer. I hope it now does a better job of answering your question!
Comment: Thanks.  What are `p` and `s` in the formula?
Comment: Why are combined outliers (1a+2) bad? In your example, the RF model fit the data structure perfectly, 99,99% OOB MSE. The model structure of the middle land between the two clusters is pretty rough, yes, and more a product of the model than of the data. But, no inference and/or predictions should be in this unknown area, so it does not matter. Absolute robustness toward outliers is inevitably to ignore rare but perhaps important possible events. Most ML algos would by default take a middle ground stance between robustness and 'flexibility' but can be tweaked to increase robustness.
Comment: @user603 Nothing to add to the technical discussion, other than that Heart of Darkness was my favorite book as a kid, with Franz Kafka's The Trial a close second (it might have been first if it had been finished and perhaps written in English - then again, maybe it was fitting that it was not finishedd). I didn't really think about it from the outlier angle, other than that I considered Heart of Darkness and The Trials as (good) outliers among the sea of BS "serious literature" I had to read and do BS light-dark imagerry and the like analysis on.
Comment: Thanks for your detailed answer.  If there are several outliers in the same high dimensional space, it begs the question what is our criteria for calling an "outlier"?  In that case, I wonder what hyper parameters may be set so that I can specify some kind of criteria for an outlier a priori?
Comment: Right, so like I'm trying to say, I guess it's a matter of definition of "outlier".  How do you build a definition of an "outlier" into the hyperparameters of an RF algorithm?
Comment: @SorenHavelundWelling: Thanks! fixed the example to show that the outliers can make the mean absolute error arbitrarily large.
Comment: When I first pondered this question, I assumed that the way an RF treats outliers would be the same regardless of classification or regression.  I'm actually interested in how this would be different if, say, I were using RF for binary classification.
Comment: If the response is binary, the only type of outliers that matter are those on the design space. Outliers on the design are not handled by the robustification strategy outlined in the edit of my answer and I do not know how to modify RF to handle them.
Comment: Is there any reason that in your code, @user603, that you eliminate the last value in the array in this line: `rel_prediction_error[i]<-mean(abs(rf$predict[-1]-y2[-1]))/mean(abs(rg$predict[-1]-y[-1]))`.  Why do you use `[-1]` to eliminate the last value?
Comment: @Hunle: yes y[1] is the outlier. rel_prediction_error is the vector of prediction errors on the *non outlying observations*. This corresponds to the sentence "For example, if we compute the mean prediction error of the uncontaminated observations" in y text.
Comment: This example seems a bit unfair, since the function you use is \-sqrt{x^4+x^4}\ and the outliers are positive.  You say you're measuring the "distance" of the outler - distance from what?
Comment: @Hunle: a) why would this example be unfair? Could you elaborate? I do not understand the relevance of your argument ' the function...the outliers are positive'. b) distance to where the original, uncontaminated observation was. We measure the sensitivity of the RF to the presence of a single outlier in the data by showing that the *whole* fitted RF surface (at all values $\pmb x\in\mathbb{R}^2$) can be affected at will by just moving one of the original data point ($\pmb x_i$) to an arbitrary location in the design space ($\pmb x_i^{\prime}$). The distance is $||\pmb x_i^{\prime}-\pmb x_i||$.
Answer: It is not the Random Forest algorithm itself that is robust to outliers, but the base learner it is based on: the decision tree. Decision trees isolate atypical observations into small leaves (i.e., small subspaces of the original space). Furthermore, decision trees are local models. Unlike linear regression, where the same equation holds for the entire space, a very simple model is fitted locally to each subspace (i.e., to each leaf).

In the case of regression, it is generally a very low-order
regression model (usually only the average of the observations in the
leaf).
For classification, it is majority voting.

Therefore, for regression for instance, extreme values do not affect the entire model because they get averaged locally. So the fit to the other values is not affected.
Actually, this desirable property carries over to other tree-like structures, like dendograms. Hierarchical clustering, for instance, has long been used for data cleaning because it automatically isolates aberrant observations into small clusters. See for instance Loureiro et al. (2004). Outlier detection using clustering methods: a data cleaning application.
So, in a nutshell, RF inherits its insensitivity to outliers from recursive partitioning and local model fitting.
Note that decision trees are low bias but high variance models: their structure are prone to changing upon a small modification of the training set (removal or addition of a few observations). But this should not be mistaken with sensitivity to outliers, this is a different matter.
Comment: I actually considered using a clustering method, as you suggest, for the detection of outliers.  But then, I'm unsure where to apply the clustering.  Should it be applied to `labeled` or `unlabeled` data?  And how would this clustering be achieved on heterogenous data that contains both categorical and numerical features?
Answer: outlier 1a: This outlier has one or more extreme feature values and is placed distant to any other sample. The outlier will influence the initial splits of the trees as any other sample, so no strong influence. It will have low proximity to any other sample, and will only define the model structure in a remote part of feature space. During prediction most new samples are likely not to be similar to this outlier, and will rarely end up in the same terminal node. Moreover decision trees regards features as if they were ordinal(ranking). The value is either smaller/equal to or larger than break point, thus it does not matter if a feature value is an extreme outlier.
outlier 1b: For classification one single sample may be regarded as an outlier, when embedded in the middle of many sample of a different class. I described earlier how a default RF model will get influenced by this one sample of odd class, but only very close to the sample.
outlier 2: This outlier has an extreme target value perhaps many times higher than any other values, but the feature values are normal. A .631 fraction of the trees will have a terminal node with this sample. The model structure will get affected locally close to the outlier. Notice the model structure is affected mainly parallel to the feature axis, because nodes are split uni-variately.
I included a RF-regression simulation of outlier_2. 1999 points drawn from a smooth rounded structure $y=(x_1^4 + x_2^4 )^{\frac 1 2}$ and one outlier with a much higher target value(y=2, $x_1$=0,$x_2$=0). The training set is shown to the left. The learned RF model-structure is shown the the right.

<code>library(forestFloor)
library(randomForest)
library(rgl)
set.seed(1)

X = data.frame(replicate(2,runif(2000)-.5))
y = -sqrt((X[,1])^4+(X[,2])^4)^1
Col = fcol(X,1:2) #make colour pallete by x1 and x2
#insert outlier2 and colour it black
X[1,] = c(0,0);y[1]=2 ;Col[1] = "#000000FF" #black

#plot training set
plot3d(X[,1],X[,2],y,col=Col)

rf = randomForest(X,y)
vec.plot(rf,X,1:2,col=Col,grid.lines = 400)
</code>
EDIT: comment to user603
Yes for extreme outliers on target scale, one should consider to transform target scale before running RF. I added below a robustModel() function which tweaks randomForest. Another solutions would be to log transform before training.
<code>.
##---code by user603
library(forestFloor)
library(randomForest)
library(rgl)
set.seed(1)

X<-data.frame(replicate(2,runif(2000)-.5))
y<--sqrt((X[,1])^4+(X[,2])^4)
Col<-fcol(X,1:2) #make colour pallete by x1 and x2

#insert outlier2 and colour it black
y2<-y;Col2<-Col
y2[1:100]<-rnorm(100,200,1);    #outliers
Col2[1:100]="#000000FF" #black
##---

#function to make models robust
robustModel = function(model,keep.outliers=TRUE) {
  f = function(X,y,lim=c(0.1,.9),keep.outliers="dummy",...) {
  limits = quantile(y,lim)
  if(keep.outliers) {#keep but reduce outliers
  y[limits[1]>y] = limits[1] #lower limit
  y[limits[2]<y] = limits[2] #upper limit
  } else {#completely remove outliers
    thrashThese = mapply("||",limits[1]>y,limits[2]>y)
    y = y[thrashThese]
    X = X[thrashThese,]
  }
  obj = model(x=X,y=y,...)
  class(obj) = c("robustMod",class(obj))
  return(obj)
  }
  formals(f)$keep.outliers = keep.outliers
  return(f)
}

robustRF = robustModel(randomForest) #make RF robust
rh = robustRF(X,y2,sampsize=250)     #train robustRF
vec.plot(rh,X,1:2,col=Col2)          #plot model surface
mean(abs(rh$predict[-c(1:100)]-y2[-c(1:100)]))
</code>
Comment: You write "no other prediction will be affected". If you shift your single outlier to put `y[1]=200` you will see that it single handedly causes the prediction error *on the uncontaminated observations* to jump by a factor of 20!
Comment: @user603 True that,  In such cases the target scale can be transformed monotonically before handed to RF. I added a 'robustModel: makes models robust' to my answer.....of course to predict such random target outlier(s) (type 2) remains impossible, but the remaining model structure do not have to suffer
Comment: The Log transform is [not, in general, a solution against outliers](http://stats.stackexchange.com/a/3530/603) (it merely hides the problem). The robustification of RF you propose is essentially  the approach advocated in Galimberti, G., Pillati, M., & Soffritti, G. (see my answer).  The main difference is that your  ''robustModel" approach has a maximum breakdown point of 25% on the response space (it can withstand 25% or arbitrary 'y'-outliers) whereas theirs has a bdp of 50%. Note that neither approach is robust to outliers in the design space.


Tukey with interaction and extracting letters in multcomp

Question: I've seen similar questions posted elsewhere with no answer, so I thought I'd put this out into the CrossValidated universe.
I am attempting to do a Tukey's post-hoc test on a model that has an interaction. To do this, I have used the interaction() function to create a new variable that combines my two main effects, Habitat (3 levels) and Detritus (2 levels) into one variable.
   reg.invert$interaction <- interaction(reg.invert$Habitat, reg.invert$Detritus)

Thus, I have the following model:
   model <- glmer (Species Richness ~ interaction + (1|Site)).

I have been trying to do this using glht() function in the package multcomp.
This is what I ran, with notes about the output:
 inter.test1 <- glht(model,  mcp(interaction = "Tukey"))
 summary(inter.test1)
 cld(inter.test1)

This resulted in 5 warnings that said

In RET$pfunction("adjusted", ...) : Completion with error > abseps.

Why am I getting these warnings? Should I not use the results of cld(), given these warnings? How can I avoid the warnings? In looking at the raw data, some of the letters seem a bit off, so I was wondering if I can trust this output.
I have also run the exact same model with a different response variable without any errors, and the extracted letters make sense.
I would be very grateful for any advice for how to successfully use Tukey's method in this model, as well as for how to extract letters to denote significant differences. Either suggestions for adjusting my code or for a new approach entirely would be fine.
Thanks for your help!

Edit: See below for what I hope is a reproducible example using my data...please bear with me- I've never made one of these before:
 library(lme4)
 library(multcomp)

  ####re-create my crazy dataframe:
 new.data<-structure(list(Site = structure(c(13L, 13L, 13L, 13L, 13L, 13L,
                              13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 12L,
                              12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L,
                              12L, 12L, 12L, 12L, 11L, 11L, 11L, 11L, 11L, 11L, 11L, 11L, 11L,
                              11L, 11L, 11L, 11L, 11L, 11L, 11L, 11L, 11L, 10L, 10L, 10L, 10L,
                              10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L,
                              10L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L,
                              9L, 9L, 9L, 9L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L,
                              8L, 8L, 8L, 8L, 8L, 8L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L,
                              7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L,
                              6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 5L, 5L, 5L, 5L, 5L, 5L,
                              5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 4L, 4L, 4L, 4L,
                              4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 2L, 2L,
                              2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
                              1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,
                              3L, 3L), .Label = c("JM", "SL", "AV", "SEB", "CANA", "SC", "NP",
                                                  "GTM", "NERR", "TAL", "JEK", "CAT", "SKID"), class = "factor"),
           Habitat = structure(c(3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,
                                 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,
                                 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,
                                 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,
                                 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,
                                 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,
                                 3L, 3L, 3L, 3L, 3L, 3L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
                                 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
                                 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
                                 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
                                 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
                                 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
                                 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
                                 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
                                 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = c("MAN",
                                                                                "MIX", "SM"), class = "factor"), Detritus = structure(c(2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L, 3L, 1L, 2L,
                                                                                3L, 1L, 2L, 3L, 1L), .Label = c("CON", "AVI", "SPAR"), class = "factor"),
           Species.Number = c(10L, 7L, 10L, 5L, 7L, 0L, 6L, 6L, 11L,
                              5L, 4L, 3L, 5L, 5L, 1L, 4L, 7L, 4L, 7L, 8L, 6L, 10L, 11L,
                              6L, 7L, 5L, 7L, 10L, 10L, 5L, 11L, 8L, 6L, 8L, 10L, 9L, 4L,
                              6L, 5L, 7L, 5L, 3L, 7L, 8L, 3L, 7L, 5L, 5L, 4L, 3L, 4L, 7L,
                              6L, 5L, 8L, 3L, 4L, 5L, 7L, 3L, 6L, 6L, 6L, 6L, 7L, 4L, 6L,
                              6L, 5L, 6L, 8L, 8L, 6L, 4L, 7L, 11L, 6L, 6L, 4L, 3L, 1L,
                              7L, 5L, 4L, 9L, 6L, 6L, 3L, 4L, 3L, 8L, 11L, 9L, 7L, 10L,
                              6L, 6L, 11L, 8L, 4L, 6L, 5L, 5L, 6L, 5L, 6L, 5L, 2L, 2L,
                              3L, 1L, 3L, 1L, 0L, 3L, 5L, 3L, 6L, 4L, 2L, 3L, 6L, 3L, 6L,
                              10L, 4L, 5L, 6L, 4L, 8L, 5L, 1L, 6L, 4L, 4L, 4L, 4L, 4L,
                              4L, 3L, 1L, 6L, 3L, 3L, 5L, 4L, 5L, 5L, 5L, 4L, 9L, 9L, 6L,
                              11L, 4L, 5L, 7L, 8L, 4L, 11L, 17L, 1L, 3L, 2L, 3L, 5L, 2L,
                              3L, 5L, 2L, 4L, 2L, 3L, 4L, 3L, 4L, 3L, 9L, 4L, 0L, 2L, 2L,
                              2L, 2L, 6L, 2L, 1L, 3L, 1L, 5L, 3L, 1L, 6L, 4L, 1L, 3L, 4L,
                              4L, 4L, 8L, 2L, 3L, 5L, 1L, 1L, 3L, 2L, 6L, 8L, 4L, 3L, 3L,
                              1L, 1L, 1L, 2L)), .Names = c("Site", "Habitat", "Detritus",
                                                           "Species.Number"), row.names = c(NA, -216L), class = "data.frame")

 ####analysis:

 new.data$interaction<-interaction(new.data$Habitat,new.data$Detritus)
 m6<-glmer(Species.Number~interaction + (1|Site), family = "poisson", data = new.data, control=glmerControl(optimizer="bobyqa", optCtrl = list(maxfun = 20000)))

 inter.test1<-glht(m6, mcp(interaction="Tukey"))
 plot(inter.test1)
 summary(inter.test1)
 cld(inter.test1, level=0.05)

Comment: There seems to be some typos in your code, e.g.: In the model you use `Interaction` while here `reg.invert$interaction` so one is capitalized the other is not. I tried on an example dataset and for me everything works fine. Maybe provide a [reproducible example](https://gist.github.com/hadley/270442).
Comment: Awesome, thanks so much for your help! I just added what I hope is a reproducible example using my dataset to the end of the post. Any suggestions would be great!
Answer: I also cannot seem to figure out what the warning is trying to tell us but one alternative approach would be using the emmeans() function in the emmeans package and then the cld() function:
inter.test1 <- emmeans(m6, "interaction")
#NOTE: I would probably rename "interaction" to something else as this word is often used
#in other function arguments or even as a function itself, i.e. interaction()!
cld(inter.test1, Letter="abcdefg")
#interaction    emmean        SE  df asymp.LCL asymp.UCL .group
#MAN.CON     0.7938065 0.1786641 Inf 0.4436314  1.143982  a
#MAN.AVI     1.2638100 0.1502522 Inf 0.9693211  1.558299  abc
#MIX.CON     1.2995688 0.1335088 Inf 1.0378964  1.561241  ab
#MAN.SPAR    1.3096194 0.1479132 Inf 1.0197148  1.599524  abc
#SM.CON      1.5982275 0.1094568 Inf 1.3836962  1.812759   bcd
#MIX.AVI     1.7414015 0.1177577 Inf 1.5106006  1.972202    cd
#MIX.SPAR    1.8103945 0.1157236 Inf 1.5835804  2.037209    cd
#SM.SPAR     1.8133390 0.1034105 Inf 1.6106582  2.016020   bcd
#SM.AVI      1.8908973 0.1014570 Inf 1.6920453  2.089749     d

#Results are given on the log (not the response) scale.
#Confidence level used: 0.95
#P value adjustment: tukey method for comparing a family of 9 estimates
#significance level used: alpha = 0.05

If you want to see the mean estimates on the response scale, i.e. back-transformed from the log scale, you can add type = response:
inter.test2 <- emmeans(m6, "interaction", type = "response")
cld(inter.test2, Letter="abcdefg")

See also the very helpful emmeans vignettes.
Comment: Hi Stefan- thanks for this suggestion! Any ideas on why the df = Inf in the emmeans output? Also, from reading one of the EMM vignettes, they state that they "really don't recommend this method, though, as it imposes a stark difference between P values slightly less and slightly more than alpha." Any additional suggestions on this? I really want to know what that warning means from the multcomp package...
Comment: Here is an explanation what the df=Inf means in the output: https://cran.r-project.org/web/packages/emmeans/vignettes/models.html#G Regarding the other part of your question GLMMs by default provide Wald Z-statistics, which are asymptotic approximations.    Also have a look here and search for degrees of freedom http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html I am sure `multcomp` uses the same asymptotic approximations for its p-values. See also answer+comments here: https://stats.stackexchange.com/questions/204741/which-multiple-comparison-method-to-use-for-a-lmer-model-lsmeans-or-glht


Retuning Inception Models with retrain.py

Question: I've converted the inception_v3 and inception_resnet_v2 .ckpt files found at
https://research.googleblog.com/2016/08/improving-inception-and-image.html
to a frozen .pb file and i'm trying to use it with the tensorflow/tensorflow/examples/image_retraining.py. Specifically there's this part:
BOTTLENECK_TENSOR_NAME = 'pool_3/_reshape:0'
BOTTLENECK_TENSOR_SIZE = 2048
MODEL_INPUT_WIDTH = 299
MODEL_INPUT_HEIGHT = 299
MODEL_INPUT_DEPTH = 3
JPEG_DATA_TENSOR_NAME = 'DecodeJpeg/contents:0'
RESIZED_INPUT_TENSOR_NAME = 'ResizeBilinear:0'
MAX_NUM_IMAGES_PER_CLASS = 2 ** 27 - 1  # ~134M

I can't figure out how to adapt this for the new inception v3 names or inception resnet v2. I've looked at the graph in tensorboard but it didn't help. What would the pool_3 layer become? Whats the size? And what about ResizeBilinear?
For inception_resnet_v2 it looks like it should be Logits/Flatten/Reshape or just Logits with size 1536 but it says those aren't in the graph.
This retraining is orders of magnitude faster than the TF slim alternative.
Answer: I used DATA_URL = 'http://download.tensorflow.org/models/inception_v4_2016_09_09.tar.gz'.
And then I did not have to change anything else. It is working properly.
Let me know if you face any problems.
It works with http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz as well.
Answer: these values should work:
bottleneck_tensor_name = 'InceptionResnetV2/Logits/Flatten/Reshape:0'
bottleneck_tensor_size = 1536
resized_input_tensor_name = 'InputImage:0'


Calculating 2 datagridview columns must be the same number of rows

Question: I have 2 data grid views, one for Purchases and one for Sales
I sum the total of sales and subtract the total of purchases from it.
the problem is, it will only do the calculation if the number of rows of both datagridviews are the same, if one of them has more rows than the other, it will not give any value in the textbox where it should show the result
that's my code
<code>private void GrossSale()
{
    int sum = 0;
    for (int i = 0; i < dataGridView1.Rows.Count; ++i)
    {
        sum += Convert.ToInt32(dataGridView1.Rows[i].Cells[2].Value);
    }
    txtGrossSale.Text = sum.ToString();
}
private void NetSale()
{
    int sum2 = 0;
    for (int i = 0; i < dataGridView1.Rows.Count; ++i)
    {
        sum2 += Convert.ToInt32(dataGridView1.Rows[i].Cells[3].Value);
    }
    txtNetSale.Text = sum2.ToString();
}
private void GrossPurchase()
{
    int sum3 = 0;
    for (int i = 0; i < dataGridView2.Rows.Count; ++i)
    {
        sum3 += Convert.ToInt32(dataGridView2.Rows[i].Cells[2].Value);
    }
    txtGrossPurchase.Text = sum3.ToString();
}
private void NetPurchase()
{
    int sum4 = 0;
    for (int i = 0; i < dataGridView2.Rows.Count; ++i)
    {
        sum4 += Convert.ToInt32(dataGridView2.Rows[i].Cells[3].Value);
    }
    txtNetPurchase.Text = sum4.ToString();
}
private void TotalGross()
{
    int sum5 = 0;
    int sum6 = 0;
    int sum7 = 0;
    for (int i = 0; i < dataGridView2.Rows.Count; ++i)
    {
        sum5 += Convert.ToInt32(dataGridView1.Rows[i].Cells[3].Value);
        sum6 += Convert.ToInt32(dataGridView2.Rows[i].Cells[3].Value);
    }
    sum7 = sum5 + sum6;
    txtGross.Text = sum7.ToString();
}
</code>
Comment: Or heck, maybe just bind a datatable and forget the looping calculations. https://stackoverflow.com/a/3779835/7948962
Comment: Try with [Compute](https://msdn.microsoft.com/en-us/library/system.data.datatable.compute%28v=vs.110%29.aspx) instead perhaps.
Comment: Thanks @JacobH for your reply, should I still use For with the compute ?
Comment: Do you have a datatable? That is key. If you do, then you just create 2 ints, compute the value and then subtract the ints. No loops needed. If you have no datatable then look at @Mark answer below.
Answer: Have a look at TotalGross().  You are looping over dataGridView2.Rows.Count, but are accessing both dataGridView1.Rows[i] AND dataGridView2.Rows[i] inside the loop...
Comment: Exactly, you loop separately over each one.
Comment: so I should use For loop for each datagridview then calculate
Comment: That fixed the problem, Thanks


If $\log_ax=3$ and $\log_bx=4$, then what is $\log_{ab}x$?

Question:
If $\log_ax=3$ and $\log_bx=4$, then what is $\log_{ab}x$?

I'm sure there is some logarithmic rule that can allow me to solve this in one or two steps, but as I am not very familiar with logarithms, I have decided to use the definition of logarithms to convert them to exponential equations.

So from $\log_ax=3$, we have $$x=a^3$$Similarly, from $\log_bx=4$, we have $$x=b^4$$Then I multiply those two equations together:$$\begin{align}x^2&=a^3b^4\\x^2&=(ab)^3\cdot b\\2\log_{ab}x&=3+\log_{ab}b\\\log_{ab}x&=\frac{3+\log_{ab}b}2\end{align}$$The only problem now is the extra $\log_{ab}b$ term. If I can get the value of that, then I will solve the problem. But right now I can't seem to find a way to complete this problem. Can anyone provide some insight on this problem? Thanks.
Comment: Don't try to figure out what $x$ is in terms of $ab$.  Try to figure out what $ab$ is in terms of $x$.  $x = a^3$ so $a = x^{\frac 13}$ and $x^4 = b$ so $b= x^{\frac 14}$ so $ab = x^{\frac 13 + \frac 14}$.  .... and from *there* you get $(ab)^{\frac 1{\frac 13 + \frac 14}} = x$.
Comment: BTW, the logarithmic rule that would make this easiest is $\log_a b = 1/(\log_b a)$. Not what you asked for, but it's good to know.
Comment: According to 3Blue1Brown's video called [Triangle of Power](https://www.youtube.com/watch?v=sULa9Lc4pck), the $3$ and $4$ get O-plussed, which gives $\frac{1}{1/3+1/4} = \frac{12}{7}$.
Answer: $x = a^3 \Rightarrow a = x^{1/3}$  and $x = b^4  \Rightarrow b = x^{1/4}$
So, $ab = x^{\frac13 +\frac14} = x^{\frac7{12}} \Rightarrow x = (ab)^{12/7} \Rightarrow \log_{ab}(x) = \frac{12}7$
Answer: Just remember
$$ \log_vu= \dfrac{\log_p u}{\log_p v}$$
where $p$ can be any arbitrary real number. Writing as power to base
$$ x=a^3 = b^4$$
The given quantity
$$ \dfrac{\log x}{\log a+ \log b}  = \dfrac{\log x}{\log x^\frac13+ \log x^\frac14}  $$
$$ = \dfrac{\log x}{\frac13 \log x+ \frac14\log x} = \dfrac{1}{\dfrac13+\dfrac14} = \dfrac{12}{7}$$
Answer: Just go back to definitions.
$\log_a x = 3\iff a^3 = x$.
And $\log_b x = 4 \iff b^4=x$.
So $a = x^{\frac 13}$ and $b=x^{\frac 14}$.
So $ab = x^{\frac 13}x^{\frac 14}= x^{\frac 13 + \frac 14}$.
So $(ab)^{\frac 1{\frac 13 + \frac 14} } = x$.
Which means $\log_{ab} x = \frac 1{\frac 13 + \frac 14}$.
Answer: Actually from $x=a^3$ and $x=b^4$ we immediately have $a=b^{4/3}$. Thus $\log_{ab} b = \log_{b^{7/3}}b= \dfrac{3}{7}$. Thus we get $\log_{ab}x=\dfrac{12}{7}$.
Answer: $$a^3=x=b^4\implies b=a^{3/4}$$
Then
$$\log_{ab}b=\log_{a^{7/4}}a^{3/4}=\frac{3/4}{7/4}=\frac37$$
and hence $\log_{ab}x=\frac{12}7$.
Answer: As $\log_ab=\dfrac{\log b}{\log a}$ when all the logarithms remain defined,
$$3=\log_ax=\dfrac{\log x}{\log a}\implies\log a=?$$
Similarly, $\log b=\dfrac{\log x}4$
$$\log_{ab}x=\dfrac{\log x}{\log a+\log b}$$
Replace the values of $\log a,\log b$ in terms of $\log x$
Answer: $$\log_{ab} x = \frac{1}{\log_x ab} = \frac{1}{\log_x a + \log_x b} = \frac{1}{\frac{1}{\log_a x} + \frac{1}{\log_b x}} = \frac{1}{\frac{1}{3} + \frac{1}{4}} = \frac{12}{7}$$


Issue while running MongoDB on Mac - DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead

Question: I m trying to run locally installed MongoDB(community edition7.0)on mac, while I m starting mongo using "mongosh" or "mongo" below is the error I m getting
(node:28156) [DEP0040] DeprecationWarning: The punycodemodule is deprecated. Please use a userland alternative instead. (Usenode --trace-deprecation ... to show where the warning was created) MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017
node --version
v21.2.0

Build Info: {
"version": "7.0.2",
"gitVersion": "02b3c655e1302209ef046da6ba3ef6749dd0b62a",
"modules": [],
"allocator": "system",
"environment": {
    "distarch": "aarch64",
    "target_arch": "aarch64"
}

Comment: Oh okay. I am connecting with the Mongo support team. They're trying to locate the reason. If resolved I'll update here.
Comment: Is this resolved? I face the same issue.
Comment: @Manoj Nope this is not resolved yet for me.
Comment: @Manoj  any update on this issue, I am also getting the same issue,
Comment: No still same problem. I contacted Mongo support using their chat support and got and a custom alternative URL. It's working.
Answer: First start mongodb as a background process like;
for intel processors:
mongod --config /usr/local/etc/mongod.conf --fork
for silicon processors:
mongod --config /opt/homebrew/etc/mongod.conf --fork
then try to use mongosh in terminal, you are gonna still get the same error but will able to use mongosh.
Comment: Tried with above commands to run in background in Sillicon processor here is the result. This is what I see,                                                                                "mongod --config /opt/homebrew/etc/mongod.conf --fork
about to fork child process, waiting until server is ready for connections.
forked process: 83233 ERROR: child process failed, exited with 1
To see additional info"                 mongosh is trowing the same error mentioned above and not able to get into the mongodb shell to access the databases since it is failing to connect.
Comment: i also get the same error when i start the server like that. but after getting that error try to write the "mongosh" command. if you see something like "test" in terminal then try to write "show dbs" command. if it is works then go on like that for now. :)
Comment: Sorry , for me it's not happening. I Mongo is never starting for me for the command "mongosh" and throws (node:85166) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead. Any hnt would be much appritiated.


Are there security or hardware management issues that would require a policy of setting displays to never sleep?

Question: At my current employer, there is a policy that is pushed out daily that includes a setting that forces the display on the computer to never go into sleep mode (on Mac that is System Preferences -> Energy Saver -> Turn Display Off After, and I forget the exact name in Windows, but also under energy settings). I have experienced this on both the Windows platform and the Mac platform leading me to conclude that this was an intentional decision. Note that I am talking about having the monitor go into a power-saving, display off mode, not locking the screen, which is (sensibly) set to a small number of minutes.
Are there specific security or hardware management issues in Windows or Mac OS X that would require forcing a monitor to always remain powered on (albeit with a screen saver running)? My own experience suggests that allowing the screen to power down does not interfere with properly locking the system either manually or via the screensaver timeout setting.
Comment: How could we possibly know why your company implemented such a policy?!?
Comment: We are also not a substitute for your IT department, and this question can only really be answered by them.
Comment: Don't be surprised if, at the end of your research, you discover the source of this policy to be "Rule 57: Because we've always done it that way".  Large companies are certainly not immune to this.
Comment: I don't want to know why _my_ company specifically did it. I'm looking for reasons why such a policy might be desired. Something like a password complexity policy, I know general reasons for even if I don't know the specific reasons my employer used to arrive at their policy. I haven't been able to think of or find any reasonable explanations. In very large corporations (30K+ employees) it can be impossible to get an explanation. I'm trying to research possible 'why's to make sure there isn't a legitimate security reason before I push back on it.
Answer: Consider the following:

If the PC never goes to sleep, then it is always accessible for things like network scans/automated updates/remote management. There are better ways to handle this, but keeping the PC on is easy.

There are kiosk or shared machines that need to stay on all the time, and it was easier to apply the policy to all PCs.

Convenience: Some VIP got really tired of waking up his/her computer and demanded all computers stay on all the time.

Make some extra cash mining crypto off-hours

There's no way to know why your company did it except to ask around. I've never seen it myself, but I found people asking how, so there's a need out there.
Comment: Here's a cynical, but quite plausible explanation:  15 years or so ago, Corp IT Nitwit #1 hears about security problems with certain x-window screensavers and figures "if we never let screens shut off, we don't have to worry about that!".  Corp IT Nitwit #2 overhears only "screensavers . . . not . . . secure" around the water cooler and proceeds to implement various methods of disabling ALL screensavers on all platforms. Poorly thought out policy becomes de facto company standard.
Comment: I have been asking around, but nobody I can reach can answer. I've also seen requests online asking how, but they've all been several years old, and it seemed more appropriate to ask here than to ask a random stranger on the internet why they wanted to do that. Your kiosk suggestion seems the most reasonable, and the crypto mining seems very unlikely with our security team who are very on-the-ball


How to Evaluate code of another ruby file?

Question: Here  have two ruby files, how to open file and evaluate code1.rb in code2.rb

code1.rb

a = 10
b = 15
c = a+b
puts "c = #{c}"

code2.rb

a= 10
e = c*5+a
puts "e = #{e}"

Expected output :
c = 25
e = 135

Comment: First,make up your mind which meachanism you want to use to pass data into code2.rb, because this will dictate, how to glue the two programs together. You could pass the data on the argument list (`ARGV` in Ruby), you can pass it via the environment (`ENV`), you can pass it over a pipe into `STDIN`, or you can pass the value via a file.
Comment: Can you put an examples for how to pass it over a pipe into STDIN, or you can pass the value via a file.
Comment: There are two different approaches, and I suggest that you post on [so] two questions about it. In particular, the file version could lead to a discussion with various proposals. Piping between programms X and Y is simply done in the shell (bash, zsh, ...)  by writing  `X|Y`. `X` writes the data to stdout and `Y` reads the data from stdin.
Answer: You can use Kernel#eval in order to achieve what you want.
but in code1.rb you will need to remove puts, or place it before, as puts returns nil.
So use this changes
code1.rb
a = 10
b = 15
c = a+b
puts "c = #{c}"
c

code2.rb
code1 = File.open('code1.rb', 'r').read
c = eval(code1)

a = 10
e = c*5+a
puts "e = #{e}"

c = eval(code1) This magic line will use return value of your code1.rb file and then we can use it in code2.rb
Or another alternative you can allow your local variables to be overwritten by eval method, but in that case you need to define them in your code2.rb. Consider this
code2.rb
a,b,c = [0, 0, 0]

code1 = File.open('code1.rb', 'r').read
eval(code1)

a = 10
e = c*5+a
puts "e = #{e}"

That works due to local variables scope and evaluation. Both variants are working solutions


Copiar e Colar dados por data - Excel VBA

Question: Preciso mover alguns dados de uma planilha para a outra de acordo com a data, mas a selação que eu fiz usando o comando IF só seleciona a última célula que atinge este critério ignorando as outras.
O que eu tenho até aqui:
Sub Copiar()

    Dim range1 As Range

    Set range1 = Range("k56:k58")

    For Each cell In range1
        If cell.Value = Range("R55").Value Then
            cell.Offset(0, 2).Select
            Selection.Copy
            Sheets("Plan2").Activate
            Range("r56").Select
            ActiveSheet.Paste
        End If
    Next
End Sub

Answer: Isto ocorre pois todas as células que atingem os critérios são coladas no mesmo local, então a última ocorrência é a vista. Percorra o código passo-a-passo com  F8 e perceberá que isso ocorre.
Código
Para que não ocorra, o valor da linha que deve ser colada deve ser incrementado.
Sub Copiar()

    Dim range1 As Range
    Dim i As Long

    Set range1 = Range("k56:k58")

    For Each cell In range1
        If cell.Value = Range("R55").Value Then
            cell.Offset(0, 2).Select
            Selection.Copy
            Sheets("Plan2").Activate
            Range("r" & 56 + i).Select
            Selection.Paste
            i = i + 1
        End If
    Next
End Sub

Obs.: Tente evitar o uso de .Select/.Activate/Selection/Activecell/Activesheet/Activeworkbook.

O código ficaria:
Sub Copiar()

    Dim range1 As Range
    Dim i As Long
    Dim ws As Worksheet

    Set ws = ThisWorkbook.Sheets("Plan2")
    Set range1 = ThisWorkbook.Sheets("Plan1").Range("k56:k58")

    For Each cell In range1
        If cell.Value = ws.Range("R55").Value Then
            cell.Offset(0, 2).Copy ws.Range("R" & 56 + i)
            i = i + 1
        End If
    Next
End Sub

Comment: Oi, obrigado pela resposta, tentei os dois códigos, mas nenhum funcionou =/
Comment: Verificou se o nome da planilha está correto em `Sheets()`?
Comment: Consegui agora, funcionou perfeitamente, o critério estava errado, Mudei:

 If cell.Value = ws.Range("R55").Value Then

Para:
If cell.Value = Sheets("Plan1").Range("R55").Value Then


Count with only certain digits allowed

Question: I have a friend ~200 years old mathematician who has forgotten some digits and now he counts things in very strange manner: when he is going to count the $n$-th thing and $n$ contains a digit he cannot pronounce and write (because of his senile memory - oh, poor man) he skip $n$ and tries to think of next smallest number after $n$ that does not contain the digit.

Poor old mathematician who forgot digits 0, 2, 5, 6, 7 counts:
  1, 3, 4, 8, 9, 11, 13, 14, 18, 19, 31, 33, 34, 38, 39, 41, 43, 44, 48, 49, 81, 83, 84, 88, 89, 91, 93, 94, 98, 99, 111, 113, ...

He claims that second thing is third, third is fourth, fourth is eighth and do not see any problems at all. However I do have a problem now, that is keep bothering me:

How do I easily name the $n$-th thing in his counting system?

Generally,

Natural numbers that contain certain digits are dropped from $\mathbb{N}$. What is a formula for the $n$th number of this sequence?

Let's look at a plot of the numbers my friend still remember:
$\hskip0.75in$
that looks like a random hops at the first glance, but take the difference between $\left(n+1\right)$th and $n$th element and you'll get this:
$\hskip0.75in$
hence the second part of the question's title. I believe that this is a some fractal structure and therefore there exist a recurrence relation that is a solution to my problem.
Firstly, I've tried to examine a very simple example: Base 3; digits allowed: 0, 2:

0, 2, 20, 22, 200, 202, 220, 222, 2000, 2002, 2020, 2022, 2200, 2202, 2220, 2222, ...
  adjacent differences:
  2, 11, 2, 101, 2, 11, 2, 1001, 2, 11, 2, 101, 2, 11, 2, ...

There are some "oscillations" seen...
$\hskip0.75in$
Notation:
Let $n$ be the number we want to get, $b$ - the base of numerical system, $D$ - set of digits allowed, then $N(D, b, n)$ - $n$th number in the sequence of numbers of base $b$ with only allowed digits $D$.
So far, I have only seen that $N(\{0,2\},3,n^3+1)=N(\{0,2\},3,n^3)+2$.
Help me, please (at least by small hints or advices) to deduce the relations that will help me to beat the problem and consequently help to reach a compromise between my friend and me. :)

Update
It may seem strange that I ask about generating sequences with some of them already found and even plotted. Indeed, I've already found an easy programming solution that uses combinations of digits to form those numbers and it's highly inefficient and not a matter of talk at this site at all. I'm asking about purely mathematical solution.

Update
I've just seen the sequence of adjacent differences from my example Base 3; digits allowed: 0, 2 that looks like this (in decimal):
$$\left\{\begin{cases}
  1+3^0,\quad n\equiv 2^0-1\pmod {2^1}\\
  1+3^1,\quad n\equiv 2^1-1\pmod {2^2}\\
  1+3^2,\quad n\equiv 2^2-1\pmod {2^3}\\
  \cdots\\
  1+3^p,\quad n\equiv 2^p-1\pmod{2^{p+1}}\\
  \cdots
\end{cases} \right\}_{n\ge0}$$
In other words, if $a(n)$ is a $n$th element of the sequence, then $$a(2^{p+1} n + 2^p - 1) = 1 + 3^p, \quad p \ge 0$$
Update
Using this ideas, I conclude that
$$a(n)=\sum_{i=1}^\infty\frac{1}{2^i}\sum_{j=1}^{2^i}cos(j\,n\,π/2^{i-1})$$
Thus, $$N(\{0,2\},3,n)=N(\{0,2\},3,n-1)+1+3^\wedge(\sum_{i=1}^\infty\frac{1}{2^i}\sum_{j=1}^{2^i}cos(j\,n\,π/2^{i-1}))$$
But this particular solution does not enlighten me on a general solution.
Answer: The original answer worked if the digits he could say included zero. I have kept it. Let him be able to say $0,1,3,4,8$, the algorithm would be to express $n$ in base $5$, then change the digits $0 \to 0, 1 \to 1, 2 \to 3, 3 \to 4, 4 \to 8$  and you get the $n^{\text{th}}$ number he can say.   So for $n=1234567,$ we have $1234567_{10}=304001232_5 \to 408001242$ as the $1234567^{\text{th}}$ number he can say.
For the original question where he can say $1,3,4,8,9$, because $0$ is among the forgotten numbers, there are $5$ one digit numbers he can say (not $4$), $25$ two digit numbers he can say, etc.  For numbers with up to $k$ digits, there are $5+5^2+\dots +5^k=\dfrac {5^{k+1}-5}{5-1}=N(k)$  To find the $n^{\text{th}}$ number he can say, find the $k$ such that $N(k) \lt n \le N(k+1).$   The number will have $k+1$ digits.  If $n=N(k+1),$  the $n^{\text{th}}$ number he can say is a series of $k\ 9$'s.  Otherwise, compute $d_{k+1}=\left \lfloor \frac {n-N(k)}{5^k} \right \rfloor$, which will be in the range $[0,4]$  With the conversion $0 \to 1, 1 \to 3, 2 \to 4, 3 \to 8, 4 \to 9$ you have the leading digit.  Now compute $n_k=n-(1+d_{k+1})5^k$ and express $n_k$.  It will be large enough to require $k$ digits.  For example, if we want the $397^{\text{th}}$ number he can say, we find $k=3, N(k)=155$ numbers he can say with up to three digits.  $d_4=\left \lfloor \frac {397-155}{125} \right \rfloor=1$, so the leading digit is $3$.  Then $n_3=397-2\cdot 125=147$ and we continue $d_3=\left \lfloor \frac {147-30}{25} \right \rfloor=4$, so the next digit is $9$.  $n_2=147-5\cdot 25=22, d_2=\left \lfloor \frac {22-5}{5} \right \rfloor=3$ so the next digit is $8$ and finally we have $2$ left and the last digit is $4$.  The $397^{\text{th}}$ number he can say is $3984$.  I hope he remembers $0$.
Comment: The point of my algorithm is that once you decide it has four digits, the one, two, and three digit numbers take up $5+25+125=155$, then each cycle through a given leading fourth digits takes $125$.  Since $155+125 \lt 397 \lt 155+125+125,$ we are in the second cycle of four digit numbers.  The first has leading digit $1$, the second has leading digit $3$.  True, it is coincidence that the leading digit in decimal and in this representation are the same.  If we wanted the $402nd$ number it would be $3994$.  I think this algorithm works.
Comment: When I've read your answer I was a bit astonished - how couldn't I think of this simple and elegant solution? If we reassign _visual representation_ of digits like this $\mathbf{1}=0,\mathbf{3}=1,\mathbf{4}=2,\mathbf{8}=3,\mathbf{9}=4$ then $1,3,4,8,9,11,13,14,18,19,31,\dots$ is legal numerical system of base $5$, but it can take a while to get used to it. To convert a number in such system to normal one we need to replace new digits by their actual _values_ and express resulting number in base $10$. Inverse process is what you've just written. Thank you a lot!
Comment: I like your solution of finding $n$th number. It's also applicable for all such counting systems I've described. One note is that if $0$ is allowed we needn't to decrement $n$.
Comment: We've missed something... I've just found that if we plug in $7$ in the algorithm, we get result of $31$, not $11$...
Comment: @DanyloDubinin:  you are right.  It is related to the $-1$ I had to put in because $0$ is not allowed.  There are then $25$, not $20$ two digit numbers he can say. I'll update.
Comment: It's not so obvious how to find such $k$ for arbitrary $n$, though. It's just a coincidence that in your case $k=\lceil\log_{10}(n)\rceil$.
Answer: At least if you just want asymptotics, it's easier to answer the inverse question first: how many of the integers between $0$ and $N$ (inclusive) can your friend name?
Let's say we work in base $b$ and that your friend can only use $k$ of the possible $b$ digits, and for simplicity let's assume that he can use $0$. The easiest case is when $N = b^n - 1$, since in this case it's clear that your friend can name exactly $k^n$ numbers: of the $n$ possible digits, your friend can freely choose each of them to be one of the $k$ allowable digits.
That is, if $f_{b, k}(N)$ is the number of integers between $0$ and $N$ that your friend can name, then
$$f_{b, k}(b^n - 1) = k^n.$$
This suggests that asymptotically we expect
$$f_{b, k}(N) \approx N^{ \log_b k }$$
and hence the inverse function should asymptotically satisfy
$$f_{b, k}^{-1}(M) \approx M^{ \log_k b }.$$
Disallowing $0$ isn't much harder and doesn't change the asymptotics much but getting more precise counts is fiddly and depends on both the digits of $M$ and on the digits that are allowed / disallowed. In your original example,
$$f_{10, 5}^{-1}(M) \approx M^{ \log_5 10} \approx M^{1.43}$$
although this won't be all that accurate for random values of $M$ because a lot depends on whether a particular first digit is allowed.
Comment: Your explanations are great. Thank you, I appreciate your analysis. I was thinking about the inverse question but couldn't figure out how it would help. Now it's clear for me.

Count with only certain digits allowed


404s on all pages except homepage

Question: I have copied a Drupal installation onto my local machine and I am now running into an issue where only the homepage will load and all other links return a 404.
Clean URLS are enabled but even attempting to manually type the URL like:
<code>?q=admin/module
</code>
fails...
I have tried the following:

Setting AllowOverride to All
including the mod_rewrite module
including the usr_module
setting the baseURL in settings.php

None have yet appeared to have any effect.
Can anybody suggest anything else to try?
Comment: check for .htaccess issues or run YOURHOST/update.php from a browser to rebuild menus etc perhaps?
Comment: I will... thanks.

In the meantime, I turned off CleanURLs in the settings.php file and I can now get access but it's not ideal.
Comment: If you are using IIS check your web.config file and follow this guide. [Drupal clearn URL guide](https://www.drupal.org/docs/7/configuring-clean-urls/clean-urls-with-iis)
Answer: Check that you have the .htaccess file present in your Drupal root folder.
Answer: This happened to me once, I copied an entire site folder and clean urls didn't work. I'd copied my files by just selecting all the files in a folder, which doesn't select hidden files, so I was missing my .htaccess file in the drupal root folder.
Answer: This problem could also be caused by editing your Apache config file and changing the default value of DocumentRoot "/var/www/html". Try keeping it at the default even if your site's files are not located in /var/www/html.

404s on all pages except homepage


Adicionar linha RichText Delphi

Question: Boa tarde
Tenho um aplicativo que o texto tem que ter Negrito, italico e sublinhado por isso estou usando um TRichText. Até aqui tudo bem
Para o mesmo funcionar na impressão eu fiz RTRLrichText que recebe os valores do TRichText.
Porém... se eu inserir linhas em brancos para que o texto aparece do meio ou para o fim da pagina na visualização e na impressão não sai como na tela. Ou seja não consigo add linhas.
Alguém sabe como ?
Comment: Tem como mostra o seu Código?
Answer: Experimente assim:
<code>Richedit1.SelAttributes.Color:=clBLue; // Cor Azul
Richedit1.SelAttributes.Style:=[fsBold,fsUnderline]; // Negrito e sublinhado
Richedit1.Lines.Add('Linha'); // Adiciona linha
Richedit1.SelAttributes.Style:=[]; // Volta ao estilo normal
Richedit1.SelAttributes.Color:=clBlack; // Volta à cor normal
</code>
Achei isso em um site
Comment: Fiz os testes aqui só não imprimir porque to sem impressora. Visualmente ele adiciona as linhas...

Adicionar linha RichText Delphi


Como conectar PHP com o SQLServer através do PDO

Question: Estou usando o seguinte código:
try {

    $con = new PDO("sqlsrv:Server=172.20.21.193,9090;Database=sgt","deb","123123");

    $con->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

} catch (PDOException $e) {
    die ("Erro na conexao com o banco de dados: ".$e->getMessage());
}

mas está dando erro ao tentar conectar. O driver do PDO está instalado. Qual outra biblioteca ou driver devo instalar, para que minha aplicação acesse o banco de dados? A versão do PHP é a 5.6.9 (Linux Debian 8.1) e o SQLServer é 2008(o acesso será remoto).
Comment: Qual erro aparece?
Comment: Este "could not find driver"
Comment: Vc precisa instalar ele primeiro, veja o procedimento nesse link http://pt.stackoverflow.com/q/48078/91
Comment: Mesmo para Linux? Estes seriam os passos?
Comment: No Linux, não sei dizer, edite a pergunta e adicione essa informação com mais detalhes ajuda os outros usuários a responder melhor.
Comment: Tente o PDO_DBLIB: http://php.net/manual/pt_BR/ref.pdo-dblib.php ou o PDO_ODBC com driver para linux, conforme descrito aqui: http://php.net/manual/pt_BR/ref.pdo-sqlsrv.php
Answer: Consegui seguindo as orientações do link: https://alephnull.uk/content/accessing-microsoft-sql-server-php-ubuntu-using-pdo-odbc-and-freetds. A linha do código ficou assim: $con = new PDO("dblib:host=172.20.21.193,9090;dbname=sgt", "deb", "123123");
Comment: Estou tendo o mesmo problema, porém a solução acima não me ajudou @Deb
Answer: o Drive PDO sqlsrv não funciona no Linux com PHP 5.6, nesses casos use dblib.
Para PHP 7.0 ou superior pode usar o sqlsrv
Link Docker File Ubuntu 16.08 com PHP 7.0 e PDO sqlsrv
https://github.com/bjverde/formDocker/blob/master/ubuntu_php7.0_mssql/ubuntu_apache_php70.Dockerfile


Interaction with endogenous variables in the first stage

Question: I am working with industry level data and trying to solve an issue with omitted variables bias, by using an instrument. The problem with my instrument is that it only varies relatively little. I.e. for most my industries its value is the same. Or in other words it only varies by groups encompassing several industries.
One of my teachers told me that I can fix that problem by putting additional explanatory variables in the first stage and interacting them with the instrument. And these variables do not have to be exogenous (with regards to my outcome variable).
The following is what I have thought about this, if someone has a better idea how solve the problem I describe above, I would very much appreciate that too!
So basically, first I was thinking of this:
Equation of interest: $$  y_i = \alpha_0 + \alpha_1 x_i + \alpha_2 \mathbf{X}+ \epsilon_i $$
Where $y_i$ is the outcome, $\alpha_1$ the coefficient of interest, $x_i$ is the endogenous variable, $\mathbf{X}$ are covariates.
First stage: $$x_i = \beta_0 + \beta_1 z_i + \beta_2 w_i + \beta_3 z_i * w_i + \beta_4 \mathbf{X} + \eta_i $$
To get $\hat x_i$, $z_i$ is an instrument that only has an effect on $y_i$ through $x_i$, $w_i$ is a covariate correlated with $\epsilon_i $
Second stage: $$y_i = \gamma_1 + \alpha_1 \hat x_i + \gamma_2 w_i + \gamma_3 \mathbf{X} + e_i$$
Then I thought I must have understood something wrong, because this seems kind of weird. But now I read a paper - Nizalova, Murtazashvili (2014) - (https://www.degruyter.com/document/doi/10.1515/jem-2013-0012/html) that explains why interaction effects of one exogenous and one endogenous variable are still consistent.
And another paper - Bun, Harrison (2019) - (https://www.tandfonline.com/doi/full/10.1080/07474938.2018.1427486) that argues similarly: specifically they write:

... we have that, even if we have an endogenous regressor x, the OLS estimator of the ceofficient $\beta_{xw}$ is consistent and standard heteroskeasticity-robust OLS inference applies.

and

... we show that endogeneity bias can be reduced to zero for the OLS estimator as far as the interaction term is concerned.

Does this mean what I outlined above works? I.e. the interaction term is not endogenous (wrt to $y_i$) eventhough $W_i$ is?
Answer: Had a quick read of section 2 of Bun and Harrison (2019). I am not sure what they show is what you are claiming (i.e., want to do).
First and foremost, they do not assume that $w_i$ is endogenous (see discussion before equation 1 and Assumption 1 where their error term $u_i$ refers to your $\epsilon_i$). Saying that $w_i$ is correlated with $\epsilon_i$ violates the exclusion restriction.
Second, their parameter of interest is the coefficient on the interaction between $x_i$ and $w_i$.
They discuss different IV strategies that rely on different assumptions: (i) identification through structural form, (ii) identification through heteroskedasticity.
They abstract away from both by using assumptions 2b (linear reduced form) and 2c (homoskedastic reduced form error).
Then they show in Lemma 1 and Propostion that using their model in equation 1 (which is the main regression and includes $w_i$) and assumptions 2b-2c, the OLS estimator $\beta_{xw}$ is consistent and standard robust SE. That is, you don't need instruments to estimate $\beta_{xw}$ as long as your main regression is like their equation (1) and you satisfy assumptions 2b and 2c (see above).
Most importantly, they do not import another endogenous variable to use in a first stage.
I hope this helps.
Edit 1: If I were you, I would contact your advisor again to clarrify.


Prove that $\forall\, x>0$ exists $n\in\mathbb{N}$ s.t. $ \frac{1}{2^n}<x$

Question: My teacher gave me this question and I don't really know where to start, she said we might need to use the Archimedes' principle but I don't know how to do it.
I would be great if you could help me.
Comment: But $\frac{1}{2^n} \leq \frac{1}{2}$ for all $n \in \mathbb{N}$... Maybe you want $\frac{1}{2^n}<x$?
Comment: [Welcome to Math.SE!](//math.meta.stackexchange.com/a/11168/290189) Please read [this post](//math.meta.stackexchange.com/a/9960/290189) and the others there for information on writing a good question for this site. In particular, people will be more willing to help if you [edit] your question to include some motivation, and an explanation of your own attempts.
Answer: The question seems incorrectly stated, for example if you take $x=1$ there is no $n$ such that $1/2^n > x$, you probably meant $<$.
For that problem, for every $x>0$ show that there exist an $n \in \mathbb{N}$ such that $\frac{1}{2^n} < x$, we need to find an $n$ such that $\frac{1}{2^n}$ is as small as we like. Turning it around, given $x >0$ we want to find $n$ such that $2^n > \frac{1}{x}$.
A little useful trick here may be to notice that $2^n > n$ for all $n \in \mathbb{N}$. So in other words if we can find an $n$ such that $n > \frac{1}{x}$, then $2^n > n > \frac{1}{x}$. Can you see how to use the Archimedean property to find such an $n$?
Comment: Yes I understood, thank you very much


How to use views hooks - specifically hook_view_data()?

Question: I am looking for details on the api to define my own view data, such as what values I can provide for the table I am defining, what values are available for the fields etc.
I have found a tutorial that gives me the basic info, but now I need more specific info.
I have found this page
https://api.drupal.org/api/views/views.api.php/7
but when I click on the link to
hook_views_data

I go to a page with virtually no information with one link which takes me back to the listing page I started at.
Is there somewhere I can find out information about what hook_views_data should define?
Comment: Views's documentation is incomplete, hard to read, sometimes outdated and buggy too. But we are not the ones who can help it. If you have a specific problem with specific thing you need to implement, ask us! Or ask Views team to fix their docs.
Answer: Views does have complete documentation for that hook, but it's not in the most obvious place.
If you install the Advanced Help module and go to http://mysite.com/help/views/api-tables you'll find the Describing tables to Views help page which contains extensive documentation and code samples.
You can also find the full Views' API docs at http://mysite.com/help/views/api with that module installed.
On top of that, the list of functions that implement hook_views_data() would also be worth a look. The aggregator, book and node module implementations look to be well commented at first glance.
Comment: Thank you, I was begining to think it was just me.  Does anyone on the web make Advanced help available to the web in general via google, would be much more useful?
Comment: Advanced help content is just content supplied witha  module (i.e. in Views it's in the /help directory).  If you want to add to the docs, just open an issue and provide a patch to the HTML files.
Comment: @Clive I never expected to say this to you: Please use http://example.com and http://example.org in examples, not a commercial domain, especially not one that does not provide links you have used.


optimal down payment estimation in credit scoring

Question: Knowing I can estimate the risk of default, via logistic regression, of a consumer on a small loan...
what would be the best way to estimate the optimal down-payment amount to ask for in order to reduce that risk ?
Comment: There are so many details missing, it's impossible to answer the question. If you talk about down-payment, it means that the loan is collateralized. What is the collateral? What is the recovery rate on default? Do you assume that downpayment amount does not impact borrower's behaviour? etc.
Comment: How do you propose that the downpayment is related to risk?  How exactly do you define "risk" in this context, anyway? By it do you mean the chance of default or perhaps do you mean the *expected value* of the loan, or maybe even the variance of the value?
Comment: @Serge I understood that as such, but it might be possible that down payment in US is related to some sort of specific schedule for payments.
Comment: My question arises because I was rather supposing that my estimation on the behaviour of the consumer (PD at t=t0) would change if he accepts to perform a downpayment (PD at t=t1). So, I thought a naive solution was to take downpayments as a feature in my model for PD and then consider the coefficient that multiply the downpayment value, as the coefficient, e.g. dx that define the relationship. So, if want a PD<0.1, then I should have a downpayment of X such that dx*X decrease PD to 0.1. Is is a correct way of doing ? Nothing better ? I don't suppose collateral neither at the moment.
Comment: maybe I am missing a point here...if I am working with loans that are not collaterized, is there another word in english than "down payment" for a first payment necessary for the contract to be issued ? I thought prepayment is not correct as it relates to a payment to close the contract before its expected end ?
Comment: This is hard to address. 1. even if your model predicts well, it's correlational. I.e. does not estimate the causal effect of what happens if you change the required downpayment. 2. This would likely change the number of customers & other loan conditions (low downpayment = few customers, unless something else is more attractive e.g. interest; too little = more customers but perhaps riskier). 3. What are you optimizing? Overall company profit? Because, clearly, you run the lowest risk on each loan, if you only take customers with incredibly low risk, but then you have a very low volume.
Comment: Also note that downpayments in the past were set according to some criteria (that probably correlate with other loan terms like interest, property value etc.) that will impact the extent of variation in downpayment (i.e. your model will not have seen as much variation as you'd like).
Answer: Down payment might not directly reduce risk but it might reduce risk of loss when customer is in default.
In international risk management framework for the banks, usually called Basel setting, things are governed by the following eq.
Expected loss= Prob of default (PD) * Loss given default (LGD) * Exposure at default (EAD)
where PD is your risk estimate for individual. It is a probability that one defaults.
It might be that LGD is affected by the down payment.
If PD is affected, then it might be a result via some sort of selection mechanism.
I have seen some results where LGD is a function of PD and collateral (C) and LGD is affected by collateral in following way:
LGD=1-RR(C,PD)
where RR is recovery rate.
NOTE:
-Added an explanation concerning Basel framework
Comment: @AlecosPapadopoulos yes, that was what I was thinking. +1 for the funny quote.. :)
Comment: @Aksakal This equation gives you only expected loss, it is not itself yet useful for pricing of risk. And EL is not enought, you have to have some sort of extra risk buffer. If you have non collateralized consumer loans, then EAD will be function of time having highest value at the start of period. There are other things at work here also. Can we believe LGD and PD will not change as time progress?
Comment: @whuber no. Basel setting is international regulatory framework for risk management in the banking sector. Link: http://www.bis.org/publ/bcbsca.htm
Comment: @Analyst Re: "some sort of selection mechanism": it appears that you wonder whether Oscar Wilde's aphorism "A bank is a place that will lend you money if you can prove that you don't need it" can be quantified and incorporated into a model.
Comment: @Serge then you have only PD and EAD components. Can these depend directly on down payment?
Comment: My question arises because I was rather supposing that my estimation on the behaviour of the consumer (PD at t=t0) would change once he has performed a downpayment (PD at t=t1). So, I thought a simple solution was to take downpayments as a feature in my model for PD and then consider the coefficient that multiply the downpayment value, as the coefficient, e.g. dx that define the relationship. From there, if want a PD<0.1, then I should have a downpayment of X such that dx*X decrease PD to 0.1. Is is a correct ? Nothing better ?
Comment: @Serge, down payment is a factor in PD, on the other hand PD impacts your decision on down payment. So, you have the simultaneous equations here.
Comment: @Analyst, this basel stuff is for term loans, and it's so simplified. Typically loans have periodic payments, and a borrower may default at any period. Basel only wants to get an estimate of the minimum capital requirement. You'd be crazy to use Basel formulae for valuations or risk based pricing.


eSocial BX: "Evento não encontrado ou não processado ou sem perfil para acesso"

Question: Estou utilizando o "Webservice de Solicitação de Download dos Eventos", o Método "SolicitarDownloadEventosPorId".
Porém, ao enviar o ID do evento, estou obtendo o seguinte retorno:
<?xml version="1.0"?>
<eSocial xmlns="http://www.esocial.gov.br/schema/download/solicitacao/retorno/v1_0_0">
    <download>
        <retornoSolicDownloadEvts>
            <arquivos>
                <arquivo>
                    <status>
                        <cdResposta>202</cdResposta>
                        <descResposta>Evento não encontrado ou não processado ou sem perfil para acesso.</descResposta>
                    </status>
                    <evt Id="XXXXXXXXXX40000002XXXXXXXXXX44100006"/>
                </arquivo>
            </arquivos>
        </retornoSolicDownloadEvts>
    </download>
</eSocial>

Pelo que li na documentação, o ID está correto e o evento está sendo encontrado:
201 Evento encontrado.
202 Evento não encontrado ou não processado ou sem perfil para acesso.
401 Número de xxx do evento inválido.

Alguém já passou por isso e sabe o que quer dizer "evento sem perfil para acesso"? Aparentemente o evento foi processado, então só resta essa alternativa.
Obrigado.
Answer: Identifiquei que o problema ocorria pois eu estava utilizando a URL de Produção Restrita (ambiente teste). Ao alterar o apontamento para Produção, obtive o retorno correto com os registros:


Can I listen to multiple domains in one ip on HTTP/3?

Question: I'm using NGINX. I found that NGINX throws error when I set multiple domains for one HTTP/3 port.
    server {
        listen 443 http3 reuseport;
        listen 443 ssl http2;
        server_name FIRST_DOMAIN;
        ssl_certificate     /etc/letsencrypt-ecdsa/live/FIRST_DOMAIN/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt-ecdsa/live/FIRST_DOMAIN/privkey.pem;
        ssl_protocols       TLSv1.3;

        return 301 https://SECOND_DOMAIN$request_uri;
    }

    server {
        listen 443 http3 reuseport;
        listen 443 ssl http2;
        server_name SECOND_DOMAIN default;
        ssl_certificate     /etc/letsencrypt-ecdsa/live/SECOND_DOMAIN/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt-ecdsa/live/SECOND_DOMAIN/privkey.pem;
        ssl_protocols       TLSv1.3;

        add_header Alt-Svc 'h3=":443"; ma=86400';

        location / {
            proxy_pass http://host.docker.internal:10002/;
            proxy_set_header  Host              $http_host;
            proxy_set_header  X-Real-IP         $remote_addr;
            proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Proto $scheme;
        }
    }

nginx: [emerg] duplicate listen options for 0.0.0.0:443 in /etc/nginx/nginx.conf:35

Can I listen to multiple domains in one ip on HTTP/3?
Comment: Did you solve? I've got the same issue
Answer: const express = require('express');

const app = express();

app.get('/', (req, res) => {
    res.send(req.hostname);
});

app.listen(40000);

I experimented about it with Node.js express. When I use HTTP/2, hostname printed. But when I use HTTP/3, undefined printed. This shows that HTTP/3 doesn't care about hostname. So, Listening to multiple domains in one HTTP/3 port is impossible. But you can open several ports for each HTTP/3 domains and add Alt-Svc header for advertise them to browser.
But according to my experience, Alt-Svc doesn't works stable depending on browser. So, you can't listen to multiple domains in one ip on HTTP/3 stably.
Comment: are you sure about this :  you can't listen to multiple domains in one ip on HTTP/3


What are all the themes in the finale of Mozart's symphony 41?

Question: I'm interested in learning more about the finale to Mozart's 41st symphony. I understand that there is a fugue with five themes, and that each of these themes are played at different times in the finale, as well as being played in counterpoint.
I wanted to know if anyone could produce a score for each of the five melodies, so that I could play them on the piano. I already know one of them: the four-note motif, C D F E. But I would like to learn the other motifs. Could anyone post a graphic with the notes to the other themes?
Here is the symphony. The 4th movement starts at 23:53.
Comment: You can find the score in http://imslp.org/. I wouldn't expect anybody here is going to spend their own time going through the movement bar by bar, to do all the work for you! And specifically asking if someone can "transcribe the notes" is off-topic for this site anyway.
Comment: Wikipedia answers the question: https://en.wikipedia.org/wiki/Symphony_No._41_(Mozart)
Comment: Something about this question seems fishy to me...
Comment: With all due respect to the OP, I just don't understand how s/he wasn't able to get this information on his/her own. No matter how I phrase my Internet search, the top result is *always* the IMSLP score to the symphony. And sometimes I get paranoid about the fear of someone asking a homework question; it's odd to me that the OP knows that this is "a fugue with five themes" but then asks for "a graphic with the notes." Something just doesn't add up. But maybe I'm just grumpy today! :-)
Comment: @Richard It's easy enough to learn that "this is a fugue with five themes" from the internet. But coupled with the OP's other question about Mozart symphony 1, I wonder if he/she can read staff notation. It's not a skill that most people are born with!
Comment: Ktm - where a google search gives you the full answer, this site may not add anything. That's why the folks are trying to get to the bottom of your question. I would not assume they are trying to ridicule you.
Comment: One person was speculating that. Another defended the question. I would imagine the downvotes are because Wikipedia already has a full answer for you. Some SE sites actually have a close reason of "not enough research" - what was your "reason to doubt "
Comment: I saw that graphic. I don't think it gives instruments, though? Also, I wasn't sure if that provides all the notes for each theme. I feel like some are longer. Also, I was wondering if someone might transcribe the notes.
Comment: @Richard Why is it so? What is strange?
Comment: To be honest, I feel like you guys are ridiculing me. Is that the atmosphere of this forum, much like ELU? This was not a homework question, and I did see the graphic before asking, but I still had reason to doubt whether it was accurate or complete. It is NOT worth being paranoid about "homework questions" if that means making an honest questioner feel bad about himself. These are some basic values to live by, and I suppose you learn these values from reading books, not music.
Comment: @DrMayhem They are speculating that this is a homework question, for one. That, and four down votes, do not make me feel very welcome here. Nor is the tone of the comments very warm.
Comment: @ktm5124 Just to clarify - this is not a forum or anything like it. It's a Q&A site which functions much more like an archive or database (this is how all SE sites work). As such, questions on these sites require some long term value for more than just the asker (as opposed to forums or Reddit). It is required that some initial research and effort be done by the asker. See [ask]. As for the downvotes, I suspect it's the usual case of an avalanche where you've gotten 1 or 2 and the rest followed. It happens, but there's no vote regulation (save fraud) so we have to live with that.
Answer: How is this not clear enough? If you compare instrument names, they are resp. from upper to lower: flutes, oboes, bassoons (divided), horns in C, natural trumpets in C, timpani in C and G, and strings (violins I & II, violas, cellos divided).
Comment: Ladies and gentlemen, yesterday I had two upvotes, and today it seems two downvotes cancel them. Isn't it an etiquette to explain why my reply does not answer OP's question? It is not that 20 points that I care.
Comment: I love the snark! And yes, I agree that I appreciate when down voters leave a comment, especially if the question has previously received up votes. If your reason to down vote isn't agreed upon by everyone, then it's not clear. Lemme know why!!