Limits and definition integrals involving logarithms

Question: Let $a \in (0,1)$ and define $$I_n(a)=\int_a^1 (\ln x)^n \, \mathrm{d}x$$
Show that limit as $a\to 0$ we have, $$\lim_{a\to 0}I_n(a)=(-1)^n \cdot n!$$
Comment: My guess is that you want to compute $$\int_0^1\log(x)^n\,\mathrm{d}x=\lim_{a\to0}\int_a^1\log(x)^n\,\mathrm{d}x$$
Comment: I've formatted your question using latex, is that what you meant? Or did you want the limits on the integral to be $1$ and $a$ as $$\int_1^a \ln^n x \, \mathrm{d}x$$?
Comment: Or perhaps in the other order, from $a$ to $1$.
Comment: Why should the limit as $n\to\infty$ be a function of $n$?
Comment: yes sorry now its true
Answer: Observe that $I_0(a)=1-a$. For $n>0$ we can use integration by parts to find, $$I_n(a)=\int_a^1\ln^n(x)dx=-a\ln^n(a)-n\int_a^1\ln^{n-1}(x)dx=-a\ln^n(a)-n I_{n-1}(a).$$ By L'Hopital's rule, $$\lim_{a\to 0}-a\ln^n(a)=\lim_{a\to 0}\frac{-\ln^n(a)}{1/a}=\lim_{a\to 0}\frac{n\ln^{n-1}(a)}{1/a}=...=(-1)^nn!\lim_{a\to 0}\frac{1}{1/a}=0.$$
Thus,
$$\lim_{a\to 0} I_n(a)=-n\lim_{a\to 0}I_{n-1}(a)=n(n-1)\lim_{a\to 0}I_{n-2}(a)=...=(-1)^nn!\lim_{a\to 0} I_0(a)=(-1)^nn!.$$
Answer: Using $x=e^t$ and $s=-t$, we get
$$
\begin{align}
\int_a^1\log(x)^n\,\mathrm{d}x
&=\int_{\log(a)}^0t^ne^t\,\mathrm{d}t\\
&=(-1)^n\int_0^{-\log(a)}u^ne^{-u}\,\mathrm{d}u
\end{align}
$$
Taking the limit as $a\to0$ gives
$$
\begin{align}
(-1)^n\int_0^\infty u^ne^{-u}\,\mathrm{d}u
\end{align}
$$
Now look up the Gamma Function.


When using cross validation, shouldn't it be mandatory to pipeline the preprocessing steps?

Question: I'm currently experimenting with sklearns transformer classes in combination with pandas. During my application on a kaggle data set and being picky on not choosing information of the test data, I realized the following:
Typically, in those challenges you have a training data set and a test data set. We perform preprocessing and say we impute the missing values in the train and test data set by the mean of the training data. Then we do our usual stuff of finding a valid model.
But wait, let's forget the test data set for a moment (on kaggle you don't even have values for the response variable on the test set). We want to estimate the generalizability of our model, this can be achieved by using cross validation on the training data. But on each testing fold, we potentially imputed missing data, i.e. we have information of the corresponding training set!
For example imputing by mean will yield different values on the whole training set and each cv fold.
So, shouldn't data preprocessing steps always be included in the pipeline?
Can this be a severe problem?
Answer:
So, shouldn't data preprocessing steps always be included in the pipeline?

Ideally, it should. It's part of the model after all. A case can be made when you have lots of data, when the estimates across folds end being almost the same. But in general it's better to assess the whole model building process performance, and that includes preprocessing when it takes into account training data.

Can this be a severe problem?

With big data probably not. With small data (and include this categorical variables with a lot of levels), this can potentially add a huge bias to your predictions. It's mostly due to the fact that your estimates are of lower quality when you only use the train data, and the estimates might jump a lot across folds, degrading your model.
Comment: thanks for the assurance. I was just wondering, since I haven't found this point stated out explicitly in the literature.
Answer: In addition to @Firebug's answer,

As for imputation for the test data set: IMHO it is legitimate to use training data to do this: This would be an "imputation model" trained on training data and applied to test data, and in principle not different from deriving, say, the mean from training data and then applying this e.g. for centering to the test data as well.
However, you need to make sure no test data leaks into the training set: the test data needs to stay unknown.
From my application-centered point of view, pre-processing is part of the model building and needs to be included in the validation (just like hyperparameter tuning is), i.e. also needs to be re-calculated inside the cross validation loop.
Some preprocessing steps can be pulled out of the validation loop without risking a data leak:

the step in question calculates strictly within each case (i.e. does not consider any other cases but the one that is processed at the moment), and
there was no preprocessing step that considered more than one case up to this point.

In other words, the cross validation loop needs to start before the first (pre)processing step that uses more than one case.
If you need a citation for this, we wrote it e.g. in
Dochow, S.; Beleites, C.; Henkel, T.; Mayer, G.; Albert, J.; Clement, J.; Krafft, C. & Popp, J.: Quartz microfluidic chip for tumour cell identification by Raman spectroscopy in combination with optical traps. Anal Bioanal Chem, 2013, 405, 2743-2746.
DOI: 10.1007/s00216-013-6726-3
Comment: thanks for your answer. Could you define what you consider to be a *case*?
Comment: Well, the "case" is the independent unit of which data is measured. In my applications, case often equals patient. If you have only one measurement per case, then each case is a row in your data matrix. But you may have several rows (e.g. repeated measurements or closely related measurements, e.g. I have measurements that are obtained on a μm-scale grid and are more similar to each other than to a measurement of another patient) per case.


What is the proportion of tenors?

Question: I would like to know what is the percentage in the population that have a tenor voice.
I guess it is difficult to find this number. And I guess it changes depending on the region of the world, culture.
Actually, I would be interested in any study about the distribution of voices (tenor, etc.) in populations.
If we can't find any study, can you tell from you experience as a voice coach or singing teacher?
Comment: In my experience as a choir director and singer, the default assumption is that there are fewer tenors than baritones or basses.  It's a very common problem of amateur choirs to have plenty of baritones and many basses to choose from, but to suffer a chronic lack of tenors.  But on the other hand, untrained voices typically can't sing as high.  It's often said that many baritones are lazy tenors.  But as I said, the borders are not hard and fast, and being a "tenor" rather than a "baritone" is often only a matter of choice and training.  Thus the difficulty in answering your question.
Comment: Probably only 48 - 52 percent of the population can be tenors seeing as they are usually men. Then if you factor in an even spread of basses, barries, and tenors you would probably be looking at 13 - 16 percent at best.
Comment: So you say there is an even spread of tenor barrys and basses. Do you have any evidence of that?
Comment: I think you've answered your own question: it's difficult to find this number.  Not only because (at least as far as I know) no one has studied this, but because it's not at all clear how to exactly identify a "tenor" voice.  It's not like there are clearly defined boundaries here, not to mention differences in training, etc.
Comment: You can easily have people whose range goes from the low end of baritone to the high end of tenor, but whose favoured tessitura is either closer to tenor or baritone. This increases the difficulty of answering your question further. (As an aside for how hard voice classification can get for females, I'm female, and my singing range right now goes from the A below low C to the C above high C. I can hit even higher notes with warmups beforehand.)
Comment: Yes but still vocal coaches must have an idea
Comment: @NeilMeyer  why restrict it to men?  There are females who cover the tenor range.  The OP did ask for "population," not male (and let's stay away from the 358 FaceBook gender designations :-) )
Answer: According to this Reddit post, the numbers are roughly as follows:

Men: 20% Bass / 45% Baritone / 35% Tenor
Women: 15% Contralto / 35% Mezzo / 50% Soprano

For males, the ordering implied by these numbers agree with this statement in Merriam-Webster, which is listed under the entry for "baritone":

baritone In vocal music, the voice or register between bass and tenor, the most common category of male voice.

For females, the ordering implied by these numbers agree with this statement from Wikipedia:

The lyric soprano is the most common female singing voice.2
...
2: Aronson, Arnold Elvin; Bless, Diane M. (2009). Clinical Voice Disorders (4th ed.). New York, NY: Thieme Medical Publishers. p. 278. ISBN 978-1-58890-662-5.

Another user in that same Reddit thread states:

In The Diagnosis and Correction of Vocal Faults, James McKinney asserts that voice types are roughly distributed on a sort of bell curve, with most people falling into the middle voice types (mezzo/baritone), which lines up pretty well with my experience.

The graph below is taken from a study on the frequency of someone's natural speaking voice and it seems to provide at least some sort of evidence that might lend support for the claim that a bell curve distribution exists for vocal classifications. I read about this study on this page, which is also where the graph is from:

Given the information in the more credible sources, I am probably inclined to trust the rough numbers from the Reddit post. Glancing through the user TotoCotogni's other posts, he seems like a relatively credible source, as the Internet goes for hard-to-answer questions.
Note: this same question was asked in another music.SE question, but it doesn't look like a numerical answer was ever given.
Comment: I am so happy to see this, because my experience was always so different. At my music college (RCM), everyone had to be in the choir in first year. So, we had about 45 Sopranos, 55 Altos, 10 Tenors and 90 Basses. Maybe us boys just smoked too much - anyway, it sounded like a football crowd...
Comment: The Merriam-Webster reference seems to contribute nothing except the ordering, which can be considered as known.
Comment: I would've thought the same thing--that the ordering could be taken as a known/established fact--but when I was doing a little research I found conflicting orders. That source is supportive only insofar as it agrees with the ordering, as you say. That does count for something and I think is worth mentioning, but it's not everything in the sense that it doesn't completely validate the claims in the Reddit post. That's why I've tried to write my post with qualifiers (may, might, could, etc.).
Comment: @guidot I've adjusted the language to make clearer the way in which the two citations are supposed to provide (limited) support for the Reddit post.

What is the proportion of tenors?


Why are the embeddings of tokens multiplied by $\sqrt D$ (note not divided by square root of D) in a transformer?

Question: Why does the transformer tutorial in PyTorch have a multiplication by sqrt number of inputs? I know there is a division by sqrt(D) in the multiheaded self attention, but why is there something similar to with the output of the encoder? Especially because the original paper doesn't seem to mention it.
In particular (https://pytorch.org/tutorials/beginner/translation_transformer.html):
src = self.encoder(src) * math.sqrt(self.ninp)

or this (https://pytorch.org/tutorials/beginner/transformer_tutorial.html):
# helper Module to convert tensor of input indices into corresponding tensor of token embeddings
class TokenEmbedding(nn.Module):
    def __init__(self, vocab_size: int, emb_size):
        super(TokenEmbedding, self).__init__()
        self.embedding = nn.Embedding(vocab_size, emb_size)
        self.emb_size = emb_size

    def forward(self, tokens: Tensor):
        return self.embedding(tokens.long()) * math.sqrt(self.emb_size)

Note that I am aware that the Attention layer has this equation:
$$
\alpha = Attention(Q,K,V) = SoftMax( \frac{ Q K^\top }{\sqrt{D}} ) V
$$
and they argue why about it in the paper in a one of the margins (something about sum of variance being 1).
Is this related to that comment and how is it related? Is this mentioned in the original paper?
cross posted:

https://discuss.pytorch.org/t/why-does-the-transformer-tutorial-have-a-multiplication-by-square-root-of-the-number-of-inputs/126738/6
https://www.reddit.com/r/learnmachinelearning/comments/okfd7g/why_does_the_embedding_of_tokens_to_the/
https://www.reddit.com/r/pytorch/comments/op0z2t/why_are_the_embeddings_of_tokens_multiplied_by/
https://www.reddit.com/r/LanguageTechnology/comments/op13ep/why_are_the_embeddings_of_tokens_multiplied_by/
https://www.reddit.com/r/deeplearning/comments/opcnwy/why_are_the_embeddings_of_tokens_multiplied_by/
https://qr.ae/pGuVsC
Comment: Good question. How come you didn't ask in /r/ML instead of these tier-2 subreddits?
Comment: @bobcat I think there are issues posting things there in weekdays. Since I don't work on weekends I never end up being able to post there.
Answer: Note that the weights in word embeddings get initialized with zero mean and unit variance. Also, the embeddings get multiplied by sqrt(d) before being added to the positional encoding. The positional encoding is also on the same scale as the embeddings.
My hypothesis is that the authors tried rescaling the embeddings by various numbers (as they certainly did with attention), and this particular rescaling happened to work, because it made the embeddings much bigger than the positional encodings (initially). The positional encodings are necessary, but they probably shouldn't be as "loud" as the words themselves.
Answer: We multiply because we use learned embeddings to convert the input tokens and output tokens to vectors of dimension d_model in the embedding layer. Refer Section 3.4 on Embeddings and Softmax of the original paper. You'd also see that an object of TokenEmbedding is used in Seq2SeqTransformer for source and target token embeddings.
Answer:
As mentioned in the transformer paper 'Attention is all you need', the embedding matrix in the input and the pre-softmax linear transformation in the linear layer outside the decoder block share the same projection matrix parameters.
vocab_size->d in the embedding matrix W, and d->vocab_size in the pre-softmax linear transformation matrix W^T in the linear layer.
Sharing the parameters in the matrix related to embedding is called tied-embedding. The parameters are initialized with Xavier with N(0,1/sqrt(d)). So multiplying sqrt(d) keep the word embedding to stay with N(0,1).
The original position embedding (cos/sin) can be considered as hard-code. So the the variance of position embedding is zero. Adding position embedding and word embedding (scaled sqrt(d)) still makes the variance of the combined embedding 1.
Answer: In deep neural network layers, we want to retain the signals having normal distributions, and avoid diminishing and exploding gradients. This is the primary goal of weight initialization such as Xavier or He.
As demonstrated Building makemore Part 3: Activations & Gradients, BatchNorm, the activation functions are basically squashing (-inf to inf) range into (0, 1) or (-1, 1) or (0,). Because of this, the variance of layer output signal tends to shrink causing difficulties to maintain normal distribution. To alleviate the this squashing effect, weight initializations are incorporating expanding gain such as $\sqrt{2}$ for ReLU or $5/3$ for tanh activation functions.
Hence, what we need to look for is where such a squashing effect is occurring, which the expanding effect of the multiplying with $\sqrt(D)$ will alleviate it.
The Transformer Architecture does not use Activation Functions except the ReLU at the Position Wise Feed Forward sub layer. Then, this is the place to look at. Or, look at how the embedding layer weights are initialized which may require the alleviation by $\sqrt(D)$.


Show that $A$ is an ideal of $R$.

Question: Let $R$ be a ring. Suppose $\emptyset\neq A\subseteq R$ and $$a,b\in A\implies a+b\in A,$$
$$r\in R\  \text{and}\  a\in A\implies ar,ra\in A .$$
Further suppose for each $a\in A,$ there exists an integer $n\neq 0$ such that $na\in aR+Ra$. It is required to prove that $A$ is an ideal of $R$. I believe all that is needed to be proved is that $$a,b\in A\implies a-b\in A.$$
However I cannot understand how to proceed after the following point.
Assume $a,b\in A$. Then there exists $m\neq 0$ such that $ma=ar_1+r_2a$ and $n\neq 0$ such that $nb=bs_1+s_2b$, where $r_1,r_2,s_1,s_2\in R$.
Please give me a hint. Thanks.
Added later: $R$ can be embedded in a ring with identity as follows. Consider $R\times\mathbb{Z}$ with addition defined as $(r,m)+(s,n)=(r+s,m+n)$ and multiplication defined as $(r,m)(s,n)=(rs+nr+ms, mn)$. Then $R\times \mathbb{Z}$ is a ring with identity namely $(0,1)$. Furthermore we have $R\cong R\times\{0\}$ with $R\times\{0\}$ a subring of $R\times \mathbb{Z}$. So $R$ is embedded in a ring with identity.
Now let's take $a,b\in A$. Consider $(a,0),(b,0)\in A\times \{0\}$. Then $(a-b,0)=(a,0)+(0,-1)(b,0)\in A\times \{0\}$. So $a-b\in A$. Therefore $A$ is an ideal.
Is this argument alright?
Why is the stipulation "for each $a\in A,$ there exists an integer $n\neq 0$ such that $na\in aR+Ra$" given?
Comment: If not, you can extend each ring to have a unit element.
Comment: $r=-1$ and $b\in A$ implies $-b=rb\in A$
Comment: $a,b\in A$. Then $-b=(-1)b\in A$ and so $a+(-b)\in A$.
Comment: @Wuestenfux That wouldn't change $A$.
Comment: How do you know that $-b\in A$?
Comment: The assumption $1\in R$ is not given.
Comment: could you please check the "added later" part?
Comment: About the "added later" is the theorem 1.10 chapter III in algebra book of Thomas hungerford.
Answer: The condition $0\in A$ is ensured by $A\ne\emptyset$ and $0a\in A$. So what you need to prove is that $-a\in A$, whenever $a\in A$.
Let's choose $n\ne0$ with $na=ar+sa$. Then
$$
(-n)a=-na=-ar-sa=a(-r)+(-s)a\in aR+Ra,$$
so it's not restrictive to assume $n>0$.
By the first property, $(n-1)a\in A$ (this is the integer $1$); moreover $(-n)a\in aR+Ra$, so $(-n)a\in A$, by the second property, so
$$
-a=(n-1)a+(-n)a\in A
$$

The "added part" in the question glosses over the fact that you'd need to prove that $(0,-1)(b,0)\in A\times\{0\}$, which is the same as proving that $-b\in A$.

Show that $A$ is an ideal of $R$.


AccessibilityService: Query all current notifications?

Question: With the new NotificationListenerService introduced in 4.3, it's possible to query all currenty active notifications using NotificationListenerService.getActiveNotifications(). Is something similar possible with the AccessibilityService? I wouldn't mind keeping track of them myself (store them temporarily once a new notification comes in), but it seems that there's no "onRemoved"-event for the AccessibilityService, so it's impossible to keep track of which notifications are still active and which have already been dismissed?
Answer: Apparently this isn't possible on pre-4.3. I've resorted to storing the notifications locally as they come in, but this isn't a very good approach since there's no way to tell when a notification was removed.


LDAP access denied (Error 49 data 5) on win server 2012R2

Question:
This is a fresh installation (windows 2012 R2), I disabled everything related to kerberos, but that did not fix the problem.
All users have this problem, the Credentials are correct.
Bind type: simple
Comment: I notice you have no internet connection, what are you using for a time server? Kerberos and Active Directory authentication is dependent on accurate time stamps.
Comment: It's a Vritual Machine. The time should be the same as the host (configured with same Timezone), But still the error is in the server itself. This is a simple bind, not Kerberos
Answer: There was a problem with my copy of windows. I downloaded another iso and that fixed the problem.


Bayesian regression full conditional distribution

Question: I have a problem with the derivation of the full conditional distribution of the regression coefficients in a simple Bayesian regression. The source of the following equations is:

Lynch (2007). Introduction to Applied Bayesian Statistics and Estimation for Social Scientists, page 170 & 171.

The posterior distribution (with uniform priors on all parameters) is given by:
$$
P(\beta, \sigma | X, Y) \propto (\sigma^{2})^{-(n/2+1)} \exp\{-\frac{1}{2\sigma^{2}}(Y-X\beta)^{T}(Y-X\beta)\}
$$
Hence the full conditional distribution of the coefficient $\beta$ only requires the kernel:
$$
\exp\{-\frac{1}{2\sigma^{2}}(Y-X\beta)^{T}(Y-X\beta)\}
$$
where $\beta$ is a $p$ dimensional parameter vector $X$ is a $n\times p$ predictor matrix and $Y$ is the $n$ dimensional vector of responses.
This can be expanded into:
$$
\exp\{ -\frac{1}{2\sigma^{2}} [Y^{T}Y - Y^{T}X\beta - \beta^{T}X^{T}Y + \beta^{T}X^{T}X\beta] \}
$$
Since Y is constant with respect to $\beta$, it can be dropped and the middle two terms can be grouped together. This leads to:
$$
\exp\{ -\frac{1}{2\sigma^{2}} [ \beta^{T}X^{T}X\beta - 2\beta^{T}X^{T}Y ] \}
$$
The next step confuses me. The author multiplies the whole equation by $(X^{T}X)(X^{T}X)^{-1}$. The problem is not the multiplication itself. Its the authors result:
$$
\exp\{\frac{1}{2\sigma^{2}(X^{T}X)^{-1}} [\beta^{T}\beta - 2\beta^{T}(X^{T}X)^{-1}(X^{T}Y) ] \}
$$
What is confusing to me is the multiplication inside the brackets. For me pre-multiplying by $(X^{T}X)^{-1}$ leads to:
$$
(X^{T}X)^{-1}\beta^{T}X^{T}X\beta - 2(X^{T}X)^{-1}\beta^{T}X^{T}Y
$$

Why is it allowed to swap the terms $\beta^{T}$ and $(X^{T}X)^{-1}$?

Since $\beta^{T}X^{T}X\beta$ gives you a scalar value and multiplying it by $(X^{T}X)^{-1}$ results in a square matrix, this is not the same as $\beta^{T}(X^{T}X)^{-1}(X^{T}X)\beta$. But the author seems to "ignore" this in his solution. What am I missing?
Edit
Here is a picture of the relevant pages:
Answer: As you show in the reproduction what is written in this book, the solution is incorrect for the simple reason that the quantity $(X^{T}X)^{-1}$ is a $p\times p$ matrix, not a scalar. Hence you cannot divide by $(X^{T}X)^{-1}$. (This is a terrible way of explaining this standard derivation!)
What you can write instead is
$$
\beta^{T}X^{T}X\beta - 2\beta^{T}X^{T}Y=\beta^{T}X^{T}X\beta - 2\beta^{T}(X^{T}X)(X^{T}X)^{-1}X^{T}Y
$$
These are the first two terms of the perfect squared norm
$$
\left(\beta-\hat\beta\right)^T (X^{T}X) \left(\beta-\hat\beta\right)
$$
where $\hat\beta=(X^{T}X)^{-1}X^{T}Y$ is the least square estimator.
Therefore, the full conditional posterior distribution of $\beta$, given $\sigma$ and $\hat\beta$ is a normal distribution$$\mathcal{N}_p(\hat\beta,(X^{T}X)^{-1})$$

Note: The prior on $\sigma$ corresponding to the joint posterior is $1/\sigma^2$ rather than a uniform prior.
Comment: To be fair, Lynch makes note in footnote 2 that one cannot "divide by the matrix" and discusses the actual math, but uses the term "divides" for pedagogic ease.


WinForm freezes while opening a UserControl

Question: I've a WinForm that contains Navigation menu that display UserControls. Now everything works fine except the form freezes when I try to open a Devexpress UserControl from another UserControl.
Here is the code I used:
private void btnOpenUserControl2_Click(object sender, EventArgs e)
{
   UserControl2 uc2 = new UserControl2(ID);

   this.Parent.Controls.Add(uc2);
   uc2.Dock = DockStyle.Fill;
   this.Hide();
   uc2.Show();
}

What could be the possible reason?
Comment: What's the reason for the call to `this.Hide()`?
Comment: My assumption was to hide the current UserControl before UserControl2 is displayed. But, I can take that out, right?
Comment: @aby yes take that out
Comment: Even so, the problem persists
Comment: @aby what do you have in the constructor or load event of uc2?
Comment: I've a method call to populate the form data based on the ID passed from UserControl1.
Answer: the form freezes is most likely one of two things:
in your UserControl2 you hndle an event ( for example form_Load), and in that event you have an infinity loop. the other reason is you have some handeled exception, or a long flow, very long flow, that cause your program to ran slow, so slow you think it's stuck.
in anyway it be very helpful if you share your UserControl2 code. but easier will be if you just pause the program while debugging it in VS when it get stuck and see where is it stucked. from there it should be fairly easy
Comment: You are right, I found out that binding DevExpress LookUpEdit control with more than 50,000 rows is taking longer than expected. Any idea how I should approach the problem?
Comment: @aby, open another thread and run those 50,000 rows in that new thread(a background thread). that way the UI won't freeze. if you need a help doing that look here - http://stackoverflow.com/questions/363377/how-do-i-run-a-simple-bit-of-code-in-a-new-thread . and please accept my answer
Answer: The solution to this problem is to use the GridLookupEdit control in server mode instead.  In this case, the gridLookupEdit will load only a small portion of data from the database and will continue loading data when you scroll the grid.  To learn how to adjust the gridLookupEdit in server mode, please refer to Server Mode topic.


Given the estimator find wheter

Question: X has an uniform distribution on interval $(0,\theta]$ where $\theta$ is a positive parameter
Given the estimator:
$$T(X_1,X2, \ldots, X_n)=\frac{2}{n} \sum_{i=1}^n X_i$$
Find whether this estimator is consistent (i am not certain whether it is a best english equivalent of what i mean). But in symbols i mean:

Generaly speaking i know it is satisfied by
$R(\theta)=\frac{\theta^2}{3n}+0$ and the fact that this expression converges to zero. However i do not understand why and where does this answer comes from.
Comment: What is the distribution of $X_i$?
Comment: Updated the question
Comment: What you've stated is that you want to know whether the estimator converges in probability to $\theta$
Comment: Could you elaborate on that i mean is my answer correct or not? Where AM i wrong?
Comment: Is there a typo in your answer $R(\theta)$, why is there a $0$ added in
Comment: I mean that is what i had as an answer IT is not typo something there should be equal to zero i think.
Answer: Assuming that $X_i$ are i.i.d.
1) You can simply use the WLLN and continuous mapping theorem to deduce that $\frac{1}{n}\sum_{i=1}^n X_i \xrightarrow{p}EX = \frac{\theta}{2}$, hence $\frac{2}{n}\sum_{i=1}^n X_i \xrightarrow{p} \theta$.
2) You can use the theorem that if  $\lim_{n\to \infty} MSE(\hat{\theta}_n) = 0$ then $\hat{\theta}_n$ converges in probability to $\theta$. In your case it is straightforward to show that
$$
MSE(\hat{\theta}) = Var(\hat{\theta})+b^2(\hat{\theta}) = \frac{2^2}{n^2}nVar(X_i) = \frac{\theta^2}{3n}
$$
which is clearly goes to  $0$ as $n \to  \infty$.
(Note that $EX_i = \frac{\theta}{2}$ and $Var(X_i) = \frac{\theta^2}{12}$)
Comment: Great answer! Thank you!

Given the estimator find wheter


How can I solve this problem"The view 'Index' was not found"

Question:
InvalidOperationException: The view 'Index' was not found. The
following locations were searched:
/Views/Home/Index.cshtml
/Views/Shared/Index.cshtml

See this image:

I try this dotnet dev-certs https --trust in cmd to make sure that certificate not valid when run ASP.NET Core in browser, but the problem still happen.
Comment: Did you check that the `Index.cshtml` is really located under the `\Views\Home` folder in your project? What version of the ASP.NET Core are you using?
Comment: Yes, I did , 7.0.9
Comment: Can you share the folder structure of your source Code?
Answer:
InvalidOperationException: The view 'Index' was not found. The
following locations were searched.I have searched everywhere and have
not been able to find the problem.

Actually,it would nicer, if you could have share your project design. Acoroding to your error message, which means compiler would be searching views folder two directory inside your project parent directory but in really that's obviously not there. In your scenario, home or shared directory doesn't have index view or you are not calling them accordingly in controller you can see below:

Consequently, you are getting this very exception which I have reproduced accordingly as following:

Solution:
Make sure, your views are available in Home or Shared folder or the folder you are referring to. You could resolve your exception by providing the correct path. You can use either of the following to get rid of your issue.
 return View("./Views/Home/Index.cshtml");

Note: This ./ is meaning the compiler would search Views folder one directory down from the parent directory which is the correct location.
return View("~/Views/Home/Index.cshtml");

Note: You can use the  tilde ("~") operator in any path-related property in server controls which would search the path.
Output:

Note: If you would like to know more details on Directory browsing you could check our official document here


Uniqueness of Pseudo inverse

Question: Given $X\succeq0$, we can write the SVD decomposition as:
\begin{align}
X&= U \begin{bmatrix}\Omega&0\\0&0 \end{bmatrix}U^T,
\end{align}
where $U$ is an orthogonal matrix and $\Omega\succ0$.
My question is whether
\begin{align}
Y&= U \begin{bmatrix}\Omega^{-1} &0\\0&0 \end{bmatrix}U^T,
\end{align}
is the unique Moore-Penrose pseudo inverse of $X$.
Comment: Every matrix has a unique pseudo inverse.
Answer: I assume that you are referring to the Moore-Penrose peudoinverse and that $\Omega \succ 0$ means that $\Omega$ is positive definite, which implies that $\Omega$ is symmetric.
Yes, your formula for the psuedoinverse is correct, and (as is generally the case) the MP pseudoinverse is unique.
Answer: If a matrix $M$ is a Moore-Penrose inverse of $A$, then $$AMA=A,~~~ MAM=M,~~~(AM)^{'}=AM,~~~(MA)^{'}=MA.$$
Omitting the existence, we just prove the uniqueness. Suppose there are two candidates MP inverses $M_1,M_2$ of $A$. Then
$$M_1 = M_1 A M_1 = M_1 (AM_1)^{'}=M_1 M_1^{'}A^{'} = M_1 M_1^{'} A^{'} M_2 A^{'}$$
$$= M_1 M_1^{'} A^{'} A M_2=M_1A M_1 A M_2 = M_1 A M_2=M_1 A M_2 A M_2 = M_1 A A^{'} M_2^{'} M_2$$
$$= A^{'} M_1^{'} A^{'} M_2^{'} M_2 = A^{'} M_2^{'} M_2 = M_2AM_2=M_2.$$

Uniqueness of Pseudo inverse


Work done to pump water out of a conical tank into a window above it

Question: Water is pumped from a conical tank of top radius 3 ft and a height of 5 ft to a window 10 ft above the tank. The tank is completely full of water. How much work is done?

This is what I have so far: $$\int_0^{15}\pi(\text{radius of slice})^2 \cdot (62.4) (15-h)\,dh$$
I am having a problem figuring out the radius of the slice
Comment: Hint: chop the conical tank into infinitesimal cylinders.
Comment: You might tell us poor Europeans where that $62.4$ comes from. (Also you might set the upper limit of integration to $5$ instead of $15$.)
Comment: Thanks for pounds mass, instead of the vague pounds, which could be pounds-force or pounds-mass.
Comment: This is what i have so far: integral [0, 15] pi(radius of slice)^2 * (62.4) (15-h) I am having a problem figuring out the radius of the slice.
Comment: @TonyK The density of water at atmospheric pressure and $\approx 50^\circ \text{F}$ is $62.4 \frac{\text{lbm}}{\text{ft}^3}$.
Answer: To draw the problem in a coordinate system, choose a positive $x$ axis in the downward direction since the motion is vertical. Take the window at $x = 0$, the top of the tank at $x = 10$ and the bottom of the tank at $x = 15$. The partition of water is therefore the closed interval $[10,15]$ on the $x$ axis.
Take the radius of the cone at $x_i$, the function is $9-3x_i/5$. Then take an element of volume as a circular disk having thickness $\Delta_i x$ and the radius of the cone at $x_i$, the volume is $\Delta_i V = \pi (9-3x_i/5)^2 \Delta_i x$. Then take $62.4$ as the weight of water in pounds per cubic foot, the force required to pump an element is $\Delta_i F = 62.4 \pi (9-3x_i/5)^2 \Delta_i x$.
If $\Delta_i x$ is close to $0$, then the distance through which an element moves is approximately $x_i$. Thus the work done pumping an element to the window is $\Delta_i W = 62.4 \pi x_i (9-3x_i/5)^2 \Delta_i x$ and the integral is:
$$
W = 62.4 \pi \int_{10}^{15} \! x(9-3x/5)^2 \, \textrm{d}x
$$
Answer: We'll assume no frictional losses, since we aren't given any information about the flow.
Let the density of water be $\rho = 62.4 \frac{\text{lbm}}{\text{ft}^3}$, the base radius of the cone be $R = 3 \text{ ft}$, the height of the cone be $h = 5 \text{ ft}$, and the vertical distance from the top of the cone to the final water elevation be $d = 10 \text{ ft}$. Of course, the gravitational acceleration is $g = 1 \frac{\text{lbf}}{\text{lbm}}$.
Consider an infinitessimally thin layer of water in the tank at height $y$ above the ground, with a height $d y$. By similar triangles, the radius of this layer of water is $\frac{y}{h} R$. Therefore, the mass of this layer is $d m = \rho d V = \pi \rho \frac{y^2}{h^2} R^2 d y$. The change in gravitational potential energy in bringing this layer to the final elevation is $d U = g (h + d - y) d m$. Therefore, the total energy required is
$$\Delta U = \int_{y = 0}^{y = h} d U = \pi \rho g \frac{R^2}{h^2} \int_0^h y^2 (h + d - y) d y$$
$$= \pi \rho g \frac{R^2}{h^2} \left[ \frac{1}{3} (h + d) y^3 - \frac{1}{4} y^4 \right]_0^h$$
$$= \frac{1}{3} \pi \rho g R^2 h d \left[ 1 + \frac{1}{4} \frac{h}{d} \right]$$
Notice that if we let $V = \frac{1}{3} \pi R^2 h$ be the volume of the cone, then the mass of the water is $m = \rho V = \frac{1}{3} \pi \rho R^2 h$, meaning
$$\Delta U = m g d \left[ 1 + \frac{1}{4} \frac{h}{d} \right]$$
This result is actually quite intuitive, since you are bringing the entire water mass to the same final height, and the center of mass of a cone is located $\frac{1}{4}$ of the height from its base.
I'll leave it to you to substitute the numerical values.


How to tell if two matrices are similar?

Question:
Two n-by-n matrices A and B are called similar if $$
     \! B = P^{-1} A P $$ for some invertible n-by-n matrix P.
Similar matrices share many properties:

Rank
Determinant
Trace
Eigenvalues (though the eigenvectors will in general be different)
Characteristic polynomial
Minimal polynomial (among the other similarity invariants in the Smith normal form)
Elementary divisors

Given two square matrices A and B, how would you tell if they are similar?

Constructing a $P$ in the definition seems difficult even if we know they are similar, does it? Not to mention, use this way to tell if they are similar.
Are there some properties of similar matrices that can characterize
similar matrices?

Thanks!
Comment: @HuiYu They are OK, but they don't allow to characterize similarity completely. I.e. two non-similar matrices can have the same characteristic polynomial.
Comment: You can have a look here: [Frobenius noral form](http://en.wikipedia.org/wiki/Frobenius_normal_form).
Comment: @DanShved: Thanks! So decomposing each matrix into some canonical form, such as Jordan form, is a way, by definition.
Comment: I think for finite dimensional spaces characteristic polynomials are the best.
Answer: For a pedagogical discussion, see Jakob Stoustrup, Linear Algebra in the Classroom: A Note on Similarity. (The author claims on his website that this paper appeared in International Journal of Mathematical Education in Science and Technology, 26(6): 917-920, 1995, but it wasn't found on the journal's website.)
The author essentially proposes a generalised eigenspace approach (see lemma 1), so that no matrix transformation is needed.
Comment: The link is broken.
Comment: @Chad It's fixed now. Thanks for the information.
Comment: What a strange reference. It indeed isn't listed on the journal's web site, but my institution's library system says it has physical copies. The article itself is a bit bizarre--the lemma it purports was new is obviously much older. Honestly, anybody who understands the theoretical basis behind Jordan canonical form would know that lemma is essentially a key step in the proof.
Comment: @JSwanson I think what he meant by "new" is not the result, but his pedagogical approach. E.g. when Sheldon Axler's famous article "Down with Determinants!" was out (published on the same year as Stoustrup's paper), my impression is that many people thought that the way he *taught* linear algebra was new. But certainly the results as well as the theory behind it were well known.


How use if else in mutate function in R

Question: I have a data.frame DT_new with 4 columns :

Graduated  (Date format)
Work   (Date format)
Married  (Date format)
Jumlah (Double format)

Sample:
 Graduated         Work      Married   Jumlah
2015-05-01   2015-05-02   2015-05-03       20
        NA   2015-05-02   2015-05-03       20
        NA           NA   2015-05-03       20
        NA   2015-05-02           NA       20

I need to aggregate Jumlah by date in Graduated or Work or Married

when Graduated value is not NA, use date from Graduated
when Graduated value is NA , use another value from Work or
Married

format what I want is :
     Dates   Total
2015-05-01      10
2015-05-02      40
2015-05-03      30

I have tried aggregate with group by in R but just group by 1 column (Graduated), such as:
DT_Totals = DT_Total %>%
  group_by(Graduated) %>%
  summarise(Total= sum(Jumlah)) %>%
  arrange(Graduated)

How can I handle my problem?
Answer: You need first create new column and then group over them.
I got function to return first not NA element in vectors defined as:
first_not_na <- function(...) {
    Reduce(list(...), f=function(x,y) {
        x[is.na(x)] <- y[is.na(x)]
        x
    })
}

And you can use it as follow
DT_new %>%
    group_by(Date = first_not_na(Graduated, Work, Married)) %>%
    summarise(Total = sum(Jumlah)) %>%
    arrange(Date)

or splitting to two steps:
DT_new %>%
    mutate(Date = first_not_na(Graduated, Work, Married)) %>%
    group_by(Date) %>%
    summarise(Total = sum(Jumlah)) %>%
    arrange(Date)

Comment: hi @Marek thx for your answer..i want to ask, function first_not_na is remove NA in what column ? thx
Comment: @ihsansat Look how I use it. It takes two or more columns and try to replace NA in first one using values from others in order which they are given.
Comment: hi @Marek, i have tried you solutions,,,i tried to put into another data.table `DT_newer` , when i View(`DT_newer`) , still row with row with column Dates with values NA :(
Comment: @ihsansat But other dates are NA or not?
Comment: my mistak @Marek, i need to filter my data before use function..it's work...thx :)
Comment: Hi @Marek, if i want to add another columns so column use is  Graduated, Work, Married, Died ,, i has tried to modified your function, i just add '.' in arguments,,..but it's not work "unused arguments" how can fix it? thx
Comment: @ihsansat You don't need to modify it. Just use `first_not_na(Graduated, Work, Married, Died)`. Three dots `...` it's special R symbol used in function to pass undefine number of parameters to function.
Comment: Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/84639/discussion-between-ihsansat-and-marek).
Answer: Just create a new date column using ifelse:
DT_new %>%
  mutate(Date1 = ifelse(!is.na(Graduated), Graduated, ifelse(!is.na(Work), Work, Married))) %>%
  group_by(Date1) %>%
  summarise(Total = sum(Jumlah)) %>%
  arrange(Date1)

Update
In case the dates are numeric (Date) type:
DT_new %>%
  mutate(Date1 = ifelse(!is.na(Graduated), Graduated, ifelse(!is.na(Work), Work, Married))) %>%
  mutate(Date1 = as.Date(Date1, origin = "1970-01-01")) %>%
  group_by(Date1) %>%
  summarise(Total = sum(Jumlah)) %>%
  arrange(Date1)

Comment: Hi @bergant thx for you answer... i have tried...but it's still make `Date1 ` with value NA :(
Comment: i want ask you, when confirmed = "2015-07-01" Dates1 = "16617"  using mutate function you, want mean about "16617" ? thx
Comment: I guess you have factors in your data. Read the data with `stringsAsFactors = FALSE` or convert your data - for example: `DT_new1$Work <- as.character(DT_new1$Work)`
Comment: @ihsansat `ifelse` is not good in handling dates. 16617 is internal representation of "2015-07-01" date.
Comment: It has nothing to do with ifelse. Factors are consistent only in one column (enumerating different dates). When comparing between different columns you **have to** convert them to character or date type.
Comment: @bergant Yes it does: `ifelse(1==1, Sys.Date(), Sys.Date()+1)`
Comment: @Marek Oh, I see. Thanks.
Comment: @ihsansat See update for converting integers back to dates.

How use if else in mutate function in R


How to read xlsx files in Mirth

Question: I am new to mirth and trying to see if anyone is knowledgeable on how mirth handles excel files.
Right now, My mirth channel works great with zip files, csv, and txt files but not xlsx. When it encounters an xlsx file it errors out.
<code>[2022-02-08 09:37:00,037]  ERROR  (com.mirth.connect.donkey.server.channel.Channel:1288): Error processing message in channel xlsx (5447c43e-7aee-4ade-9460-d490edef7f04).
com.mirth.connect.donkey.server.channel.ChannelException:
    at com.mirth.connect.donkey.server.channel.Channel.dispatchRawMessage(Channel.java:1243)
    at com.mirth.connect.donkey.server.channel.SourceConnector.dispatchRawMessage(SourceConnector.java:192)
    at com.mirth.connect.donkey.server.channel.SourceConnector.dispatchRawMessage(SourceConnector.java:170)
    at com.mirth.connect.connectors.file.FileReceiver.processFile(FileReceiver.java:418)
    at com.mirth.connect.connectors.file.FileReceiver.processFiles(FileReceiver.java:328)
    at com.mirth.connect.connectors.file.FileReceiver.poll(FileReceiver.java:239)
    at com.mirth.connect.donkey.server.channel.PollConnectorJob.execute(PollConnectorJob.java:49)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)Caused by: com.mirth.connect.donkey.server.data.DonkeyDaoException: org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00
    at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.insertContent(JdbcDao.java:287)
    at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.insertMessageContent(JdbcDao.java:199)
    at com.mirth.connect.donkey.server.data.buffered.BufferedDao.executeTasks(BufferedDao.java:110)
    at com.mirth.connect.donkey.server.data.buffered.BufferedDao.commit(BufferedDao.java:85)
    at com.mirth.connect.donkey.server.data.buffered.BufferedDao.commit(BufferedDao.java:72)
    at com.mirth.connect.donkey.server.channel.Channel.dispatchRawMessage(Channel.java:1215)
    ... 8 moreCaused by: org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:168)
    at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:135)
    at com.mirth.connect.donkey.server.data.jdbc.JdbcDao.insertContent(JdbcDao.java:284)
    ... 13 more ```
</code>
Answer: At present, Mirth connect is not able to natively read Excel files. You might want to do a custom java class for this then import the jar file into the custom-lib. Overall, it all depends on what you wish to achieve. For instance, you can loop through and grab the Excel files using file reader on the source connector then send via HTTP sender on a Channel's destination connector, to any HTTP end point (i.e. REST API app in PHP or R) capable of processing the Excel files, and returning JSON/XML responses if needed. If all you want is a saving entries to a database, you can consider using a command such as executeRuntimeCommand("C:\Windows\System32\cmd.exe", "/C", "C:\sql_save.bat")
inside a transformer step to launch a Windows batch script to do the save for you.
Answer: The obvious answer in the Java world (which you can call from Mirth) doesn't work. Poi, the standard framework to read from Office files can't be used. It has lots of dependencies, most of which are already inside Mirth. However, no version of Poi seems to match what Mirth has loaded. Thought I could work the dependencies such that I would get a fresh classloader, but apparently not.
Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).

How to read xlsx files in Mirth


How to prove convergence to $1-\frac{1}{\log(n)}$

Question: I am trying to prove that:
$$\left(\frac{1}{2}+\frac{1}{2}\left(\frac{1}{3}+\frac{2}{3}\left(\frac{1}{5}+\frac{4}{5}\left(...+\frac{p_{m-2}-1}{p_{m-2}}\left(\frac{1}{p_{m-1}}+\frac{p_{m-1}-1}{p_{m-1}}\left(\frac{1}{p_{m}}\right)\right)...\right)\right)\right)\right)\approx1-\frac{1}{\log(n)}$$
Where $m=\pi(\sqrt{n})$
Some help would be really welcomed! Thanks in advance.
Comment: Use `$\log$` for $\log$ instead of `$log$` for $log$.
Comment: I don't understand what is the LHS. Are you trying to estimate the probability that $n$ is not prime ? Then assuming a probability distribution such that $Pr[n \not \equiv 0 \bmod p_j, j \in 1 \ldots J] \approx \prod_{j=1}^J Pr[ n \not \equiv 0 \bmod p_j] \approx \prod_{j=1}^J (1-\frac{1}{p_j})$ then $Pr[n$ is not prime $] = 1-Pr[n$ is prime $]=1-\prod_{ p \le \sqrt{n}} Pr[ n \not \equiv 0 \bmod p] \approx 1- \prod_{ p \le \sqrt{n}} (1-\frac{1}{p})$. That the last expression is $\approx 1-\frac{1}{\log n}$ is quite equivalent to the [PNT](https://en.wikipedia.org/wiki/Prime_number_theorem)
Comment: $m=\pi(\sqrt{n})$? $m$ should be an integer?
Comment: @MartinRosenau I think $\pi(\cdot)$ is the [prime counting function](https://en.wikipedia.org/wiki/Prime-counting_function).
Comment: Thanks for your format advice @Shaun; it is already fixed.
Comment: @MartinRosenau, as already commented, $\pi(\sqrt{n})$ is the prime counting function.
Comment: @reuns, thanks a lot for your comment; indeed, I knew that the assertion is quite equivalent to the PNT, so I was looking for some way to prove the convergence to $1-\frac{1}{\log(n)}$ and bound it without using the PNT.
Answer: I have found that Merten's third theorem, which remarkably can be proved without the assistance of the Prime Number Theorem, establishes that:
$\lim_{x\to \infty}\log{(x)}\prod_{p\le{x}}\left(1-\frac{1}{p}\right)=e^{-\gamma}$
Using this theorem, the proof I was looking for follows easily.


Hierarchical menu of all Terms of a Vocabulary & Contents of the Terms

Question: I have Category vocabulary under which there are several terms, i.e Tabs, Smartphones, Cellphones, Umbrellas. There may me one or more contents of custom type under each of these terms (A content could be linked with one term only). I want to generate a menu (automatically, if possible) like this:

Now I've installed Taxonomy Menu module which allows to generate menu with one level hierarchy only (with only terms of a vocabulary, but no contents of the terms).
Is there a pretty solution of my problem? Thanks for caring :)
Comment: What do you mean by contents of the terms?
Comment: @shoaib-nawaz I have some custom Content-types named `products` - just like articles. Each content have exactly one `term` assined with it.

By "contents of a the terms" I meant all contents havin that with a particular term.
Answer: I think this is just a straightforward menu - nothing automatic needed. When you add a new node, just choose it's parent menu item.
Comment: Hmm That's a solution I guess. But what it the creator does not has permission to add menu items?

I think generating it dynamically would be a better solution. Thanks for the answer by the way :)
Comment: You can limit which menu a node can be added to. It's built in to D7.
Answer: Here's a module I found that comes with its own caveat: "...if incorrectly configured it could create thousands of menu items."
http://drupal.org/project/views_build_menu
Answer: What kind of menu block are you using to display the menu?  You may need something like the nice menus module to display the hierarchy.  I have generated taxonomy menus of three levels using Taxonomy Menu.
Comment: The problem is not about generating Taxonomy Menu. What I want is listing associated contents under each of these taxonomy `terms`


Calling multiple $http services in a for loop from Angularjs controller and services javascript promise issue

Question: My Angularjs application have a controller and services java scripts. Services call WCF REST services using $http. I need to call multiple services in a For loop based on data in object used in For loop. Below is the scenario:
"I want to upload few files to a folder on server and save file names to database. I have an array of file details that user selected to upload. I want to verify file names already in database (based on EntityId) then rename new file before uploading to server." So below is the for loop and service calls:
for (var i = 0; i < filesDataToUpload.length; i++) {

        var fileDataToUpload = filesDataToUpload[i];

        FileUploadService.GetUploadDetailsByEntityId(fileDataToUpload.ENTITY_ID).success(function (response) {
            $scope.UploadDetailsByEntity = response;
        });

        var newFileName = VerifyAndRenameBeforeSave(fileDataToUpload, $scope.UploadDetailsByEntity);

        FileUploadService.UploadFile(newFileName, fileData).success(function (data) {
            //some code
        }).error(function (error) {
            $scope.error("An error occured while uploading file -" + error.ExceptionMessage);
            isSuccess = false;
        });

        FileUploadService.InsertUploadDetails(FileUploadDetails).success(function (data) {
            //some code
        }).error(function (error) {
            $scope.error("An error occured while creating the Entity -" + error.ExceptionMessage);
            isSuccess = false;
        });

    }

Problem:
Problem:
My problem is - I do not get the response from first service call that I want to use in subsequent methods of verifying and renaming the files and saving to database. I found that it is because of the promise behavior of $http. So I moved my file verify and renaming methods and save part of functionality inside the
.success(function(response){
//verify that file does not already exists and rename it to the next number of file
//upload the file to the server

//Save new file name and other details to the database
}

But it works only for first time and not the next iteration of the for loop. For second iteration it retrives the record for first EntityId.
I already checked this and this
Can any one help me to get this resolved?
Answer: It's happening because the .success function returns a promise while it actually waits for the async task's response. Try it like this -
for (var i = 0; i < filesDataToUpload.length; i++) {
        var fileDataToUpload = filesDataToUpload[i];
        FileUploadService.GetUploadDetailsByEntityId(fileDataToUpload.ENTITY_ID).success(function (response) {
            $scope.UploadDetailsByEntity = response;
            var newFileName = VerifyAndRenameBeforeSave(fileDataToUpload, $scope.UploadDetailsByEntity);

            FileUploadService.UploadFile(newFileName, fileData).success(function (data) {
                    //some code
                    FileUploadService.InsertUploadDetails(FileUploadDetails).success(function (data) {
                            //some code
                    }).error(function (error) {
                            $scope.error("An error occured while creating the Entity -" + error.ExceptionMessage);
                            isSuccess = false;
                    });
            }).error(function (error) {
                    $scope.error("An error occured while uploading file -" + error.ExceptionMessage);
                    isSuccess = false;
            });
        });
}

Comment: Thanks @atefth, I tried doing that but my for loop itself does not wait to complete first iteration and finishes then later on my .success returns result.
Comment: For some reasons, I decided to change my implementation logic that avoided this issue. I am doing it like below:
Comment: 1. Moved the `FileUploadService.GetUploadDetailsByEntityId` and `VerifyAndRenameBeforeSave` logic to happen on the fly when user selects each file to upload (I want to select one file at a time and not multiple files). This helped me to avoid problematic service call to be executed inside the for loop. 2. I moved rest of the For loop code in a function. ` angular.forEach(filesDataToUpload, function (fileDataToUpload) {
            loop(fileDataToUpload);
        }, 1000);`


PHP API call timing out on shared hosting using file_get_contents()

Question: So I'm trying to query an API and get a JSON object back. All works well on my local WAMP install, but when I deploy it to the server, the API query fails :(
I've tried using cURL and file_get_contents(). With the same outcome.
I've tried two hosts, same problem.
I looked at this question which sounds exactly like my problem. But no luck.
And this other question sounds more like my problem, the problem was to do with the hosts firewall. But no detail on how he fixed it (and I can't comment yet)?
The script executes fine(on server) if the URL is say, some wikipedia page. But not the API service I'm targeting.
The API uses port 8080, so I'm suspecting that's got something to do with it, now I'm after tips on how I might be able to adress this.
Thanks.
[Update:] My Webhost says the problem is with my (quote) "codes"... So here's my test script. (Which runs fine on localhost)
<?php
$APIcall = 'http://ontoserver.csiro.au:8080/ontoserver/resources/resolve?id=392608003';

echo 'Contacting: ' . $APIcall.'
';
$OntoResponse = file_get_contents($APIcall, true);

echo 'Fetch data:
';
var_dump($OntoResponse);
?>

But when I publish it I get the following error: "Warning: file_get_contents(http://ontoserver.csiro.au:8080/ontoserver/resources/resolve?id=392608003) [function.file-get-contents]: failed to open stream: Connection timed out"
I'm stumped..
Thanks again.
Comment: Did you try insert port to the url? E.g.: host:8080/path
Comment: Thanks, yes my request is like http://APIprovider.com:8080/service/data?id=1234. I'm sure it's a server side problem with my host (I'm awaiting advice from them)
Answer: The script executed without problem on my local WAMP server. My solution was to host the script on a different host (Azure), and it worked fine first time. (no special setup)
I never worked out the specific cause for this, but I'm fairly confident it's something in my shared web hosting configuration. I was trying to host my script using budget shared hosting service(Crazy Domains).
Note: Other than this, I've been happy with my host provider. But help or configuring setup to do this seems to be beyond their offering.


Failover Cluster file server performance issue with Windows Server 2016

Question: I've run into an interesting file server transfer speed performance issue with a failover cluster I recently configured in Server 2016. The specific issue is that when I access the file share from the clustered storage path (e.g. \\store01\share01) file transfer speed (writes in particular, it seems) is way, way slower than when I access it via the local path on the current owner node (e.g. \\srv04\e$\Shares\Share01).
For example, I copied 499 .txt files (totaling 26.07 MB) using Robocopy:

\\srv04\e$\Shares\Share01: 0:0:03 - 635 MB/min
\\store01\share01: 0:02:20 - 11.286 MB/min

This is an issue regardless of the current owner node or where the data is transferred from. Although I didn't follow it at the time, I more or less installed and configured the service as indicated in this guide. I've tried messing with a few settings, but they're all back to default (as far as I know). I've looked around a bit and haven't found anything specifically mentioning a huge performance issue with using a failover cluster, so I've been doing some random research without much to show for it.
Few things about the configuration that might be relevant:

The cluster currently has two nodes. Both are running Server 2016 and both have two Nic Teams (configured in Windows, Switch Independent) consisting of two 1Gbit connections each.
The actual storage being used is a Synology that both machines are accessing via iSCSI, configured using these instructions.
Everything else seems to work fine, in the way that simulating a failover works and the other node takes over a few seconds later.

I'm guessing this is one of those "obvious to anybody who knows more than I do" sort of situations. Or Maybe I'm just hoping for that. Either way, I appreciate any guidance! I tried to keep it short, so please let me know if you need any other information.
Thanks in advance.
Answer: Your first issue is NICs teamed for iSCSI. You never do that unless your both target and initiator support multiple connections per session and in your case neither of them does.
https://www.starwindsoftware.com/blog/lacp-vs-mpio-on-windows-platform-which-one-is-better-in-terms-of-redundancy-and-speed-in-this-case-2
http://scst.sourceforge.net/mc_s.html
Solution: you have to un-team your NICs and use MPIO.
Your second issue is Synology itself. It's not what you use for primary storage, it's backup unit at best.
Solution: you copy your content to local disks and use Synology as backup repository or whatever.
Comment: I didn't say you need just ONE path, I told you need to get rid of LACP because it doesn't work with your target and initiator AS IS. I gave links you could read and understand WHY.
Comment: "Allow multiple sessions from one or more iSCSI Initiators:" has no relationship with your current issue, it's something target should do to make distributed locks being essential for Hyper-V, VMware etc functional.
Comment: I know that the Synology, at least, supports "Allow multiple sessions from one or more iSCSI Initiators:". There's a checkbox and everything. Really beside the point, though. Thank you for your suggestion. And I'm just working with the resources that I have. Can you elaborate a bit on why my current configuration would function just fine (performance wise) when being accessed via one path vs. the other?
Comment: Sorry, I should have clarified. I meant one file path (\\store01\share01 vs.\\srv04\e$\Shares\Share01). Either way, I'll break up the nic teaming and try mpio and see if there's any improvement. Thank you.
Answer: After removing the NIC teaming and placing the connections on the Synology/Server on a different subnet for good measure, I still saw no performance improvement.
I did finally come across the solution, though. It turns out Continuous Availability being on (the default) for the share was the culprit. There is documentation that says it may cause a "slight" performance penalty (like here) due to bypassing the write cache, it seems in some cases that "slight" performance penalty is actually "gigantic." Here's an article that goes into a pretty helpful background about Continuous Availability and when you might want to use it (to summarize, you might want to turn it off if your share is configured for "General Use File Server" and you're concerned with performance).
So, long story short, I disabled Continuous Availability on the share being used by the cluster, restarted both servers for good measure, and the performance issue was resolved. While I'd prefer to have it on to guarantee data integrity during a failover event, those are going to be so few and far between in my environment that there's no question the performance penalty wasn't worth it.
Comment: CA is non-cached I/O by design so yes, that's normal. https://blogs.technet.microsoft.com/filecab/2016/03/15/offline-files-and-continuous-availability-the-monstrous-union-you-should-not-consecrate/ "You can enable CA on non-SOFS shares in a cluster, and you can use end-user applications like Word with them, but for a variety of reasons, it's not something we recommend. "


CodeIgniter and S3 putObject - getting the right path to file

Question: I am having trouble passing the correct URL of an image from CI to S3 servers.
My image is stored in the root folder under assets/image.jpg
$this->load->library('s3');
$this->s3->putObject('../../image.jpg', $bucket, 'testimage.jpg', 'public-read');

The bucket creates the file testimage.jpg however it is just a 20kb file that is corrupt.
Can someone let me know how to get the right path to the image?
I have already tried:FCPATH . 'asset/image.jpg' and it did not work.
NOTE: I am doing this on my localhost, if that matters.
Answer: I have found the answer to the question and hopefully it can help someone else some time:
$this->s3->putObject(S3::inputFile('asset/image.jpg'), $bucket, 'testimage.jpg', 'public-read');
It was required to put S3::inputFile('folder/file').
Comment: Thanks! I had the same issue, and you solved it for me :-)


Cannot execute binary file on CentOS amd64

Question: I am trying to run a simple program on CentOS.
The program:
using System;
namespace helloworld
{
    class Program
    {
         static void Main(string[] args)
         {
            Console.WriteLine("Hello World!");
            returnHello();
         }
         static string returnHello()
         {
            return "hwdy ha";
         }
     }
}

My publish settings are:

Configuration: Release|Any CPU
Target framework: netcoreapp2.1
Deployment mode: self contained
target runtime: linux-x64
target location: bin\Release\netcoreapp2.1\publish\

In a terminal I run

dotnet publish --self-contained -r linux-x64 -c release

Then take the folder \publish and copy it over to my CentOS machine.
using file helloworld, I see the file details:

ELF 64-bit LSB executable, x86-64,version 1(SYSV), dynamically linked
  (uses shared libs), for GNU/linux.2.6.32,
  BuildID[sha1]=0x22d087f79061d441f9f6d007bf336f380c1214db, stripped

using uname -r:

3.2.0-3-amd64

In CentOS, I go into the publish folder and run: bash helloworld. In response, I get

helloworld: cannot execute binary file

.
I am not sure what I am missing here, to me it seems the file is targeting the right OS but I cannot execute the file in CentOS. Any help is appreciated. Thanks
Comment: Did you try CPU ANY?
Comment: yes, sorry I missed writing that.
Comment: What version of CentOS is this? 6?
Comment: @OmairMajid, when I run "uname -or" I get 3.2.0-3-amd64 GNU/Linux. I am not entirely certain if it is 6 or not.
Comment: Try `cat /etc/*-release`.
Comment: @OmairMajid results: PRETTY_NAME="Debian GNU/Linux wheezy/sid"
NAME="Debian GNU/Linux"
ID=debian
ANSI_COLOR="1;31"
Answer: TLDR: You need to switch to a more recent os for .NET Core, like CentOS 7 or Debian 9.
First, based on the /etc/*release output, you are using Debian wheezy (aka Debian 7). A couple of things to note: This was released in 2013 and End-of-Life'd in 2016. In other words, you are using an operating system that's not supported anymore, by anyone. It's not supported by .NET Core. It's not supported by Debian itself. (It's certainly not supported by CentOS)
Second, every release of .NET Core has a list of platforms (and specific versions of platforms) it supports. Older versions of those platforms lack features, compatibility, or something else that makes .NET Core unsuitable for running there.
.NET Core 2.1 supports these operating systems. For Debian, it only supports Debian 9 (and, I assume, later versions as they are released).
Since .NET Core supports Debian 9, it makes use of a number of features in the C library (glibc) that require a recent version of glibc. Debian 7 doesn't have that recent version of glibc, so it can't run .NET Core 2.1 applications.
You should try a more recent version of Debian, CentOS or some other Linux distribution. I know that .NET Core 2.1 works well on RHEL (and CentOS by extension). It should work well on recent versions of Debian and Ubuntu too.
There are a couple of ways to modify .NET Core and to work around this error, but it's rather advanced and would leave you with unable to update to new point releases of .NET Core, so I don't reccommend it: you can bootstrap .NET Core itself (as in, re-compile .NET Core itself) to support Debian 7 and then use that to run your application.
Comment: Hello, I wish that I could update that machine, however unfortunately this suggestion is not possible for me. I do appreciate the recommendation though. The company I work for is concerned that updating that machine will cause other issues with legacy software and refuses to allow this. I found that I could build Go programs successfully for this machine though, so I do have a solution going forward. Again, thank you!
Answer: Don't run bash on a binary executable, just run it directly with ./helloworld.
bash is a shell-script interpreter and refuses to read non-text files.  e.g.
$ bash /bin/ls
/bin/ls: /bin/ls: cannot execute binary file

If ./helloworld doesn't work, also try strace ./helloworld and ldd ./helloworld to check for a bad ELF interpreter (dynamic linker) path embedded in the executable.
Comment: Thank you. I tried ./helloworld and got Permission denied, I then tried the "ldd ./helloworld" and got "/usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by ./helloworld)". for the "strace ./helloworld", I got strace: command not found.
Comment: I looked at the above comment, and then ran "chmod +x helloworld", I think I was missing the executable permission. Now when I run ./helloworld, I only get "/usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by ./helloworld) "
Comment: I had made the same hello world program in go (named: helloworldgo) and built it for linux amd64. I wanted see which would build for linux easier. Running the go version with ./helloworldgo ran it just fine. @Peter thanks for pointing that out.
Comment: @Patrick: yup, "permission denied" is definitely just from missing `+x` permission.  `strace` and `ltrace` can be useful tools for debugging low-level problems like this (to see what actual system calls processes are making, e.g. what paths they're looking at.  `strace -e file ./hello` is often useful).  I'd recommend installing `strace`; all mainstream Linux distros package it.
Comment: Your library version error shows a C++ library ABI version mismatch between the build host and where you're trying to run it.  Unlike `libc`, `libstdc++` does have change in incompatible ways without incrementing the `.so.6` to a different filename, so they do symbol-versioning stuff to give dynamic-linker errors instead of hard-to-debug mysterious crashes.
Comment: I see, unfortunately I cannot change anything on the target machine, so I will be going with golang for this task since it didn't encounter the issue. Thanks for your help.
Comment: @Patrick: You might be able to get your build machine to compile+link against the appropriate libstdc++ version, instead of whatever it uses.


Variable number inputs to CNN

Question: In the image recognition problem I am trying to solve, we are supposed to develop a model that predicts the cost of a house based on images of several rooms in the house.  Each house has several images but not all houses have the same number of photos. Some houses have more photos, some have less. How can one setup a CNN to use such data?
Unfortunately, I cannot share any data because this is related to a problem I am solving at work.
Can someone point me towards sample code/problems/kaggle competitions that deal with this problem?
Answer: Here's some things to try out:

Run each photo through a convnet to extract features, concat them and run the concatenated feature vector through a fully connected network to get the price. To deal with varying number of inputs, you could decide for a maximum amount of inputs per house, and use zero-filled placeholder for inputs for houses with less inputs than maximum. Given enough data, and enough variation in data, the model should learn to ignore placeholders by treating them as noise.
Have one model to predict the house price from an image. Predict house prices for multiple images separately, and average predictions afterwards.
Comment: Thank you! I will try out that solution. I will leave this question open for a few 6 days to see if there are other people who answer before marking your answer as the solution.  (Also, it appears that my upvote doesn't count because my reputation here is less than 15)
Comment: Sure thing! Your problem seems interesting, let me know how it goes :). Working with neural nets is still relatively research level work - do expect this to take time.
Comment: I just want to point out that, in the first suggestion, you would need to decide what type of photo occupies which location in the concatenate feature. E.g., the order should bedroom 1, bedroom 2, kitchen, etc. If a house does not have bedroom 2, that region should be all zeros. This would make the whole learning stable. For the same reason, the concatenated features would have very large dimensionality since the type of photos could vary so much. As a first step, I recommend simply averaging the features from each photo. Later on, you can try using something like attention mechanism.

Variable number inputs to CNN


Union of arbitrary path connected sets is necessarily a path connected?

Question: Assume $X$ is a topological space that does not have a countable dense set. Let $\{ A_\alpha \}$ be an arbitrary set of path connected sets such that if $\alpha$, $\beta$ are two different indices then $A_\alpha$ is different from $A_\beta$ and that for every $\alpha$ there exists $\beta$ such that $A_\alpha $ intersects $A_\beta$.Then does it necessarily follow that the arbitrary union of these $A_\alpha$'s is path connected? If it is not do please provide a counterexample and also a hint as to how to strengthen the conditions so that the result holds ,without removing the condition that the $X$ does not have any countable dense set if possible of course.
Comment: Your assumptions are way too weak. One way for that to be true is when $\bigcap A_\alpha\neq\emptyset$.
Comment: @freakish What about intuitively forming a chain of them "connected" to each other will that still be a weak condition although i am still not sure how to formalize iit
Comment: Such condition makes sense when $\{A_\alpha\}_{\alpha=0}^\infty$ is countable and $A_\alpha\cap A_{\alpha+1}\neq\emptyset$. But even then it is not clear to me why $\bigcup A_\alpha$ has to be path connected, although I don't have a counterexample. Regardless, this will fail for arbitrary subcollection, even when $\{A_\alpha\}$ is finite, e.g. $[0,1]$, $[1,2]$, $[2,3]$ but $[0,1]\cup [2,3]$ is disconnected.
Answer: Take four closed path-connected sets $A_1$, $A_2$, $A_3$, $A_4$ such that $A_1$ intersects $A_2$ and $A_3$ intersects $A_4$, but $A_1 \cup A_2$ is disjoint from $A_3 \cup A_4$. These satisfy your conditions, but  $A_1 \cup A_2 \cup A_3 \cup A_4$  is not connected.
Comment: Thanks for the answer, could you also comment on whether intuitively chaining them together subsequently after each other would somehow remedy this?


If measure of a every set is $0$ then measure of the union of those sets is $0$?

Question: I want to prove the following property:
If $\mu(E_j) = 0$ for all $j \geq 1$, then $\mu(\cup_{j\geq1} E_j) = 0$.
In order to prove this can I just state that if $E_j$ has measure $0$ then it follows that it must be the empty set $\emptyset$ and therefore the union of empty sets is the empty set itself so it follows that $\mu(\cup_{j\geq1} E_j)  = \mu(\emptyset) = 0$.
Please tell me if my reasoning is correct and if not please tell what I need to do in order to prove this propertie.
Thanks in advance!
Answer: A set of measure zero is generally not the empty set. For example, the 1-dimensional Lebesgue measure of the non-empty set $\{1\}$ is zero.
Actually, here we can use the sub-additivity of measure.
$$\mu\left(\cup_j E_j\right)\leq \sum_j\mu(E_j)=0$$
so $\mu\left(\cup_j E_j\right)=0$.


Do ports change for every request? Are Ip ports are different from system ports?

Question: A typical ip address has >65k ports then again a system that runs localhost offline has port which sometimes face conflicts. Am very under-read in this area please give some light. Pardon my English.
Thanks
Answer:
A typical ip address has >65k

No. The IP protocol understands 65536 possible ports (it is a 2 byte identigier) which is exactly 64x1024, and computers are base 2.
Also, a connection is not determined by the client side port, but by BOTH ports, giving it a lot more than 65536 possibly connections. In fact, theoretically (the CLIENT side port is normally range limited) you can have X (65536, more like 5000) connections PER CLIENT to one server because the IP Addresses are ALSO part of the unique combo.
Now, to your questions:

Do ports change for every request?

OBVIOUSLY not - they change for every CONNECTION. HTTP i.e. can keep a connection open and send multiple requests over one, and HTTP 2.0 can even send multiple requests at similar time (overlapping) over one connection.

Are IP Ports different from system ports?

No. System ports are speciic IP ports (below 1024 actually) that only the system utilizes and the user is not allowed to use.


Patch a contrib module with a patch that contains new files

Question: How do I patch the recaptcha module with the invisible recaptcha patch using drush make? I patch modules all the time in project.make.yml with yml that looks just like this:
<code>...
# Recaptcha 8.x-2.2 - 2016-07-30
recaptcha:
  subdir: contrib
  version: '2.2'
  patch:
    # Integrate Invisible reCAPTCHA option from Google
    - 'https://www.drupal.org/files/issues/recaptcha-invisible-option-2852269-56.patch'
...
</code>
What's different with the invisible recaptcha patch is the patch contains two new files.
Answer: I would recommend you consider switching to composer and composer-patches. I don't think drush make is still actively maintained?
Quick google search pointed me to http://www.anexusit.com/blog/how-to-apply-patches-drupal-8-composer for example. That works fine, also for patches that add new files. The only case where that is a problem is core, due to different path structures in drupal/core and then it doesn't apply it correctly.
Comment: We use composer as well even though our composer.json is not stable because we use grunt-drupal-tasks for our build.

I changed the `composer require` to:

`composer require drupal/captcha drupal/recaptcha:2.x-dev`

and the patch was applied successfully during the build process.

But now I'm on the dev version of the module.


C fopen vs open

Question: Is there any reason (other than syntactic ones) that you'd want to use
FILE *fdopen(int fd, const char *mode);

or
FILE *fopen(const char *path, const char *mode);

instead of
int open(const char *pathname, int flags, mode_t mode);

when using C in a Linux environment?
Comment: `fopen` is part of standard C library, `open` is not. Use `fopen` when writing portable code.
Comment: @Aziz, `open` is a POSIX function though.
Comment: If you have a file descriptor created by `pipe()` — or perhaps `socket()`, though that's less likely — and you want to use standard `FILE *` I/O on it, then you need to use `fdopen()` to create the stream.  There are also numerous options that can be specified with `open()` that cannot be specified with `fopen()` — perhaps most prominently `O_EXCL`, but there are certainly others too — and those might be good reasons for using `open()` (instead of `fopen()`) followed by `fdopen()`.
Answer: Of those, fdopen is not like the others. fdopen is what you would use if you first called open and then wanted a FILE *. There is no sense doing that if you have the choice to just call fopen instead. There are cases where you won't be able to get fopen to do exactly what you want, but they're outside the scope of this question.
So, let's just pretend fdopen isn't even there and get on with the question.
There are four main reasons to use fopen instead of open.

fopen provides you with buffering IO that may turn out to be a lot faster than what you're doing with open.
fopen does line ending translation if the file is not opened in binary mode, which can be very helpful if your program is ever ported to a non-Unix environment (though the world appears to be converging on LF-only (except IETF text-based networking protocols like SMTP and HTTP and such)).
A FILE * gives you the ability to use fscanf and other stdio functions.
Your code may someday need to be ported to some other platform that only supports ANSI C and does not support the open function.

In my opinion (and experience) the line ending translation more often gets in your way than helps you, and the parsing of fscanf is so weak that you inevitably end up tossing it out in favor of something more useful.
And most platforms that support C have an open function.
That leaves the buffering question.  In places where you are mainly reading or writing a file sequentially, the buffering support is really helpful and a big speed improvement.  But it can lead to some interesting problems in which data does not end up in the file when you expect it to be there.  You have to remember to fclose or fflush at the appropriate times.
If you're doing seeks (aka fsetpos or fseek the second of which is slightly trickier to use in a standards compliant way), the usefulness of buffering quickly goes down.
Of course, my bias is that I tend to work with sockets a whole lot, and there the fact that you really want to be doing non-blocking IO (which FILE * totally fails to support in any reasonable way) with no buffering at all and often have complex parsing requirements really color my perceptions.
Comment: To clarify when buffering gets in the way. It's when you use seek. The following read with whatever command (`fgets`, `fgetc`, `fscanf`, `fread`) , will always read the whole size of the buffer (4K, 8K or whatever you set). By using the direct I/O you can avoid that. In that case it's even better to use `pread` instead of a seek/read pair (1 syscall instead of 2).
Comment: @m-ric: Well, that's a somewhat unrelated question, but yes. All platforms that support `ioctl` also support the `fileno` call which takes a `FILE *` and returns a number that can be used in an `ioctl` call. Be careful though. `FILE *` related calls may interact surprisingly with using `ioctl` to change something about the underlying file descriptor.
Comment: Handling interrupted `read()` and `write()` calls is a convenient fifth reason to use the libc family of functions.
Comment: I own a map rendering library (CartoType) which does its own buffering. I experimented with moving from high-level to low-level io (from fopen to open, etc.) and benchmarked the performance before and after. Low-level io was better, but only by 3%. I have not yet decided whether a 3% improvement is worth it. Probably yes, because I can encapsulate the platform-dependent code.
Comment: Almost any program that does not stream IO, where you end up doing seeks into the middle of the file.  I should update my response to clarify.
Comment: So would it be correct to conclude that 'fgets' would block the thread until it reads a full line.  And that non-buffered IO (such as read) would inherently not block the thread because they have no buffer to hold the data?
Comment: @L. Moser: No.  The blocking and buffering are not strongly coupled.  You would have to set the file descriptor to `O_NONBLOCK` with `fcntl` and and the `EAGAIN` error when `read` failed in order to avoid blocking a thread with `read`.
Comment: @LJM - To clarify further on that somewhat muddled answer from 2 years ago... :-) Yes, fgets will do exactly that. And there is no reasonable way for it not to. But just using `read` does not magically get you non-blocking IO. You have to set the O_NONBLOCK flag on the descriptor somehow, and then properly handle the EAGAIN condition/error on `read`. The latter, in particular, is non-trivial since you frequently need to remember some sort of parser state until there is data available to `read`. Some people think you should just give up and use cheap threads that block.
Comment: What about opening a device file and send an ioctl? Can I use `fopen` which returns `FILE*` where ioctl() expects and `int fd`?
Comment: The second reason is rather doubtful because:
[For portability, it is strongly recommended that you always use the 'b' flag when opening files with fopen().](http://php.net/manual/en/function.fopen.php)

I strongly believe that the same code should produce binary equal files regardless of the operation system on which it is executed. Otherwise this code is not truly portable.
And I do not see any problems when using Unix line endings everywhere because all significant applications on Windows (even Notepad [since the last year](http://archive.li/Dskp0)) are support them.
Comment: @tav - I happen to know that there is some pressure from the top at Microsoft to abandon CRLF line endings in Microsoft products in general. So, I expect them to slowly fade out.
Answer: open() is a low-level os call. fdopen() converts an os-level file descriptor to the higher-level FILE-abstraction of the C language. fopen() calls open() in the background and gives you a FILE-pointer directly.
There are several advantages to using FILE-objects rather raw file descriptors, which includes greater ease of usage but also other technical advantages such as built-in buffering. Especially the buffering generally results in a sizeable performance advantage.
Comment: actually there are other disadvantages. `fopen()` does not provide the same level of control when opening files, for example create permissions, sharing modes, and more. typically `open()` and variants provide much more control, close to what the operating system actually provides
Comment: @L. Moser, yes, when you are already buffering the data, and hence the additional buffer adds unnecessary copying and memory overhead.
Comment: Are there any disadvantages to using the buffered 'f...' versions of open?
Comment: There are also the extreme cases where you `mmap` the file and do changes with normal I/O (as incredible as it sounds we do actually that in our project and for real good reasons), buffering would be in the way.
Comment: You also may want to use other system functions, like using open() for preloading files into page cache via readahead().
I guess rule of thumb is "use fopen unless you absolutely need open()", open() actually lets you do fancy stuff (setting/not setting O_ATIME and similar).
Answer: fopen vs open in C

fopen is a library function while open is a system call.

fopen provides buffered IO which may be faster compared to open which is non-buffered.

fopen is portable while open not portable (open is environment specific).

fopen returns a pointer to a FILE structure (FILE *) while open returns an integer that identifies the file.

A FILE * gives you the ability to use fscanf and other stdio.h functions.
Comment: `open` is a POSIX standard, so it's quite portable
Comment: *`open` is a **system call**.*  Ugh.  I don't know why this meaningless artificial demarcation exists.  `open()` on just about every extant POSIX system is a **library function** exactly like `fopen()` is.  The `open()` **library function** is usually implemented around the `openat` **system call**.  And even then, before there was an `openat()` system call, `open()` was a **library function** wrapped around the `open()` system call anyway.  `fopen()` is just a bigger library function wrapper around the same system call that the `open()` library function uses.
Comment: `fopen()` is not [async-signal-safe](https://man7.org/linux/man-pages/man7/signal-safety.7.html) (none of the buffered io is), therefore it should not be used in any signal handler - or a function that may be called by it and be taken with care if used in functions that use asynchronous threads. Using `open()` adds complexity and misses the buffered-io, ...
Comment: @osvein Not really portable anymore.  Works fine with Linuxes, but is now deprecated in Windows.  You are supposed to use _open, _close now.  Sort of a minor point.  I guess the headers needed on Linux and Windows are also different.  I think it would be hard to write portable code without some "#if OS_LINUX" lines in it.
Comment: @Jiminion *Works fine with Linuxes, but is now deprecated in Windows.*  [Microsoft also quite risibly "deprecates" `fopen()`](https://stackoverflow.com/questions/14386/fopen-deprecated-warning).  [And `strcpy()`](https://stackoverflow.com/questions/4012222/c-strcpy-gives-a-warning-c4996).  Along with probably half the required functions in C.  *You are supposed to use _open, _close now.*  No, you're not "supposed to".  "Microsoft wants you to because it's not portable" [is probably closer to the mark](https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish#Examples_by_Microsoft).
Comment: @osvein - Andrew is right. And the distinction is important to remember. The open libc function is a function that wraps a system call. On newer versions of libc it actually wraps the `openat` system call.
Answer: Unless you're part of the 0.1% of applications where using open is an actual performance benefit, there really is no good reason not to use fopen. As far as fdopen is concerned, if you aren't playing with file descriptors, you don't need that call.
Stick with fopen and its family of methods (fwrite, fread, fprintf, et al) and you'll be very satisfied. Just as importantly, other programmers will be satisfied with your code.
Comment: I still use `open` because I dislike the `const char *mode` instead of `int mode`
Answer: If you have a FILE *, you can use functions like fscanf, fprintf and fgets etc. If you have just the file descriptor, you have limited (but likely faster) input and output routines read, write etc.
Answer: open() is a system call and specific to Unix-based systems and it returns a file descriptor. You can write to a file descriptor using write() which is another system call.
fopen() is an ANSI C function call which returns a file pointer and it is portable to other OSes. We can write to a file pointer using fprintf.
In Unix:
You can get a file pointer from the file descriptor using:
fP = fdopen(fD, "a");

You can get a file descriptor from the file pointer using:
fD = fileno (fP);

Answer: Using open, read, write means you have to worry about signal interaptions.
If the call was interrupted by a  signal  handler the functions will return -1
and set errno to EINTR.
So the proper way to close a file would be
while (retval = close(fd), retval == -1 && ernno == EINTR) ;

Comment: For `close`, this depends on the operating system. It's incorrect to do the loop on Linux, AIX and some other operating systems.
Comment: In addition, using read and write will suffer the same issue, i.e., they can get interrupted by a signal before processing the input/output entirely and the programmer has to handle such situations, while fread and fwrite handle signal interruptions nicely.
Comment: I've seen this way too many times, but never a good reason why this would ever be issue. Maybe you could provide an example where it would actually be a necessity to not use `SA_RESTART` in your signal handler and instead have syscalls return `EINTR`?
Answer: I changed to open() from fopen() for my application, because fopen was causing double reads every time I ran fopen fgetc .  Double reads were disruptive of what I was trying to accomplish.  open() just seems to do what you ask of it.
Answer: open() will be called at the end of each of the fopen() family functions. open() is a system call and fopen() are provided by libraries as a wrapper functions for user easy of use
Answer: Depends also on what flags are required to open. With respect to usage for writing and reading (and portability) f* should be used, as argued above.
But if basically want to specify more than standard flags (like rw and append flags), you will have to use a platform specific API (like POSIX open) or a library that abstracts these details. The C-standard does not have any such flags.
For example you might want to open a file, only if it exits. If you don't specify the create flag the file must exist. If you add exclusive to create, it will only create the file if it does not exist. There are many more.
For example on Linux systems there is a LED interface exposed through sysfs. It exposes the brightness of the led through  a file. Writing or reading a number as a string ranging from 0-255. Of course you don't want to create that file and only write to it if it exists. The cool thing now: Use fdopen to read/write this file using the standard calls.
Answer: I think this question has been answered adequately already but its an interesting question so let me throw in my 2 cents. I think open() sits closer to the OS kernel and is lower level than the fopen() function. The open() function is part of the POSIX standard and could be preferable in a few situations. If you are developing Linux drivers or want to do non-blocking IO on POSIX compliant systems, you'll want to use open() instead of fopen(). However, if all you want is cross platform file I/O which works on both Linux and Windows, fopen() would be a better choice.
Answer: Opening a file using fopen
Before we can read(or write) information from (to) a file on disk, we must open the file. To open the file, we call the function fopen().

fopen() searches the disk for the file to be opened.
Then, it loads the file from disk into a place in memory called the buffer.
Finally, it sets up a character (char) pointer that points to the first character of the buffer.

Sometimes, during the buffering process, fopen() may time out.
So, when comparing the fopen() library call (high-level i/o) to the open() (low-level i/o) system call, it is faster and more appropriate to use open() rather than fopen().
Comment: is open faster than fopen?
Comment: yes ,**open** is the system call, which is faster than fopen -  comparitively @obayhan

C fopen vs open


Data is not shown in SqlLite in Windows phone

Question: I am developing the app that storing data in the Database in Windows phone8 using SqlLite. The data is inserted successfully. But when I go and saw from Mozila->SQL Lite Manager, the data is not shown. Is it possible to read the data?
Comment: are you sure that while inserting date you are not getting any exception? please let me know, may be I can help you.
Comment: No,there is no exception.
Answer: It is not yet possible to read data from Mozila->SQLite Manager in WP8. If you want to see what data is stored in your database, execute a query in app. For more information follow Here


XUnit - Testando Theory com MemberData Passando uma lista de classes

Question: Estou utilizando um projeto Xunit para fazer testes na minha API.
Como é minha primeira vez fazendo testes, estou pesquisando como fazer as coisas
Mas estou com um problema que até então eu não encontrei uma solução, que seria na hora de passar instancias de objetos para um metodo de teste
Atualmente eu utlizo um IEnumerable para instanciar minha classe e atribuir os valores manualmente
Eu utilizei de duas maneiras que encontrei na internet para testar meus objetos criados
Exemplo 1

Exemplo 2

Dessa forma ele acusa erro:
System.ArgumentException : Object of type 'System.Collections.Generic.List`1[ServicoBanco.Domain.Commands.ClienteCommand]' cannot be converted to type 'ServicoBanco.Domain.Commands.ClienteCommand'.

[Theory(DisplayName = "Tratar Valores Recebidos")]
        [Trait("TesteUnidade", "Command")]
        [MemberData(nameof(CommandListTeste))]
        public void TestaCommand_TratarValoresRecebidos_RetornaVerdadeiro(ClienteCommand command)
        {
            Assert.True(true);
        }

Eu quero passar uma lista para a função de teste e que ela receba cada objeto individualmente
Exemplo:
List objetos
MeuObjeto obj1 = new {valores}
MeuObjeto obj2 = new {valores}
MeuObjeto obj3 = new {valores}
// Atribuo meus objetos nessa lista
//Chamo a função de testes
// objetos que é passado por parametro é a lista dos objetos que passei
[Theory]
[MemberData(nameof(objetos))]
public void TestaCommand(MeuObjeto meuObjeto)
{
// aqui dentro cada "meuObjeto" é uma instancia dos objetos que eu criei a atribui os valores
}
Tem como fazer isso ?
Comment: o problema é que está criando os dados de teste dentro de uma lista, e no método de teste espera um objeto, mude para `new object[] { new ClientCommand(), new ClientComand().... }`
Comment: Poderia explicar como fica ?
Comment: exatamente como coloquei no comentário anterior, no seu primeiro exemplo no lugar de `new List..` criar somente cada objeto `ClientCommand`
Answer: Depois do Cliente Command eu abro chaves e preencho as propriedades manualmente
{
   Id = 1,
   Nome = "Nome",
   // etc
}


Glyphicons do bootstrap não estão funcionando

Question: Alguém poderia me dizer por que glyphicons do bootstrap não estão funcionando?
Ele fica assim

Ao invés de ficar assim...

Código:
 Novidades
Todos estão desse jeito.
Comment: Você alterou o diretório `fonts` ou `css`?
Comment: possível duplicata de [Fonte personalizada no HTML dentro do IE10?](http://pt.stackoverflow.com/questions/51022/fonte-personalizada-no-html-dentro-do-ie10)
Comment: @EduardoSilva

http://i.imgur.com/TlTHY5Z.png

http://i.imgur.com/rMvkxRa.png
Comment: @GuilhermeNascimento Isso não tem nada á ver com minha pergunta.
Comment: Parece que não Leonardo, mas as glyphicons são fontes também, por isto a mensagem do meu comentário veio com *POSSIVEL*, pode ser uma duplicata como também pode não ser. Poderia fornecer um exemplo completo de como inseriu os CSSs e fontes? Talvez um link online?
Comment: Qual navegador ocorre? Todos? Firefox, Chrome, IE11?
Comment: @GuilhermeNascimento Só testei no chrome

http://i.imgur.com/TlTHY5Z.png

http://i.imgur.com/rMvkxRa.png
Comment: Então testa nos demais navegadores. **Nota:** imagens raramente ajudam a detectar o problema, forneça um exemplo "funcional", obrigado.
Comment: No internet explorer se quer aparece aquele icone quadrado.

Isso não acontece apenas com os glyphicons, mas também com os fa icons.
Comment: Após a resposta, posso presumir que o motivo do fechamento seria: *Fora do escopo > Esse problema não pode **ser reproduzido**, ou é um **erro de digitação**.*
Comment: @GuilhermeNascimento Ok, obrigado pela ajuda.
Answer: Resolvi, faltava os arquivos das fontes.


Insert statement won't work, getting syntax near "?":

Question: For some reason I'm getting the error but I don't see why. I've looked up some tutorials and nothing works.
cursor.execute('''INSERT INTO Users (?, ?, ?, ?)''', (UUN, UUID1, UUID2, UUID3))

The 4 variable names are UNN, UUID1, UUID2, UUID3.
Comment: `INSERT INTO Users() VALUES (?, ?, ?, ?)`
Comment: You didn't specify `VALUES` in your query string
Comment: Follow the correct syntax: https://www.sqlitetutorial.net/sqlite-insert/
Answer: You are missing the column names from your query. Assuming they're the same as those variable names,
cursor.execute(
    'INSERT INTO Users (uun, uuid1, uuid2, uuid3) VALUES (?, ?, ?, ?)',
    (UUN, UUID1, UUID2, UUID3),
)


Error: $injector:modulerr Module Error

Question: Em um projeto que estou desenvolvendo, estou incluindo os arquivos:
app.js
angular.module('app', [
    'app.controllers',
    'app.services'
]

controller.js
angular.controller('HomeController', function($scope, HomeService)
{
    // code
})

services.js
angular.service('HomeService', function()
{
    // code
})

template.html

        // scripts

        // body

Quando abro a página me deparo com o erro:

Pesquisei no Google e nas poucas pesquisas que achei onde se tratava do mesmo erro que estou tendo, não consegui uma solução para o mesmo. Alguns falam que está faltando algum módulo ou coisa do tipo, mas como mostrei, está tudo inserido e os nomes estão corretos.
O que posso estar fazendo de errado ?
Answer: Se seu código está exatamente como o descrito, você, em nenhum momento, declarou estes módulos 'app.controllers' e 'app.services' causando assim o problema na injeção.
Seguindo a linha do seu código, deveria ser mais ou menos assim:
app.js
angular.module('app', [
    'app.controllers',
     'app.services'
]);

controller.js
angular.module('app.controllers')
      .controller('HomeCtrl', ['$scope', function($scope) {
        // código aqui
      }]);

service.js
angular.module('app.services')
   .service('HomeService', [function] {
   // código aqui
   });

Desta forma, os módulos que você injetou como dependência estão declarados. Você também poderia fazer da seguinte forma:
controller.js
var appControllers = angular.module('app.controllers');
appControllers.controller('HomeCtrl', [function () {
  // código aqui
}]);

Caso você queira adicionar mais controladores / services (utilizam do mesmo principio para declaração):
controller.js
angular.module('app.controllers')
      .controller('HomeCtrl', ['$scope', function($scope) {
        // código aqui
      }])
      .controller('OutroCtrl', [function () {
       // mais código aqui
      }])
;

ou desta forma:
var appControllers = angular.module('app.controllers');

appControllers.controller('HomerCtrl', [function () {
 //código aqui
}]);

appControllers.controller('OutroCtrl', [function () {
 // mais código aqui
}]);

Espero que isto resolva!
Comment: Complementando a resposta: Isso é possivel pois `angular.module` retorna o objeto module onde será aplicado controllers e services
Comment: MAS, você não vai conseguir acessar `app.controllers` se não os tiver registrado antes, vai dar o mesmo erro
Comment: Realmente resolveu o meu problema. Obrigado!
Answer: O conceito da resposta do @Marco Antônio está correto, mas contém alguns erros de compilação, por isso vou por aqui uma forma mais simples de fazer:
Ah, e outra coisa, não precisa anexar os modulos app.controllers e app.services se você não está criando eles, mas registrando controllers e services direto no modulo principal.
Just Copy'n' Paste
var app = angular.module('app',[]);

app.controller('HomeController',function($scope){
    //codigo aqui
});

app.service('HomeService', function(){
    //codigo aqui
});

Error: $injector:modulerr Module Error