Methods of initializing K-means clustering

Question: I am interested in the current state of the art for selecting initial seeds (cluster centers) for K-means.
Googling leads to two popular choices:

random selection of initial seeds, and,
using the KMeans++ selection technique: Arthur & Vassilvitskii 2006 k-means++: The Advantages of Careful Seeding

Are there any other promising methods that anyone here is aware of, which might not be as popular?
Answer: Allow me, without going far, simply to copy-paste a list of options from my own function !kmini (a macro for SPSS), found in collection "Clustering" here.
Method to create or select initial cluster centres. Choose:

RGC  - centroids of random subsamples. The data are partitioned
randomly by k nonoverlapping, by membership, groups, and centroids of
these groups are appointed to be the initial centres. Thus, centres
are calculated, not selected from the existent dataset cases. This
method yields centres that lie close to each other and to the general
centroid of the data.
RP   - randomly selected points. k distinct cases
of the data are randomly selected to be the initial centres.
RUNFP    -
farthest points (running selection). First k cases are taken as
centres and then during the run through the rest of the cases of the
dataset there progressively replacements among the centres are done;
the aim of the replacements is to obtain in the end k points most
distant from each other in the variable space. These points (cases)
occupying peripheral positions in the data cloud are the produced
initial centres. (The method is used as the default in SPSS k-means procedure QUICK CLUSTER. See details in SPSS Algorithms. See also described here).
SIMFP    - farthest
points (simple selection). The first centre is selected as a random
case from the dataset. The 2nd centre is selected as the case
maximally distant from that centre. The 3rd centre is selected as the
case maximally distant from those two (from the nearest of the two), - and so on.
KMPP - random farthest points, or k-means++. The first centre is selected as a random case from the dataset. The 2nd centre
is selected also randomly, but the probability of selection of a case
is proportional to the distance (square euclidean) of it to that
(1st) centre. The 3rd centre is selected also randomly with the
probability of selection proportional to the distance of a case to
the nearest of those two centres, - and so on. (Arthur, D.,
Vassilvitskii, S.. K-means++: the advantages of careful seeding. //
Proceedings of the 18th annual ACM-SIAM symposium on Discrete
algorithms. 2007., 1027–1035.)
GREP - group representative points.
The method idea – to collect as centres k most representative,
"deputy" cases. The 1st centre is taken as the case closest to the
general data cenroid. Then the rest of the centres are selected from
the data points in such a way that each point is considered as to
whether it is closer (and how much, in terms of squared euclidean
distance) to a set of points than each one of the latter is to any of
the already existing centres. I.e. each point is examed as a
candidate to represent some group of points not yet well enough
represented by the centres already collected. Point most
representative in this respect is selected as the next centre.
(Kaufman, L. Rousseeuw, P.J. Finding groups in data: an introduction
to cluster analysis., 1990. See also: Pena, J.M. et al. An empirical
comparison of four initialization methods for the K-means algorithm
// Pattern Recognition Lett. 20 (10), 1999, 1027-1040.)
[There is also a nice method, not yet implemented by me in the macro, to generate k points which are from random uniform but "less random than random", somewhere between random and greed; see potential theoretical basis for that method]
One more method is to do hierarchical clustering by Ward's method. You may do it on subsample of objects if the sample is too big. Then means of the k clusters produced by it are the initial seeds for k-means procedure. Ward's is preferable over other hierarchical clustering methods because it shares the common target objective with k-means.

Methods RGC, RP, SIMFP, KMPP depend on random numbers and may change their result from run to run.
Method RUNFP may be sensitive to case order in the dataset; but method GREP is not (apart from occasions when there are many identical cases, ties, in the data). Method GREP may fail to collect all k centres if k is large relative the number of cases in the data (n), especially when k>n/2. [The macro will inform if the data do not allow that method to collect k centres]. Method GREP is the slowest one, it computes [in my implementation] matrix of distances between all cases, therefore it won't suit if there are many tens of thousands or millions of cases. You could, however, do it on a random subsample of the data.
I'm not discussing presently which method is "better" and in what circumstance, because I haven't done extensive simulational probing of the question so far. My very preliminary and superficial impressions have been that GREP is particularly worthy (but it is expensive), and that if you want really cheap method still competitive enough, then just random k points, RP, is a decent choice.
Comment: p.s. see also answer https://stats.stackexchange.com/a/350191/3277
Comment: I posted something - https://stats.stackexchange.com/questions/462465. I'd be happy for your input.
Comment: I'd be happy to see your answer to something like - Deterministic yet effective ways to initialize K-Means.
Comment: @Royi, if you have a question about it why not post the question?
Comment: Do you have many methods to share? I have created few "Find Farthest Samples" tricks, but are many good ones worth opening a question on?
Comment: If you have anything you regard as worthy, share it in a form of a question if something worthy can be asked about it by the question.
Answer: Last time I did a comprehensive literature review on this, which was admittedly almost 20 years ago, the two main recommendations were:

To use Ward's Method (this is a standard hierarchical cluster
analysis algorithm) to find initial centers.
Use random starts.

In big data applications, Ward's method doesn't work so well, although it can be applied to a sub-sample.
I did some simulations, which I didn't ever get around to publishing, and found that:

100 Random starts consistently outperformed Ward's Method.
The default SPSS algorithm for finding start points  outperformed
100 random startpoints.
At 1,000 random start points,
randomization consistently was the best method.

The main take out I took from this is that the SPSS algorithm is surprisingly good, but if one has the resources, 1000+ random start points is the way to go.
Comment: In your simulations did you notice any change in behaviour for high dimensional data?
Comment: Not that I can recall. However, my simulations wouldn't have used more than about 20 variables I think. However, the higher the dimensionality, the greater the number of random starts required all else being the same.
Comment: A note: The default SPSS algorithm (btw your link is broke) is what I acronamed as RUNFP in my answer.
Answer: With ttnphns nomenclature, I tested RGC, RP, and KMPP on:

2D/3D points
bag of words from textual documents
curves with essentially $L^2$ distance.

I don't recommend RGC because the resulting centers are very close to each other: the mean of many points is close to the global mean (law of large numbers). This can slow down convergence a lot: it takes some time before clusters start to individualize.
RP is generally good and would recommand as the first easy choice.
KMPP is very popular and works very well in small dimension: compared to RP it tends to reduce the probability of ending in a local minimum.
However when I was working on big datasets (1M points that are bag of words from textual documents with big dimension), RP slightly outperformed KMPP in the sense that it ended with slightly fewer iterations. I was surprised of this. In big dataset/high dimension, the convergence to the global minimum is impossible, you measure quality as "how good the local minimum is" = "how small final SOD is". Both methods had the same quality.
Note that it is important to use a randomized method if you want to use replications to improve the quality.
Comment: Thanks. I will be dealing with large dimension data so this is quite useful.
Answer: I assume from your question that you are interested in good solutions to k-means problems (otherwise any initialization method would do). You might want to have a look at the "Breathing K-Means" algorithm (I am the inventor). Instead of just being an initilization method for Lloyd's algorithm (a.k.a. the k-means algorithm) it adds and removes groups of centroids based on error and utility while running Lloyd's algorithm after each add or remove step. This algorithm beats k-means++ on 39 of 40 test problems, mainly taken from the literature. It is on par for the one test problem where k-means++ finds the optimum (and thus cannot be beaten). Software: https://pypi.org/project/bkmeans (API compatible with scikit-learn) Preprint: https://arxiv.org/abs/2006.15666.
Comment: Since the last step of "Breathing K-Means" is a run of Lloyd's algorithm, one can interpret all activities before that as a new kind of initialization for this final step. It may seem involved, but so are approaches using genetic algorithms or simulated annealing to find a suitable initial codebook. One can also interpret "Breathing K-Means" as an approach that overcomes the limits of the traditional strict separation of initialization and local optimization by mixing runs of Lloyd's algorithm with "smart" insertions and deletions.
Comment: If this a new algorithm to _initialize_ k centres? If not, it would be off-topic.


How do I make different sounds using modes when I'm using the same chords that I'm building the modes off?

Question: People keep saying use modes to get a different sound. How can you get a different sound from modes when they use the same chords from whichever you built the modes off, doesn't really matter if it's major or minor, that part is confusing?
Comment: Can you edit and clarify what kind of difference you're looking for? Is this about making a piece in the Lydian mode not sound like it's major, for example?
Comment: it's about the chords of the modes how can you add a different sound to your music using modes when the chords of the modes are the same as the major parent scale why use modes at all.
Comment: I'm still not clear on what you mean by "different sound" and your comment doesn't expand on that at all. But it does seem like you're asking how to write in a particular mode without what you've written sounding like it's in a different mode.
Comment: In what context do people tell you to "use modes to get a different sound"?  Is the difference you seek similar to the difference between major and minor keys?
Comment: I think what's making this an unclear question is that it's about unclear advice. "Sound" is about as vague  as you can get in music. But here's a question: do you perceive a difference between major and minor? A different "feel"? If so, that's your starting point.
Comment: you misunderstand me you build modes off parent scales i understand that, what i don't understand is how do you make different sounds using modes when the modes use the exact same chords as the parent scale but in a different order, i know when you play the modal scale, that has it's own sound but the chords come from the parent scale.
Comment: The problem word here is "sound." Whoever gave you this advice failed to clarify it. Chords are sounds you make, yes. And many modes share the same "chord-sounds." But I think whoever gave this advice meant something more along the line of "feel": an overall gestalt based on context. Also: changing the mode isn't the only way to get a different "feel," maybe not even the most impactful way. I wouldn't worry too much about the advice.
Comment: i know each mode scale has it's own unique sound quality C D E F G A B C ionian mode D E F G A B C D dorian mode i you where to sound the scale out starting on E and ending on E that would sound different to the other modes, but the chords you make from the modal scales come from the parent scale but there just in a different order my question is how do you get a different sound using modes when the chords from the modal scales come directly from the parent scale?
Comment: Does https://music.stackexchange.com/q/5382/78419 help? It does seem to me that you're thinking only of scales, without really understanding what a mode is.
Comment: i know what a mode is it's like a scale but you start on a different degree.
Comment: NO. A mode, for the purposes of this conversation, is *not* a scale. Major and minor share the same notes; the difference is the "tonality," aka "key," you're in. I.e. which note is "do" and which is "sol," the gravitational pull of tonal harmony toward a certain "home base." This is how multiple modes can share the same scale degrees and still be in any sense different entities. Please read the many pages that have been linked at this point.
Comment: Also, feel free to edit this question into something that is not a duplicate, and it can be reopened.
Answer: You don't "use" modes by selecting a scale and randomly playing notes and chords from it without having any sense, skill and control over what you are doing with your notes and chords. A modal scale does not guarantee that any random note chaos will create a modal feeling as long as the notes and chords are selected from the right scale. You need to know what picture you are drawing. Where is the center and what's in it.
A scale doesn't create a mode, it's the musician who creates a mode, often using a scale as a tool. A musician can use a scale and FAIL to create the intended mode in the listener's ear. It can happen if the musician emphasizes the wrong notes or doesn't emphasize the right notes for that mode. The same set of notes, say, the white keys of the piano, can be used to create many different harmonic feelings, and the seven diatonic modes are examples of such harmonic feelings. But there are others, for example, if you cannot feel a harmonic center pitch, then it's not any mode at all.
First you need to have the skill of creating the harmonic feeling of a mode. Only after you can do that, then you can think about when and why you might want to use that mode's feeling. You must be able to bake a cake, and then you can use a cake. If you do not have the skill, you can buy a ready-made cake and use it.
Similarly, you can select a ready-made musical piece that produces the mode you want, and then you can talk about using the feeling somewhere for some purpose. I suppose there are audio and MIDI libraries which have ready-made musical material for various modes. Producers can use those, if they cannot build their own material from lower level ingredients.
Answer:
How can you get a different sound from modes when they use the same chords from whichever you built the modes off, doesn't really matter if it's major or minor, that part is confusing?

Major and minor keys both use major and minor chords, yet they sound different.  The difference lies in the relationship of each chord to the "home" pitch, and, perhaps more significantly, in the relationship of each degree of the scale to the home pitch.  Most notably, "modal" music is usually identified (in modern music) by having a raised fourth degree or, most especially, a lowered seventh degree.
(In fact, an increasing fashion for chromatic alteration -- raising the leading tone and lowering the fourth and sixth degrees of Mixolydian and Dorian modes, respectively -- was the primary cause of the modern major and minor scales' developing from modes between the late medieval and early baroque periods of music history.  So nowadays chromatic alteration is generally incompatible with the "modal" label, since it typically means that the music is in fact tonal.)
But in popular music, the boundary between modal and tonal music is less clear, in part because of the blue note.  You can use a C7 chord as a functional tonic in a song in C major without having to say that the song is in in the Mixolydian mode.  The difference between the Dorian mode and the natural minor, a.k.a. the Aeolian mode, is subtle to nonexistent, owing to the flexibility of the sixth and seventh degrees of the minor scale.
In practical reality, where music uses chromatic alteration, the primary feature that makes a song modal is likely to be the melody, with chords that don't necessarily come from that mode's diatonic scale.  For example, "Light My Fire" by the Doors contains (looking only at the first verse without the introduction) the chords of A minor, F♯ minor, G, A, D, B, and E.  The pitch class set implied by these chords is A, B, C, C♯, D, D♯, E, F♯, G, and G♯.  None of the diatonic modes contains all of these pitches.
None could, since there are 10 of them.  But the melody uses only the pitches A, B, C, D, E, F♯, and G, quite solidly A Dorian.
Comment: Interestingly(?), quartal harmony feels different with regards to creating a "home pitch" sensation, even while using the same diatonic notes.
Comment: @piiperiReinstateMonica thanks for calling attention to the triadic bias in this answer.  On reflection, though, I think the question is probably implicitly about triadic harmony so I won't edit :-)  But your comment makes me realize that I don't know much in the way of actual quartal music.  I'll have to explore a bit.
Comment: Yes, I saw that your answer tries to take into account characteristics of different harmonic styles - in a way that probably goes over the OP's head by a mile, but it's still good stuff - so why not expand the horizon a bit more. And then there's atonality, which deliberately tries to avoid establishing a tonal center. Is atonality possible with a diatonic scale? I think it has to be, at least with random note mash.


Multimodal prior

Question: In Bayesian method, a posterior can be either unimodal or multimodal. But, I cannot find any multimodal prior case yet.
I wonder if it is possible, and there is any case that is using multimodal prior.
Comment: The prior used to assess the next study (event, set of evidence, whatever) is the posterior from the previous Bayesian analysis (unless this is the very first one). Thus, if a posterior can be multimodal, next time's prior should be multimodal.
Answer: Any distribution can be used as prior. The simplest example of commonly used bimodal prior is Jeffreys beta prior with parameters $\alpha=\beta=1/2$.
Comment: From a practical standpoint, before collecting data what type of problem lends itself to using a bimodal prior?  I am sure there are good answers to this.  It may be what the OP had in mind.
Comment: @MichaelChernick I don't have a general answer to this as this is not a general practice. In this particular case it's a uninformative prior of such form.
Answer: Tim's example is a natural answer to the question (and to address Michael Chernick's comment, the Beta(1/2,1/2) prior is both noninformative and allowing for estimating probabilities close to zero or one). I have however an additional answer to the question which is that any prior is bimodal when using the right reparameterisation!
Indeed consider the reparameterisation from $\mathbb{R}^+$ into the unit interval $(-1,1)$:
$$f:\ t\longrightarrow \dfrac{t^a}{1+t^a}\qquad a>0$$
which has for Jacobian
$$J(x)=\frac{\text{d}\,f^{-1}(x)}{\text{d}\,x}=\frac{1}{a}\dfrac{x^{\frac{1}{a}-1}}{(1-x)^{\frac{1}{a}+1}}$$
with explosive behaviours at $x=0$ and $x=1$. Therefore, given a prior $\pi_1$ on ${\Theta}=\mathbb{R}^+$, the reparameterisation from $\theta$ to $\zeta=f(\theta)$ leads to a prior
$$\pi_2(\zeta)=\pi_1\left\{\zeta^\frac{1}{a}\big/(1-\zeta)^\frac{1}{a}\right\}\times \frac{1}{a}\dfrac{\zeta^{\frac{1}{a}-1}}{(1-\zeta)^{\frac{1}{a}+1}}$$which may also enjoy asymptotes in zero and 1 if $a$ is large enough. If not, faster concentrating transforms should fill the job.

Multimodal prior


downloading a large database

Question: I tried to export a large sized database using phpmyadmin but later the downloaded file seems to be of incorrect size, many of the tables inside the database are eliminated during download. The file size is about 700MB-1GB
In Export of phpmyadmin, i save the file as mydb.sql then in mysql I use "source" command to save it into mysql. The DB file is originally stored in another server (in another country). I access phpmyadmin with password and username. I guess now you understand my situation, I would like to get down this large db file then use it locally for test and development.
Answer: It is much better to use mysqldump for exporting that large database.
You can do this with
mysqldump -u username -p -h hostname --databases yourdatabase > yourfile.sql
But you must have user that can connect from outside of server.
Perfect solution will be do this export localy on server then download it via ftp. But for this you need shell account.
Comment: Thank you for your reply I update to clearify better, I hope you could help me.
Comment: Thank you for your update, it works correctly now. I have used Putty and it also works as expected, the same as what you told me, except I didn't enter the hostname because I connect to the host server using Putty already. Thanks a lot!
Answer: most probably you'd like to know how to export from mysql dataset that is more than few megabytes. web tools are not very well suited for that. if you have access to the shell on the server or remote access to mysql instance where the data is stored - try using mysqldump.
if you dont have shell access you can try using phpmyadmin to dump table after table and there  select row ranges [how many rows per dump? that you'll have to find out empirically].
Comment: Thank, I finished it with the above method. I will try to dump table after table next time if I have another chance.

downloading a large database


Laravel 10 Str::excerpt() with forward slash (/) in phrase throws preg_match(): Unknown modifier 'y'

Question: In my Laravel 10 (PHP 8.1.13) application I try to use Laravel's Str::excerpt() with a / in the second argument (phrase):
Str::excerpt('This is m/y name', 'm/y', [
            'radius' => 3
        ]);

This results in preg_match(): Unknown modifier 'y' exception thrown in
vendor / laravel / framework / src / Illuminate / Support / Str.php : 343:
preg_match('/^(.*?)('.preg_quote((string) $phrase).')(.*)$/iu', (string) $text, $matches);

With every other special char it works just fine. I have already tried to escape the forward slash with a leading backlash, resulting in the same error.
Str::excerpt('This is m/y name', 'm\/y', [
            'radius' => 3
        ]);

It's obviously that it has to do with the preg_quote((string) $phrase) inside Laravel's Str::excerpt(). But is there a way to escape the forward slash or do I need use a different excerpt implementation?
Comment: Oddly, this might be something to bring up as a [Laravel bug report](https://github.com/laravel/framework), especially if you need to have the slash in your phrase.
Comment: Would you not escape it on the pattern? `'This is m\/y name'`
Comment: Try with combining double quotes and also double backslash escape. Just an idea.
Comment: `Str::excerpt` is in a vendor file and not recommended to change.
Comment: @KarlHill: It is part of the framework, so it's not a good idea to change it.
Comment: @Tpojka: I tried Str::excerpt("This is m\y name", "m\\/y", [
            'radius' => 3
        ]);
and Str::excerpt("This is m\y name", "m\/y", [
            'radius' => 3
        ]);
Comment: @jhmckimm: Good idea, but doesn't fix it :(
Comment: @Clansi I see, what's happening is that your input is being passed to `preg_quote` which is escaping the backslash... How annoying. I think what's gone wrong here is that someone has assumed `preg_quote` would also escape forward slashes, when in fact it does not. This is definitely a bug then, I would file a report with Laravel.
Answer: Just found that it has been fixed in 10.29 (I was using 10.27): https://github.com/laravel/framework/pull/48765


Not getting a smooth curve using ggplot2

Question: I am trying to fitting a mixed effects models using lme4 package. Unfortunately I cannot share the data that i am working with. Also i couldn't find a toy data set is relevant to my problem  . So here i have showed the steps that i followed so far :
First i plotted the overall trend of the data as follows :
p21 <- ggplot(data = sub_data, aes(x = age_cent, y = y))
p21+ geom_point() + geom_smooth()

Based on this , there seems to be a some nonlinear trend in the data. Hence I tried to fit the quadratic model as follows :
sub_data$age_cent=sub_data$age-mean((sub_data)$age)
sub_data$age_centsqr=(sub_data$age-mean((sub_data)$age))^2

m1= lmer(y ~ 1 + age_cent +  age_centsqr +(1 | id) , sub_data, REML = TRUE)

In the above model i only included a random intercept because i don't have enough data to include both random slope and intercept.Then i extracted the predictions of these model at population level as follows :
pred1=predict(m1,re.form=NA)

Next I plotted these predictions along with a smooth quadratic function like this
    p21+ geom_point() + geom_smooth(method = "lm", formula = y ~ I(x) + I(x^2)
 ,col="red")+geom_line(aes(y=pred1,group = id) ,col="blue", lwd = 0.5)

In the above plot , the curve corresponds to predictions are not smooth. Can any one helps me to figure out the reason for that ?
I am doing anything wrong here ?
Update :
As  eipi10 pointed out , this may due to fitting different curves for different people.
But when i tried the same thing using a toy data set which is in the lme4 package , i got the same curve for each person as follows :
m1 <- lmer(Reaction ~ 1+I(Days) + (1+ Days| Subject) , data = sleepstudy)

pred1new1=predict(m1,re.form=NA)

p21 <- ggplot(data = sleepstudy, aes(x = Days, y = Reaction))
p21+ geom_point() + geom_smooth()
p21+ geom_point() + geom_smooth()+ geom_line(aes(y=pred1new1,group = Subject) ,col="red", lwd = 0.5)

What may be the reason the for different results ? Is this due to unbalance of the data ?
The data i used collected in 3 time steps and some people didn't have it for all 3 time steps. But the toy data set is a balanced data set.
Thank you
Comment: The `predict` function is returning predictions for every data row. It looks like you have predictions for each level of `id`, and you added `group=id` as an aesthetic in `geom_line`, so you'll get a separate line for each `id`.
Comment: You need both an unbalanced data set and a non-linear (e.g. polynomial) model for the fixed effects to see this effect.  I'll put together an example when I get a chance.  You should be able to get prettier graphs by using `newdata=expand.grid(levels(group),unique(x_values)))` to fill in missing values in your prediction frame.
Comment: Also, it looks like `pred1` is a vector of predictions. I don't know if it's causing any issues here, but in general it's safer to include the predictions in the data frame passed to `ggplot` (`sub_data` in this case) so that you can be sure the relationship between the values in `pred` and the rows of `sub_data` remain consistent.
Comment: @eipi10 Thank you for the suggestions. does this can be due to unbalanced   of data ?  Please refer the updated question , where i include a plot using a toy data set.
Comment: @BenBolker It is great if you can put some example. Thanks . Also can you explain which variable you treated as x_values ? Is it the time related  variable ?
Comment: @BenBolker  Also in the toy example , the the X variables which Days had same set of values (0-9 Days) for each Subject. But in my example, I used the age  of each person at each wave (time step)  as the X variable. So the age values are not same for each person at each time step. I think that may be another reason for this different results.
Answer: tl;dr use expand.grid() or something like it to generate a balanced/evenly spaced sample for every group (if you have a strongly nonlinear curve you may want to generate a larger/more finely spaced set of x values than in the original data)
You could also take a look at the sjPlot package, which does a lot of this stuff automatically ...
You need both an unbalanced data set and a non-linear (e.g. polynomial) model for the fixed effects to see this effect.

if the model is linear, then you don't notice missing values because the linear interpolation done by geom_line() works perfectly
if the data are balanced then there are no gaps to get weirdly filled by linear interpolation

Generate an example with quadratic effects and an unbalanced data set; fit the model
library(lme4)
set.seed(101)
dd <- expand.grid(id=factor(1:10),x=1:10)
dd$y <- simulate(~poly(x,2)+(poly(x,2)|id),
                 newdata=dd,
                 family=gaussian,
                 newparams=list(beta=c(0,0,0.1),
                                theta=rep(0.1,6),
                                sigma=1))[[1]]
## subsample randomly (missing values)
dd <- dd[sort(sample(nrow(dd),size=round(0.7*nrow(dd)))),]
m1 <- lmer(y ~ poly(x,2) + (poly(x,2)|id) , data = dd)

Naive prediction and plot:
dd$pred1 <- predict(m1,re.form=NA)

library(ggplot2)
p11 <- (ggplot(data = dd, aes(x = x, y = y))
    + geom_point() + geom_smooth(method="lm",formula=y~poly(x,2))
)
p11 + geom_line(aes(y=pred1,group = id) ,col="red", lwd = 0.5)

Now generate a balanced data set. This version generates 51 evenly spaced points between the min and max - this will be useful if the original data are unevenly spaced.  If you have NA values in your x variable, don't forget na.rm=TRUE ...
pframe <- with(dd,expand.grid(id=levels(id),x=seq(min(x),max(x),length.out=51)

Make predictions, and overlay them on the original plot:
pframe$pred1 <- predict(m1,newdata=pframe,re.form=NA)
p11 + geom_line(data=pframe,aes(y=pred1,group = id) ,col="red", lwd = 0.5)

Comment: seems reasonable.
Comment: Thank you for your  clarification. As i told you in my previous comment, my X variable is the age at each wave (time step). I used age as the time steps are not equally separated. So that will bring additional unbalance to the data.
Comment: Also the underlying objective of this work is :to determine the overall trend of the data and whether that overall trend has captured by my linear mixed effects models ,by plotting predictions at population level. In that perspective ,is it possible to to tell me whether i am using ggplot package correctly ? Thank you once again


Proper way to handle not needed/old/stale AWS Athena partitions

Question: We have encountered a situation where we need to figure out how to handle stale AWS Athena partitions due to Athena's service limits (20,000 partitions per table).
Say we want to have only one table and add a number of partitions there every day that reference a timestamp-like path to some logs located on S3 (example: <code>/foo_bucket/logs/year=2019/month=03/day=11/hour=20</code>).
We found out that this approach lets us reach the partition limit in about 2 years. After that, we want to clean up old logs and partitions that are associated with them.
Questions:

What will happen with partition metadata if an associated S3 path gets removed? This S3 bucket automatically removes old objects. Will the partition automatically be deleted as well or will it reference a non-existent S3 path?
What will happen when we reach partition limit per table before S3 automatically deletes old objects? Will old partition metadata get deleted by Athena? I know that deleting a partition does not touch S3 object data (link).

Thanks!
Answer: Question 1:
As the metadata and data are totally unlinked in AWS Athena, you will need to call the command that basically says "hey, refresh my table partitions list" wich is MSCK REPAIR TABLE
https://docs.aws.amazon.com/athena/latest/ug/msck-repair-table.html
Question 2:
I think you will get and error when Athena tries to add a new partition in the metastore as the old partition metadata is not removed.
Feedback:
Maybe it's not a bad idea to consolidate old content into less granular partitions. For example, you can keep a day granularity for the last 3 - 4 months. Anything older than that can be stored per month. That will allow you to keep much more content in your table.
I hope it helps :)
Comment: You will see a message saying something like: "Partition yyyy-mm-dd missing" and it will be removed from the metastore. In your scenario if you are not reaching the partitions limit then you can work that way, old content will be removed and the MSCK REPAIR TABLE will refresh all the table metadata removing the old ones and adding the new ones. You can easily test this with just an empty table and play creating/deleting folders in S3.
Comment: If your goal is to remove the entire table, you don't need to worry about partitions. All that information will be removed once you drop the table at metastore level.
Comment: What will `MSCK REPAIR TABLE` return if it finds partitions without associated S3 paths? Will it remove the partitions, their metadata? Is the following the likely scenario in this case: 1. S3 cleans up old objects. 2. On next partition addition operation, after `MSCK REPAIR TABLE` query, Athena finds partitions with unlinked data and removes them. 3. Profit: no data on S3 and no partitions associated with that data.
Comment: Would athena allow to correctly perform `MSCK REPAIR TABLE` if paths are inconsistent, e.g. for the past 3 months it is `/foo_bucket/year=2019/month=08/day=01/hour=00`for everything earlier then that it is just `/foo_bucket/year=2019/month=05/day=01`?
Comment: No, a table is defined with a partition structure on creation time. You have to always respect that structure. If you have a granularity of hours and you want to consolidate one month in a single partition you need to put all month content into a single "hour" folder. For example, content from 2019-02-03 04:05:06 should me moved to partition /foo_bucket/year=2019/month=02/day=01/hour=00. But when you query old content, you really don't need to remember this things, just filter by year and month only. I feel I didn't explained myself well enough :D. If you don't understand it, let me know :)
Comment: Ok, I see now what you mean and it is quite nifty workaround with a simple renaming file prefixes. But what if you have month=01/day=02/**hour=00**/file_1 and month=01/day=02/**hour=01**/file_1? Wouldn't one overwrite another after "putting" these files in "folder" ..month=01/day=01/? I guess, keeping full timestamp in the file name and not only in directory/prefix structure should be a rule thumb)
Comment: I suggest not running `MSCK REPAIR TABLE` on a table with 20K partitions, it will take a very, very long time. You're probably better off listing the partitions using the Glue API and doing the pruning yourself if it's something that's going to happen often.
Comment: Since partitions are "metadata" for the table, I wonder if they should be removed (potentially with `MSCK REPAIR TABLE`) before removing the table itself?


reopening sys.stdout after it is closed by with statement

Question: I'm having trouble with a printing information that is input from a yaml file using PyYAML. I'm trying to reduce line-count, without affecting functionality. In some runs the output has to be appended to a file, in others to stdout.
At first I used this multiple times in my function processData:
    if logName:
        fp = open(logName, 'a')
    else:
        fp = sys.stdout
    print(........, file=fp)
    print(........, file=fp)
    if logName:
        fp.close()

That worked, but has the disadvantages of not using the with statement when something goes wrong.
The actual problem is not the complex print statements, but that I
1) don't want to duplicate code when printing to file or to sys.stdout
2) want to use the with statement so that files get closed if there are print errors
3) there are several such blocks, I don't want to call a different function for each of them, and so preventing code duplication
Then what I tried is:
def processData(yamlData, logName=None):
    ......
    with open(logName, 'a') if logName else sys.stdout as fp:
        print(........, file=fp)
        print(........, file=fp)
    .....
    with open(logName, 'a') if logName else sys.stdout as fp:
        print(........, file=fp)
        print(........, file=fp)

If there is not a logName, this errors to "ValueError: I/O operation on closed file". Any suggestions on how to get this to work without the original duplication? Can I reopen sys.stdout?
Comment: Instead of printing you can use a logger. It's a best practice.
Comment: Thanks I'll look into that. Some many useful packages, so little time...
Comment: @Brad Look no further! Enter _Python_, the first language and standard library where _"there should be one, and preferably,_ only _one obvious way to do it"!_ If there's some functionality in the standard library, you can bet they thought hard to make it cover _every_ typical use case.
Comment: (OT) btw, was that a Duke Nukem reference? ;)
Answer: You can "wrap" sys.stdout in a class to prevent it from being closed in the first place.
The with statement calls __enter__ and __exit__ on instances of that class at the beginning and end resp., so just make sure the __exit__ doesn't do anything:
class StdOut:
    def __enter__(self):
        return sys.stdout

    def __exit__(self, typ, val, trace):
        pass

stdout = StdOut()

and then use stdout instead of sys.stdout
Answer: Literal question - reopening stdout
At the lowest C level, stdout is a well-known file descriptor (an integer pointing to an entry in the runtime- or system-managed descriptor table), initialized in a process upon its creation. It cannot be reopened (with standard C means) once it's closed and must be duplicated beforehand if you still need it.
A disposable copy of sys.stdout can be created like this:
stdout_copy=open(os.dup(sys.stdout.fileno()),sys.stdout.mode)

(In Python 2, os.fdopen() was used instead of open and specifying mode wasn't needed 'cuz it was able to infer it from the descriptor.)
You may need to use sys.__stdout__ instead if sys.stdout has been replaced.
The other question - wrapping functionality into with logic
First of all, consider the standard way to log - namely, the logging module - to avoid reinventing the square wheel. Opening and closing the file on demand can very well be implemented with its machinery, and it's not even needed in the vast majority of cases.
Now, the only way to cut on the repetitive parts of the code is to wrap the repeating part into a subroutine (or a code block that processes a list with elements describing what it should do each iteration, but it can only be used once). There are three conceptual parts here, regardless of syntax (it can be try/finally just as well as with):

the wrapping construct

incl. exception handling

the opening+closing code
the wrapped code

Wrapping just "the opening+closing code" is the easiest, the other answer is one possible way, but it leaves the repeating with and print(........, file=fp) parts.

Wrapping the entire construct is harder since you'll have to pass a code chunk to your would-be subroutine, and Python intentionally omits anonymous code blocks - you'd have to def and then immediately use it which is rather awkward.

a decorator or passing your code as a callback are two possibilities.
If your code can be reduced to a pattern (like a set of messages), you can pass just that pattern instead and have the subroutine process it.
Comment: You need to pass `"w"` to `fdopen()`, otherwise you open stdout for reading.
Comment: @MartijnPieters Thanks. That was written for Py2. I specifically checked that `fdopen` inferred the mode from the descriptor (since it's already open in some mode, it's pointless to specify anything else). Looks like it's no longer smart enough. Darn, now I'll have to resort to duplication to explicitly specify the mode.


linux3/gcc46: "-fnon-call-exceptions", which signals are trapping instructions?

Question: (Environment: gcc/g++ 4.6.1 in -std=gnu++0x mode on Linux 3.0 / x86_64...)
<code>#include <stdlib.h>
#include <signal.h>
#include <iostream>

using namespace std;

class SegmentationFault {};

void ThrowSegmentationFault(int)
{
    throw SegmentationFault();
}

void ohno(char* x)
{
    *x = 42;
}

int main()
{
    signal(SIGSEGV, ThrowSegmentationFault);

    try
    {
        ohno(0);
    }
    catch (const SegmentationFault&)
    {
        cout << "success" << endl;
    }
}
</code>
By compiling the above with the -fnon-call-exceptions flag, it allows the SIGSEGV signal handler to throw an exception, and when run it will print "success". The documentation of the -fnon-call-exceptions gcc flag reads as follows:

Generate code that allows trapping instructions to throw
  exceptions. Note that this requires platform-specific runtime support
  that does not exist everywhere. Moreover, it only allows trapping
  instructions to throw exceptions, i.e. memory references or floating
  point instructions. It does not allow exceptions to be thrown from
  arbitrary signal handlers such as SIGALRM.

My question is which of the signals are trapping instructions and which are not?
<code>#define SIGHUP           1
#define SIGINT           2
#define SIGQUIT          3
#define SIGILL           4
#define SIGTRAP          5
#define SIGABRT          6
#define SIGIOT           6
#define SIGBUS           7
#define SIGFPE           8
#define SIGKILL          9
#define SIGUSR1         10
#define SIGSEGV         11
#define SIGUSR2         12
#define SIGPIPE         13
#define SIGALRM         14
#define SIGTERM         15
#define SIGSTKFLT       16
#define SIGCHLD         17
#define SIGCONT         18
#define SIGSTOP         19
#define SIGTSTP         20
#define SIGTTIN         21
#define SIGTTOU         22
#define SIGURG          23
#define SIGXCPU         24
#define SIGXFSZ         25
#define SIGVTALRM       26
#define SIGPROF         27
#define SIGWINCH        28
#define SIGIO           29
#define SIGPOLL         SIGIO
/*
#define SIGLOST         29
*/
#define SIGPWR          30
#define SIGSYS          31
#define SIGUNUSED       31
</code>
Answer: SIGILL, SIGTRAP, SIGBUS, SIGFPE, SIGSEGV, SIGSTKFLT are the most probable synchronous signals (i.e., generated by hardware as consequence of an instruction trying to do something invalid).


Note Accentuation (Dynamic) - When should/shouldn't you apply accentuation?

Question: I am having some concern over whether it is appropriate or not to apply accentuation to every part of the song, apart from of course the drums, It's innately intuitive that these beats should be accentuated to give a sense of dynamic rhythm.
The ambiguity for me, though, is when it comes to everything else, basically the parts of a song which contributes a harmony to the sound (as opposed to the percussiveness of the drums), so, primarily, the bass and lead.
Here is a visual sample of an attempt to apply accentuation to the lead:

I've tried to go with the standard Sw, Smw, Swmw type pattern, but i've also tweaked this pattern where I think it may be more appropriate. Though, to be honest, am really not sure if the overall the context of the song actually benefits from this. I've tried other variants, as well as just 'going with the flow', where I ditch the theory entirely and apply what ever accent I wish to go for. None of these patterns seem to make it sound significantly better however. I think it might sound slightly better with the standard accentuation taught in theory, but I think that's just because I know I may applying the 'correct' accentuation, where, when applying a blind test to myself, sometimes I cannot tell the difference at all.
Simultaneously however, I am very much attracted to the idea of dynamic rhythm, like the rolling of the tide, or the rotation of the earth, but acoustically. This works so well with the drums, so am wondering, in your personal experience, do you think it's better to just focus on the relative volume between the different parts, or is it worth investing the time to vary the notes within a specific instrument itself.
Answer: Strong opinions ahead...
Almost1 all *ahem* proper instruments will inevitably introduce some dynamic variations, because the player will (deliberately or involuntarily) hit each note a bit differently. And a good musician will intuitively get this "right", whatever that means exactly (there will be more than one right way).
Surpressing this natural variation should IMO be considered a special effect. Do it if you want to sound robotic / industrial. Otherwise, track the part on a real instrument or MIDI keyboard, and keep the dynamics as recorded.
Setting the dynamic level of each note manually, OTOH, will either imitate a human response (more or less convincingly – it is very laborious), or sound even weirder/mechanical than a completely flat dynamic level.
tl;dr: you should almost always apply accentuation, but do it "live".

1Even organ, harpsichord and synthesizers without velo – while these don't allow per-note dynamic variations, they still allow timing variations, which are perhaps even more important.
Answer: A complicated proposition to be sure: you are now wrestling with: What I've been taught doesn't fit for what I want to do.  This is normal for everyone who writes music.
Your patterns of articulation won't make any sense if they don't complement the way the phrase is written.  The whole point of articulation is to articulate (see: "express") the line that you've written.  Think about what you've written.  Think about which notes should be emphasized.  Are you hearing (imagining) the notes go "doo DAH doo DAH" or "DOO dah DOO dah"? Your articulations should better define the music you hear in your head, NOT the other way around.
Here is a good exercise for practicing how to articulate your phrases:

MY cat went to the store.
My CAT went to the store.
My cat WENT to the store.
My cat went TO the store.
My cat went to THE store.
My cat went to the STORE.

Reach each sentence out loud and emphasize the bold word.  Each time, the meaning of the sentence changes very slightly.  For example, "my" cat, instead of someone else's cat.
I think that live sequencing instruments is definitely an option, however, I wouldn't say it's the only option.  I very, VERY often sing parts that I'm writing. Singing a musical line shows me how I'm thinking about it and how I'm hearing it. After I sing I change the written music according to the sounds I'm hearing, NOT the other way around.
Alternatively, you could play the parts on a non-sequencing live instrument (guitar / piano) and do the same thing: analyze how you're playing and adjust the notation accordingly.  If you find you're getting louder in a section, then change the channel volume appropriately for that section.

Not all notes are equally important.  Ask yourself, "What are the most important notes in my phrase?"  You should always be aiming for a target.  Working toward a target will give your music forward motion, and it'll make it easier for your to shape your phrases since you'll have a better idea of what's important or not.
If you cannot play any instruments, learn one.  Writing music on the computer is okay, but it is a vacuum if you have no real-world experience.
Comment: Hello Jimusicnotes, thanks very much for your answer. I think you raise a very good point when it comes to emphasis, in that you wouldn't say, want to accent the 'the', over a keyword like 'love', 'hate', or 'cat', (obviously, whenever the word cat is found within a lyrical line, that will always take presidency :D). I also very much appreciate your tip on singing the words, and I do indeed find myself applying accentuation without even trying, as singing comes so innately to all of us. Anyway, thanks again!


Mapserver : loadWeb(): Unknown identifier. Parsing error near (C):(line 1)

Question: I am new to Mapserver.
I have done sample application..
   MAP
   NAME "Hello_World"
   SIZE 400 300
   IMAGECOLOR 249 245 186
   IMAGETYPE png
   EXTENT -1.00 -1.00 1.00 1.00
   WEB
       TEMPLATE "C:/Mapserver/ms4w/Apache/htdocs/Hello/Hello.html"
       IMAGEPATH "C:/Mapserver/ms4w/Apache/htdocs/tmp/"
       IMAGEURL "/tmp/"
  END
  LAYER
       NAME 'TEST_LAYER'
       STATUS default
       TYPE point
       FEATURE
       POINTS 0.0 0.0 END
       TEXT "Hello World"
       END
  CLASS
      STYLE
         COLOR 255 0 0
      END
     LABEL
     TYPE bitmap
     END
 END # end class
 END # end layer
 END # end mapfile

And my Html file is :

<title>MapServer Hello World</title>

       <form method=POST action="/cgi-bin/mapserv.exe">
          <input type="submit" value="Click Me">
          <input type="hidden" name="map" value="C:/Mapserver/ms4w/Apache/htdocs/Hello/mapdata/hello.map">
          <input type="hidden" name="map_web_imagepath" value="C:/Mapserver/ms4w/Apache/htdocs/tmp/">
       </form>

When I run the through URL
http://localhost:88/cgi-bin/mapserv.exe?map=C:/Mapserver/ms4w/Apache/htdocs/Hello/mapdata/hello.map
It generated image in the temporary folder.
When I click the "Click me" button , Its say  loadWeb(): Unknown identifier. Parsing error near (C):(line 1)
I am using MapServer version 5.6.6
Please anybody help me.
Thanks in advance.
Answer: I got similar problem when I call WMS via URI.
http://sun-web-extdev.ga.gov.au/cgi-bin/mapserv?map=map.map&map_web_template=template.html&map_imagetype=png
action to fix this problem:
remove template line in .map file and using following WMS URI:
http://sun-web-extdev.ga.gov.au/cgi-bin/mapserv?map=map.map&map_web=template+template.html&map_imagetype=png
from version 5 WMS discard config template file in .map file instead using append parameter as above.
in your case try add
Answer: in your html file put this comment in the first line

and erase this line:
<input type="hidden" name="map_web_imagepath" value="C:/Mapserver/ms4w/Apache/htdocs/tmp/">

this worked for me.
Answer: I got the same error while creating some tutorial on using MapServer on Windows.
The solution was change all "/" to "//" on the map file. Since Windows is sensitive to backslashes
Using the example IMAGEPATH path above
This
IMAGEPATH "C:/Mapserver/ms4w/Apache/htdocs/tmp/"
Changed to this Worked
IMAGEPATH "C://Mapserver//ms4w//Apache//htdocs//tmp//"


Global PHP "sort through" vs other fields

Question: In drupal views, you can create a field called "global php" and insert your own php code. Also as well, it has a sort feature which is handy when your view style is a table.
However, the other fields does not have a "sort" feature on them. Is there a way to add it to them? Or does it just have to come with it, like global php does?
Answer: It has to be supported by the module supplying the field. In hook_views_data(), each defined field has a 'sort' element that defines the sort handler function for the field.


Suppress Console.Write in referenced package

Question: In one project, I reference a NuGet package which outputs text to the Console when some of its methods are called.
I would prefer it didn't, as I am writing things to the Console in my own code.
Is there a way to suppress the Console writing happening in that package, but still be able to write to Console myself?
Comment: Is it writing to console (like `Console.Write`) or actually using logging which you happen to have going to console?
Answer: You can use Console.SetOut to set the default output to any other TextWriter - but still use the original Console.Out. Here's a demo of that:
using System;
using System.IO;

class Test
{
    static void Main()
    {
        var originalOut = Console.Out;
        Console.SetOut(TextWriter.Null);

        LibraryMethod();
        originalOut.WriteLine("This should still go to the console.");
    }

    static void LibraryMethod()
    {
        Console.WriteLine("Imagine this were in the referenced package.");
        Console.WriteLine("This isn't going anywhere. It's being discarded.");
    }
}

That does work - but it's pretty annoying. I would personally ask the author of the original package to modify the package if possible - it's pretty odd for a library to write to the console by default. It should at least allow you to specify which TextWriter to write to.
Comment: Can also use `TextWriter.Null` i guess, which is pretty much the same thing
Comment: @TheGeneral: Ooh, nice - didn't know about that. Will edit that in.
Comment: It turns out, after trying this, that it wasn't using the standard `Console.Out` or `Console.Error` streams. The package being referenced was Tikaondotnet, which uses `IKVM`. So by setting `java.lang.System.setOut(null);` and `java.lang.System.setErr(null);` (yes, in C# code), the output was successfully suppressed.


SonarLint Analysis failed when using VS Code extension for JavaScript files

Question: I've tried to set up the SonarLint Extension for Visual Studio Code on Windows, and followed the instructions to use Connected mode. I think I've set it up correctly because in the  SonarLint output logs, shows some 200 status codes for connecting to my organisation server (FYI I changed the URL in the log below):
Connected SonarLint engine started for example-project
GET 200 http://example.com/sonar/api/qualityprofiles/search.protobuf?projectKey=example-project | time=152ms
GET 200 http://example.com/sonar/api/properties?format=json&resource=example-project | time=70ms
GET 200 http://example.com/sonar/api/components/tree.protobuf?qualifiers=BRC&baseComponentKey=example-project&ps=500&p=1 | time=78ms
GET 200 example.com/sonar/batch/issues?key=example-project | time=74ms

It then shows that it's trying to analyse a file that I have open:
Analysis triggered on file:///c%3A/Users/example/src/example/src/js/example.js with configuration:
[
  moduleKey: example
  baseDir: c:\Users\example\src\example
  workDir: c:\Users\example\src\example\.sonarlint
  extraProperties: {}
  inputFiles: [
    c:\Users\example\src\example\src\js\example.js (UTF-8)
  ]
]

However, it then shows the following error messages:
Available languages:
  * JavaScript => "js"
  * Java => "java"
Start analysis
Declared extensions of language JavaScript were converted to js: **/*.js
Declared extensions of language Java were converted to java: **/*.java,**/*.jav
Quality profiles:
  * java: java-sonar-way-77058 (251 rules)
  * js: js-sonar-way-33138 (86 rules)
[Info  - 10:43:48 AM] Index files
Language of file 'file:///c:\Users\example\src\example\src\js\example.js' is set to 'js'
Setting filesystem encoding: UTF-8
[Info  - 10:43:48 AM] 1 files indexed
'JavaSquidSensor' skipped because there is no related file in current project
Execute Sensor: JavaScript Squid Sensor
[Info  - 10:43:49 AM] 1 source files to be analyzed
Initializing metadata of file file:///c:\Users\example\src\example\src\js\example.js
[Error - 10:43:50 AM] Analysis failed.
org.sonar.squidbridge.api.AnalysisException: Unable to analyse file: c:\Users\example\src\example\src\js\example.js
    at org.sonar.plugins.javascript.JavaScriptSquidSensor.analyse(JavaScriptSquidSensor.java:174)
    at org.sonar.plugins.javascript.JavaScriptSquidSensor.analyseFiles(JavaScriptSquidSensor.java:133)
    at org.sonar.plugins.javascript.JavaScriptSquidSensor.execute(JavaScriptSquidSensor.java:349)
    at org.sonarsource.sonarlint.core.analyzer.sensor.SensorWrapper.analyse(SensorWrapper.java:52)
    at org.sonarsource.sonarlint.core.analyzer.sensor.AllSensorsExecutor.executeSensor(AllSensorsExecutor.java:79)
    at org.sonarsource.sonarlint.core.analyzer.sensor.AllSensorsExecutor.execute(AllSensorsExecutor.java:67)
    at org.sonarsource.sonarlint.core.analyzer.sensor.PhaseExecutor.execute(PhaseExecutor.java:36)
    at org.sonarsource.sonarlint.core.container.analysis.AnalysisContainer.doAfterStart(AnalysisContainer.java:141)
    at org.sonarsource.sonarlint.core.container.ComponentContainer.startComponents(ComponentContainer.java:125)
    at org.sonarsource.sonarlint.core.container.ComponentContainer.execute(ComponentContainer.java:110)
    at org.sonarsource.sonarlint.core.container.storage.StorageAnalyzer.analyze(StorageAnalyzer.java:75)
    at org.sonarsource.sonarlint.core.container.storage.StorageContainerHandler.analyze(StorageContainerHandler.java:76)
    at org.sonarsource.sonarlint.core.ConnectedSonarLintEngineImpl.lambda$analyze$0(ConnectedSonarLintEngineImpl.java:152)
    at org.sonarsource.sonarlint.core.ConnectedSonarLintEngineImpl.withReadLock(ConnectedSonarLintEngineImpl.java:339)
    at org.sonarsource.sonarlint.core.ConnectedSonarLintEngineImpl.withReadLock(ConnectedSonarLintEngineImpl.java:329)
    at org.sonarsource.sonarlint.core.ConnectedSonarLintEngineImpl.analyze(ConnectedSonarLintEngineImpl.java:150)
    at org.sonarlint.languageserver.SonarLintLanguageServer$ConnectedAnalysisWrapper.analyze(SonarLintLanguageServer.java:718)
    at org.sonarlint.languageserver.SonarLintLanguageServer$ConnectedAnalysisWrapper.analyze(SonarLintLanguageServer.java:699)
    at org.sonarlint.languageserver.SonarLintLanguageServer.analyze(SonarLintLanguageServer.java:605)
    at org.sonarlint.languageserver.SonarLintLanguageServer.didOpen(SonarLintLanguageServer.java:539)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.lambda$null$0(GenericEndpoint.java:61)
    at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.notify(GenericEndpoint.java:148)
    at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.handleNotification(RemoteEndpoint.java:216)
    at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.consume(RemoteEndpoint.java:183)
    at org.eclipse.lsp4j.jsonrpc.Launcher$Builder.lambda$wrapMessageConsumer$0(Launcher.java:308)
    at org.eclipse.lsp4j.jsonrpc.validation.ReflectiveMessageValidator.consume(ReflectiveMessageValidator.java:71)
    at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.handleMessage(StreamMessageProducer.java:188)
    at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.listen(StreamMessageProducer.java:90)
    at org.eclipse.lsp4j.jsonrpc.json.ConcurrentMessageProcessor.run(ConcurrentMessageProcessor.java:95)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException: 12 is not a valid line offset for pointer. File [path=c:\Users\example\src\example\src\js\example.js] has 1 character(s) at line 105
    at org.sonar.api.internal.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
    at org.sonar.api.batch.fs.internal.DefaultInputFile.checkValid(DefaultInputFile.java:312)
    at org.sonar.api.batch.fs.internal.DefaultInputFile.newPointer(DefaultInputFile.java:246)
    at org.sonar.api.batch.fs.internal.DefaultInputFile.newRange(DefaultInputFile.java:261)
    at org.sonar.javascript.cpd.CpdVisitor.visitNode(CpdVisitor.java:80)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:74)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visitChildren(SubscriptionVisitor.java:94)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.visit(SubscriptionVisitor.java:76)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.scanTree(SubscriptionVisitor.java:68)
    at org.sonar.plugins.javascript.api.visitors.SubscriptionVisitor.scanTree(SubscriptionVisitor.java:62)
    at org.sonar.plugins.javascript.JavaScriptSquidSensor.scanFile(JavaScriptSquidSensor.java:235)
    at org.sonar.plugins.javascript.JavaScriptSquidSensor.analyse(JavaScriptSquidSensor.java:163)
    ... 37 more

Comment: hello. can you share the file which is failing? Also what is the SonarJS version on the SonarQube server you are connecting to? Does it fail on the same file when you are not in the connected mode?
Comment: @TiborBlenessy thanks for your reply, the SonarJS version on the SonarQube server is `22.214.171.12466`. This issue happens on every JS, and it only happens when in connected mode, it works fine when not in connected mode.
Comment: Can you try to upgrade SonarJS on the server side? The version you are using is very old.
Comment: @TiborBlenessy we updated to SonarJS: 5.0 Build 6962, and now it works. Thank you
Answer: As suggested in the comments, the problem was that the server it was trying to connect to was on an old version of SonarJS. We updated it to SonarJS: 5.0 Build 6962, and now the SonarLint extension for VS Code works perfectly in connected mode.


PHP foreach cannot echo values

Question: I am sure that I am missing something that simple, but the following is a foreach loop and if I do a var_dump of $value, then I see all:
foreach ($vals as $key => $value) {
  var_dump($value);
}

The following is a portion of the var_dump:
array (size=4)
  'tag' => string 'MAILSERVICE' (length=11)
  'type' => string 'complete' (length=8)
  'level' => int 4
  'value' => string 'Priority Mail Express 2-Day<sup>™</sup>' (length=57)

array (size=4)
  'tag' => string 'RATE' (length=4)
  'type' => string 'complete' (length=8)
  'level' => int 4
  'value' => string '61.15' (length=5)

array (size=3)
  'tag' => string 'POSTAGE' (length=7)
  'type' => string 'close' (length=5)
  'level' => int 3

array (size=4)
  'tag' => string 'POSTAGE' (length=7)
  'type' => string 'open' (length=4)
  'level' => int 3
  'attributes' =>
    array (size=1)
      'CLASSID' => string '2' (length=1)

array (size=4)
  'tag' => string 'MAILSERVICE' (length=11)
  'type' => string 'complete' (length=8)
  'level' => int 4
  'value' => string 'Priority Mail Express 2-Day<sup>™</sup> Hold For Pickup' (length=73)

array (size=4)
  'tag' => string 'RATE' (length=4)
  'type' => string 'complete' (length=8)
  'level' => int 4
  'value' => string '61.15' (length=5)

After seeing those results, I changed the foreach loop to the following, thinking that I was going to just mail service, along with it's rate, but instead just a blank white page:
foreach ($vals as $key => $value) {
  if ($value[$key]['tag'] == "MAILSERVICE") {
    echo $value[$key]['value'];
  }
  if ($value[$key]['tag'] == "RATE") {
    echo $value[$key]['value'];
  }
}

Using error_reporting(E_ALL);
    ini_set('display_errors', 1);
Shows undefined index, but those warnings are the same if I just do a var_dump($value)
I know that is has to be something simple that I missing and thanks in advance
Comment: The `$key` belongs to `$vals` Either remove it `$value['tag']` or do `$vals[$key]['tag']`
Comment: how about `print_r($value, true);`
Comment: @Dharman, thank you. I have posted the answer and given you the credit
Answer: Thanks to @Dharman for showing me the light:
foreach ($vals as $key => $value) {
  if ($vals[$key]['tag'] == "MAILSERVICE") {
    echo $vals[$key]['value'];
  }
  if ($vals[$key]['tag'] == "RATE") {
    echo $vals[$key]['value'];
  }
}

Answer: $value is the array composed by the 4 items you provided, the  undefined index is probably $value[$key] , i suppose instead of
$value[$key]['value']

you should use
$value['value']

PHP foreach cannot echo values


Why does 1..99,999 == "1".."99,999" in R, but 100,000 != "100,000"?

Question: In the console, go ahead and try
> sum(sapply(1:99999, function(x) { x != as.character(x) }))
0

For all of values 1 through 99999, "1" == 1, "2" == 2, ..., 99999 == "99999" are TRUE. However,
> 100000 == "100000"
FALSE

Why does R have this quirky behavior, and is this a bug? What would be a workaround to, e.g., check if every element in an atomic character vector is in fact numeric? Right now I was trying to check whether x == as.numeric(x) for each x, but that fails on certain datasets due to the above problem!
Comment: No, that sum is zero, not "TRUE"...
Comment: @JoshuaUlrich can you explain how those are duplicates?
Comment: For the problem described in the last paragraph, you could `match` your input character vector against `1:100000` (which is an integer vector): `match(as.character(1:100000),1:100000)`.
Comment: I think `!is.na(as.numeric(x))` (or some equivalent using `all()` for the vectorized case) should work for a test ...
Answer: Have a look at as.character(100000). Its value is not equal to "100000" (have a look for yourself), and R is essentially just telling you so.
as.character(100000)
# [1] "1e+05"

Here, from ?Comparison, are R's rules for applying relational operators to values of different types:

If the two arguments are atomic vectors of different types, one is
   coerced to the type of the other, the (decreasing) order of
   precedence being character, complex, numeric, integer, logical and
   raw.

Those rules mean that when you test whether 1=="1", say, R first converts the numeric value on the LHS to a character string, and then tests for equality of the character strings on the LHS and RHS. In some cases those will be equal, but in other cases they will not. Which cases produce inequality will be dependent on the current settings of options("scipen") and options("digits")
So, when you type 100000=="100000", it is as if you were actually performing the following test. (Note that internally, R may well/probably does use something different than as.character() to perform the conversion):
as.character(100000)=="100000"
# [1] FALSE

Comment: @SeñorO In `x == y`, if `y` is character then `x` is converted, so it is *as if* `as.character(100000)=="100000"` were called. It doesn't matter which side of the binary operator this is; as long as one of the pair is character, the other will be coerced to character if not already so.
Comment: From `?'=='`: "If the two arguments are atomic vectors of different types, one is coerced to the type of the other, the (decreasing) order of precedence being character, complex, numeric, integer, logical and raw." So is it correct to say that LHS side is getting coereced to character and not the other way around?
Comment: @GavinSimpson -- Thanks for the clarifications. I've edited one of them (the bit about "*as if*") into my answer.
Comment: Thanks, I was able to solve the problem using:

`old_opts <- options(scipen = 1000)`
`on.exit(options(old_opts))`


.ftpaccess file and Pure-FTPD

Question: I've been looking for a way to have specific users who have access my FTP to have read-only permission on particular directories. I came across some articles on creating .ftpaccess files (which I've read are similar to .htaccess files) to create customized configurations for specific directories and sub-directories.
After reading everything I could find about .ftpaccess files through Google and attempting to create said files I've had no luck.
Does anyone know the syntax that is needed to get these files to work? And is there a particular setting that I need to have enabled to enable these files? I've looked through the conf files but found nothing.
Answer: AFAIK, Pure-FTPd doesn't support per directory access control. .ftpaccess belongs to ProFTPD.
You probably want to switch to ProFTPD with something like this:
<Directory /path/to/particular/directory>
    <Limit CWD PWD DIRS READ>
        AllowUser specific_user
    </Limit>

    <Limit ALL>
        DenyUser specific_user
    </Limit>
</Directory>

.ftpaccess file and Pure-FTPD


Statistical Independence in the real world

Question: I read the following article about statistical independence. In summary, the article argues that "It is time for science to retire the fiction of statistical independence," and goes on to explain different reasons why.  Having read the article, I tend to agree.  I wanted to know the following:

What do other cross-validated users think?
Are there scholarly resources that you all can point me to that either confirm or reject the notion set forth by the article?  More specifically, whether real-life datasets do (or do not) exhibit statistical independence?

Thanks!
Comment: I think we are not as naive and ill-informed as "the overwhelming common practice" to which the author refers.  There are many false assumptions, straw men, and downright incorrect claims in that little piece.
Comment: I guess the author of this article has never tried flipping coins before....
Comment: His complaint seems to be that sometimes, people use the wrong model and inappropriately apply the independence assumption. That's certainly plausible... but independence exists as a concept not for the service of science but because it arises mathematically. It's almost like the author claims there's no role for mathematics that may be misused in scientific applications.
Answer: It seems to me that the author assumes that most scientists don't know about or understand how to deal with correlation and assumes almost that the use of methods to handle correlated data doesn't exist (perhaps outside of Makov Chains).  That's not the case.  There are many statistical methods that account for correlated data and most statisticians, epidemiologists, ecologists, and other scientists know (or should) when to use the appropriate methods.  I don't think scientists need to abandon methods that assume independence as they are quite useful -- if they weren't simulations and real-world experiments which have demonstrated their usefulness would not abound.  Instead, if anything, scientists need better training or education to understand when to use methods that account for correlation and when to use methods that require assumptions of independence.
That's just my two cents.
Answer: I don't subscribe to the author's view at all. In particular from my experience it is absolutely not the case that "[..] the overwhelming common practice is simply to assume that sampled events are independent". On the contrary, the issue of correlation is something we have to deal with on a regular basis (during my work in the financial industry). And, mostly important, we are clearly aware of this!
I totally agree, though, with the statements on simplifying the real world. For me the famous words attributed to George Box are the leading guide here:

All models are wrong; some models are useful.
Comment: Thanks for reminding us of those famous words from George Box, I think that is very true and relevant here.
Answer: Of course the notion of statistical independence as has been popularly preached till now, is pretty much a myth (most of it). I don't think so anybody should disagree that the universe and everything inside it, works in conjunction with everything else.
In fact as far as statistical independence goes, its only in the datasets or kind of, it exists in very specific terms. But in general, dependency is an integral part of the universe.
Comment: Yes, but much of it is negligible in terms of our tools of measurement.
Comment: Exactly. Perhaps thats why we treat such variables as independent since their dependence is negligible.


Find integral, what's wrong with these 2 solution?

Question: I am meeting an interesting problem and solved two different method.
Could you see following 2 solution, and find errors?

QUESTION.
  $$    f(x)=2x^3 -2x^2 - \int_{0}^{x+1}f(t)dt  $$
where $f(x)$ is symmetric about the point $(1,0)$
and differentiable at all point.
Find $$    \int_{2}^{4}f(x)dx  $$ .

$$ $$
Sol.1
By the SYMMETRIC about $(1,0)$, we get $f(1)=0$ $  \    \  $$    $     $\cdots(a)$
From $    f(x)=2x^3 -2x^2 - \int_{0}^{x+1}f(t)dt  $
$\Longrightarrow$  $f'(x)=6x^2 - 4x -f(x+1)$
$\Longrightarrow$ $f'(0)=-f(1)=0 $ $  \    \  $ (by (a).)
$ $
By the SYMMETRIC about $(1,0)$,  $f'(0)=f'(2)$.
$\Longrightarrow$ $0=f'(0)=f'(2)=6  \cdot2^2 -4\cdot2-f(3)=24-8-f(3)=16-f(3)  $
Thus $f(3)=16$.
$ $
From $   f(x)=2x^3 -2x^2 - \int_{0}^{x+1}f(t)dt  $
$$\begin{align}
&  \Longrightarrow  16=f(3)=2\cdot 3^3-2\cdot3^2-\int_{0}^{4}f(t)dt =36-\int_{0}^{4}f(t)dt \\
&  \Longrightarrow  20=\int_{0}^{4}f(t)dt\\
\end{align}$$
$$$$
$0=f(1)=2\cdot1^3-2\cdot1^2  -   \int_{0}^{2}f(t)dt $
$ \Longrightarrow 0= \int_{0}^{2}f(t)dt $
$$$$
SO,
$$ \int_{2}^{4}f(x)dx = \int_{0}^{4}f(x)dx  -\int_{0}^{2}f(x)dx  = 20-0=20 $$
$$$$$$$$

NOW, 2nd solution

$$$$
 Sol.2
$ \int_{0}^{2}f(t)dt = 0 $.
$ f(-1)=-4$.
By the SYMMETRIC about $(1,0)$, $  \   $   $f(3)=-f(-1)$.
So,
$$\begin{align}
 & 4 = -f(-1)=f(3)=2\cdot3^3-2\cdot3^2-\int_{0}^{4}f(t)dt\\
&  \Longrightarrow  32=\int_{0}^{4}f(t)dt\\
\end{align}$$
$$$$
Thus, $$ \int_{2}^{4}f(x)dx = \int_{0}^{4}f(x)dx  -\int_{0}^{2}f(x)dx  = 32-0=32 $$
$$$$ $$$$

Sol.1 and Sol.2 say different things.
What mistake led me this result?

$$$$
Thanks in advance.
$$$$
$\color{red}{\text{----------- **EDIT** ----- **Please read below, thank you** ---------------------------------}}$
$$$$
P.S.1
 $$$$  Let me 'darely' DEFINE symmetric about point $(a,b)$
$$$$
If $f$ is symmetric about point $(a,b)$
if and only if
$f(x)+f(2a-x)=2b$
if and only if
$f(a-x)+f(a+x)=2b$
$$$$
P.S.2
$$$$
By P.S.1
$f'(x)=f'(2-x)$
So, $f'(0)=f'(2)$
$$$$
Comment: I didn't read all of it but at first you say $f(1) = 0$. But $f(-1) = - 2-2 = -4$ and not $f(-1) = 0$
Comment: @AndréSchemaitat: $f(-1) = -4$ is valid, what he means by saying that "$f$ is symmetric about the point $(1; 0$)" is that $(1; 0)$ is the point of symmetry of the graph of $f$ (not $(0; 0)$ as you might've thought).
Comment: I don't know but $f$ seems to be ill-defined, i.e. there's no such $f$, and hence you can deduce many things from this ill-defined $f$.
Comment: @user143993 Can you please explain exactly what you mean by "$f(x)$ is symmetric about the point $(1,0)$."
Comment: @cpiegore $$$$  Let me 'darely' DEFINE symmetric about point $(a,b)$
$$$$

If $f$ is symmetric about point $(a,b)$
$$$$
if and only if
$$$$
$f(x)+f(2a-x)=2b$
$$$$if and only if
$$$$
$f(a-x)+f(a+x)=2b\\$$$$$
Comment: Okay. Then just ignore my comment.
Comment: @user143993 I think you are correct since your definition includes odd functions as a special case.
Answer: Edit 2: Ok,first I apologize for what I wrote.. It seems that I understimated the problem.. Here is what I've done:
The simetry with respect to $(1,0)$ traduces in $$f(x+1)=-f(-x+1) $$(#). Just replacing in the equation that defines $f(x)$(call it equation (1)) we get $f(-1)=-4$. By the simmetry, $f(3)=4$, too. Replacing in (1), we get $\int_{0}^{4} f(t) dt =32$. Now, using that $f$ is differentiable in all of R, we can use the chain rule in (#), giving $$f'(x+1)=f'(-x+1)$$. Deriving (1), we get $$f'(x)=6x^2-4x-f(x+1)$$. Evaluating in $2$, we get $f'(2)=12=f'(0)=-f(1)=\int_{0}^{2} f(t) dt$.
Finally,$$\int_{2}^{4} f(t) dt=\int_{0}^{4} f(t) dt-\int_{0}^{2} f(t) dt=32-12=20$$.
Concerning your solutions: About the first, I don't know where do you get from that $f(1)=0$. In the second I don't get $\int_{0}^{2} f(t) dt=0$. I think you are asuming that $(1,0)$ belongs to the graph of $f$, but it's not necessary.
Comment: HOW do we know $f(3) = f(-1)$ or $f(1) = 0$ or anything else about the function $f$?
Comment: As far as I can see the ONLY value that we know with any certainty is $f(-1) = -4$.
Comment: @cpiegore: Because the OP stated that $(1; 0)$ is the point of symmetry of the graph of $f$.
Comment: @Diego Can you explain how "$f(x)$ is symmetric about the point $(1,0)$" implies $f(1)=0$ because I really don't get it.
Comment: @cpiegore I think $$ $$ $f$ is symmetric about point $(a,b)$
implies
$f(x)+f(2a-x)=2b$ or $f(a-x)+f(a+x)=2b$
$$$$
So, $f(1-x)+f(1+x)=0$
Thus, $f(3)=-f(-1)$ $$$$
Comment: @Diego
I don't think $f(x)=x^2$ is proper example. $ $ Because $f(x)=x^2$ isn't symmetric at any point.
$ $
It is just symmetric y-axis.
Comment: @Diego  I added my question for your opinions, please check my question bottom (P.S.). Really thank you
Comment: @Diego I think...$$ $$ $f$ is symmetric about $(a,b)$ and differentiable $$$$ implies $$$$ $f'$ is symmetric about $x=a$ (the line)$$$$(Please check bottom of my question.)$$$$
Comment: Thanks for your answering. You wrote $f(x+1)=-f(-x+1)$. NOW insert $x=0$, we get $f(1)=-f(1)$. SO $f(1)=0$
Comment: And then insert x=1 at equation (1), we get `integral from 0 to 2` is 0
Comment: $f(x+1)=-f(-x+1)$ should work for every x except 0. So I don't think that f(1) =0 for the reasons you give. Nevertheless, I think that for f to be continuous and simmetric with respect to (1,0), it has to be f(1)=0. That could be the reason why yo get 20 too.
Answer: $f(x)=2x^3 -2x^2 - \int_{0}^{x+1}f(t)dt \text{ solving this for } \int_{0}^{x+1} f(t) dt \\ \int_0^{x+1} f(t) dt=2x^3-2x^2-f(x)  \\ \text{ Evaluating } \int_2^4 f(t) dt= \int_0^4 f(t) dt- \int_0^2 f(t)dt \\ =(2(3)^3-2(3)^2-f(3))-(2(1)^3-2(1)^2-f(1)) \\ =(36-f(3))-(0-f(1))\\ =36-f(3)+f(1) \\ =36-f(3)+0 \text{(since } f(1)=0 \text{ is given } \\ \\ \text{ time to play with } f(3) \\ \int_0^{0} f(t) dt=2(-1)^3-2(-1)^2-f(-1) \implies 0=-2-2-f(-1) \implies f(-1)=-4 \\ \text{ and we know } f(3)=-f(-1) \text{ since } f \text{ is point symmetric about } (1,0)   \\ \text{ so we were almost done earlier and now we can finish } \\ \int_2^4 f(t) dt=36-f(3)+0=36-f(3)=36-4=32\\  $
Comment: If someone already posted this I didn't see this answer very clearly... But I don't think we need any differentiating.
Comment: I give up, I can't find any mistake in your answer, but neither in mine. Anyway, I think yours is more straightforward, and then probably the right one.....
Comment: @randomgirl At first, thanks for your answering. If you think your answer is correct(actually I think you are right!),,, why don't you check Sol.2 in my origin question?
Comment: I just wanna know ERRORS in Sol.1 / Sol.2...
Answer: As far as I can see, there are 2 problems with your solutions.  First $f(x) = 2x^2 - 2x^3 - \int_{0}^{x+1} f(t)dt$ isn't really an algebraic equation, so you cannot expect to perform various algebraic manipulations on it and get a consistent answer.  Second it isn't clear what you mean by "$f(x)$ is symmetric about the point $(1,0)$."
If you differentiate the formula for $f(x)$ you get the differential equation $f'(x) = 6x^2-4x-f(x).$  Its solution is $f(x) = ce^{-x}+6x^2-16x+16$.  Using $f(-1) = ce+38 =-4$ we have $c = -42/e$.  Now we can integrate directly to find that $\int_{2}^{4}f(x)dx = -42e^{-3}+42e^{-5}+48$
Comment: When you differentiate,you should get f(x+1).
Comment: @cpiegore I added my question for your 'second opinion -not clear symmetric- ', please check about.. Really thank you.
Comment: @cpiegore You may as well solve differential equation about $f'(x)=6x^2 - 4x -f(x+1)$ rather than $f'(x) = 6x^2-4x-f(x).$ $$$$ Because $$f(x)=2x^3 -2x^2 - \int_{0}^{x+1}f(t)dt
\
\Longrightarrow
\
f'(x)=6x^2 - 4x -f(x+1)$$
Comment: @cpiegore$$$$ (might sound like stupid thing...) Is there anyting to do with between [not algebraic equation] and [may not performing some algebraic manipulation]? - about 1st problem you thought - $$$$
Comment: @user143993 I only said that because I didn't understand how you were getting $f(1) = 0$ or $f(3) = 16.$ Since you have explained what you meant by symmetry your solutions make more sense now.


Diferenças entre um transpilador e um compilador

Question: Ultimamente se tem ouvido falar muito do termo transpilador (transpiler). Porém, não me parece clara a diferença de um transpiler (Babel, por exemplo) para um compilador (compiler) tradicional. Os dois traduzem código de uma linguagem para outra, não?
Quais as reais diferenças entre um compilador e um transpilador?
Comment: Nossa, é duplicada, e o pessoal sai tacando respostas, sem pesquisar :\
Comment: Algo que pego no pé e o pessoal não entende, não é só o AP quem deve dar uma breve pesquisada pra ver se não é dup, quem vai responder pode muito bem fazer isto, afinal de conta somos uma comunidade, não custa nada ser **um pouquinho altruísta** e colaborar apontando respostas existentes, eu já peguei fama de chato de tanto que falo disso, mas o pessoal fica afobado pra responder. Uma pena mesmo.
Comment: A outra duplicada: [O que é Transpilação?](https://pt.stackoverflow.com/q/189894/18246)
Comment: @GuilhermeNascimento eu acho que é desnecessário, colocar isso da forma como o "Valace" colocou,  como se responder uma duplicada fosse passível de negativo... Quer dizer, não está sendo levada em consideração a boa vontade de quem tá respondendo.Aliás qualquer tipo de comentário, uma vez que o sistema nos oferece a ferramenta de marcar como duplicada.  E me diga uma coisa, onde tá dizendo que antes de responder o AR tem de pesquisar para ver se não há duplicadas(não estou dizendo que é uma boa prática, apenas que responder não é uma má prática ao meu ver)?
Comment: Pesquisa ai e coloca para gente ver... @WallaceMaxters
Comment: Pronto agora, marca como duplicada...@WallaceMaxters
Comment: @MagicHat fica tranquilo :) . Não estou dizendo que é regra, estou falando de sobre colaborar melhor com a comunidade, nada é perfeito, mas não custa nada dar uma ajudinha
Comment: @GuilhermeNascimento pow mais ajuda do que tentar responder... Quando eu vejo possivel dup, ou alguma direção apontando para uma resposta eu considero, mas até as respostas saírem num tinha nada... E daí vem alguém criticando, colocando de forma que tá errado responder, ai eu pego mal. Mas tô tranquilex.... Agora não existe nem no meta ou na central de ajuda, algo que fala sobre isso?
Comment: @MagicHat não é questão de regra ou debates, é só mesmo uma sugestão de como fazer uso da ferramenta, no caso botão `Fechar` > `Duplicada de ...`. Mas é algo que meio que já desisti de levar adiante, tem realmente problemas bem maiores no site, mas que de fato não serão resolvidos também :/
Comment: @GuilhermeNascimento faz parte, preciso pegar o hábito de dar uma pesquisada antes, inconscientemente parto do pressuposto que ao digitar a pergunta o sistema já oferece algo ao AP, então se chega na lista de recentes seria pq não há dup, mas sabemos que não é assim que funfa...
Answer: De maneira clara e objetiva:

Compilador: recebe código-fonte em linguagem de alto/médio nível (mais próximo dos humanos) como entrada e, como saída, produz código em linguagem de baixo nível (mais próximo da máquina). Ex.: C -> código objeto, Java -> JVM bytecode.
Transpilador: recebe código-fonte em linguagem de alto/médio nível como entrada e, como saída, produz código em outra linguagem de alto/médio nível. Ex.: Typescript -> Javascript.
Answer: A grosso modo o transpiler é um tipo de compilador que traduz um código escrito em uma linguagem médio/alto nível para um código de uma outra linguagem de médio/alto nível.
Já o compilador traduz da linguagem médio/alto nível para uma linguagem de nível mais baixo, uma linguagem de montagem, por exemplo.
Fonte


How to find an Xml Element when it's prefixed with a namespace?

Question: I have an Xml File as below:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ea:Stories ea:WWVersion="2.0" xmlns:aic="http://ns.adobe.com/AdobeInCopy/2.0" xmlns:ea="urn:SmartConnection_v3">
<ea:Story ea:GUID="D8BEFD6C-AB31-4B0E-98BF-7348968795E1" pi0="style="50" type="snippet" readerVersion="6.0" featureSet="257" product="8.0(370)" " pi1="SnippetType="InCopyInterchange"">
<ea:StoryInfo>
<ea:SI_EL>headline</ea:SI_EL>
<ea:SI_Words>4</ea:SI_Words>
<ea:SI_Chars>20</ea:SI_Chars>
<ea:SI_Paras>1</ea:SI_Paras>
<ea:SI_Lines>1</ea:SI_Lines>
<ea:SI_Snippet>THIS IS THE HEADLINE</ea:SI_Snippet>
<ea:SI_Version>AB86A3CA-CEBC-49AA-A334-29641B95748D</ea:SI_Version>
</ea:StoryInfo>
</ea:Story>
</ea:Stories>

As you can see all elements have "ea:" which is a namespace prefix.
I'm writing an XSLT file to show the SI_Snippet text which is "THIS IS THE HEADLINE".
How to write the xpath in the XSLT file? Should it contain the namespace or should it be excluded?
//ea:Story[ea:SI_EL='headline']/ea:SI_Snippet or
//Story[SI_EL='headline']/SI_Snippet

Actually both fail in the online tool I used: http://xslt.online-toolz.com/tools/xslt-transformation.php
So there should be another way?
If later, how does it know which namespace to look at? Should I be passing the namespace to the XslTransformer at runtime?
Answer: You should declare the namespace in your XSLT and then use the prefix that you give it:
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:ea="urn:SmartConnection_v3">
    <xsl:template match="/">
       <xsl:value-of select="//ea:Story[ea:SI_EL='headline']/ea:SI_Snippet" />
    </xsl:template>

</xsl:stylesheet>

Note the xmlns:ea="urn:SmartConnection_v3" in the root element.  This is important.
Answer: Try to use XDocument?
var xml = @"<?xml version=""1.0"" encoding=""UTF-8"" standalone=""yes""?>
<ea:Stories ea:WWVersion=""2.0"" xmlns:aic=""http://ns.adobe.com/AdobeInCopy/2.0"" xmlns:ea=""urn:SmartConnection_v3"">
<ea:Story ea:GUID=""D8BEFD6C-AB31-4B0E-98BF-7348968795E1"" pi0=""style="50" type="snippet" readerVersion="6.0" featureSet="257" product="8.0(370)" "" pi1=""SnippetType="InCopyInterchange""">
<ea:StoryInfo>
<ea:SI_EL>headline</ea:SI_EL>
<ea:SI_Words>4</ea:SI_Words>
<ea:SI_Chars>20</ea:SI_Chars>
<ea:SI_Paras>1</ea:SI_Paras>
<ea:SI_Lines>1</ea:SI_Lines>
<ea:SI_Snippet>THIS IS THE HEADLINE</ea:SI_Snippet>
<ea:SI_Version>AB86A3CA-CEBC-49AA-A334-29641B95748D</ea:SI_Version>
</ea:StoryInfo>
</ea:Story>
</ea:Stories>";

XDocument xdoc = XDocument.Parse(xml.ToString());
XElement v = xdoc.Descendants().FirstOrDefault(x => x.Name.LocalName == "SI_Snippet");

EDIT
XPathNavigator navigator = xmldDoc.CreateNavigator();
XmlNamespaceManager ns = new XmlNamespaceManager(navigator.NameTable);
ns.AddNamespace("ea", "urn:SmartConnection_v3");
var v = xmlDoc.SelectSingleNode("//ea:SI_Snippet", ns);

Comment: I understand if you'll want to scream! but I'm using .NET 1.1 for this project so XDocument doesn't exist and need to use XPaths in XSLT.
Comment: @TheLight well according to the OP my answer works ;) Other than that, try my edit?


Authenticating username and password against LDAP using ADODB.Connection

Question: I have the following code:
<code>object recs;
ADODB.Connection conn = new ADODB.Connection();
ADODB.Recordset rs = new ADODB.Recordset();

conn.Provider = "ADSDSOObject";

string SQLStmt = "SELECT cn FROM 'LDAP://" + server + "/o=nwu/ou=users' WHERE cn='" + userName + "'";
conn.Open("ADs Provider", "****", "****");

rs = conn.Execute(SQLStmt,out recs, 0);
</code>
This reads the information for the username.
How do I actually check if the password matches this user?
Answer: Your application must authenticate the connection, that is, set the authorization state of the connection using the LDAP bind request.
Comment: Could you perhaps point me in the right direction? An article or something demonstrating this?
Answer: You can find the answer to your question and many more details in the MSDN website.
Read this: Building Secure ASP.NET Applications: Authentication, Authorization, and Secure Communication


Limit of a sequence of fixed points also a fixed point?

Question: Suppose I have a continuous function $f : [0,1]^n \rightarrow [0,1]^n$ (maybe $n$ is infinite). Suppose I have a sequence $\{a_n\}_{n=1}^\infty$ of points in $[0,1]^n$ where each $a_n$ is a fixed point of $f$, and that $\{a_n\}$ converges to a value $a$. Is $a$ also a fixed point of $f$?
Thank you!
Answer: $$f(a)=f\left(\lim\limits_{n\to\infty}a_n\right)=\lim\limits_{n\to\infty} f(a_n)=\lim\limits_{n\to\infty} a_n=a$$
Answer: Consider the function $g:[0,1]^n\to\mathbb{R}^n$, $g(x)=f(x)-x$. It is continuous, $g(a_n)=f(a_n)-a_n=0$ for all $n\in\mathbb{N}$, therefore $0=\lim_{n\to\infty} g(a_n)= \lim_{n\to\infty}f(a_n) - \lim_{n\to\infty}a_n = f(a)-a$, i.e. $a$ is also a fixed point of $f$.
Answer: This follows as $\{x\in [0, 1]^n ; f(x)=x\}$ is closed


can you use mutual information to determine how much one variable tells you about another

Question: I wrote a Naive Bayes Classifier and want to be able to test how much information one variable gives about another.  The idea is to use the ones that are the most orthogonal and avoid using highly redundant variables.
My idea was to use the mutual information, but have found that it is not as clear cut as I first thought.  For example, if my variables are X = { 1, 2, 3, 4 } and Y = { 9, 10, 11, 12 }, then the mutual information would seem to be 2.  But if X = { 1, 2, 3, 4, 5, 6, 7, 8 } and Y = { 9, 10, 11, 12, 13, 14, 15, 16 }, then the mutual information is 3.  In both cases, however, if these numbers are lists (like columns in a database or spreadsheet), then X = 1 will always be paired with Y = 9 etc., meaning knowing one will always tell you the value of the other.  So if both cases have a 1 to 1 mapping between X and Y and the mutual information is different, how useful is the mutual information in determining how much you know about one variable by knowing the other?
Would some quantity like $$\frac{H(X,Y)}{H(X) + H(Y)}$$ or $$\frac{H(X,Y)}{I(X,Y)}$$ work better?  Does anyone have experience with determining how much one variable will tell you about another?
Here is a summary of the two examples.
Example 1

X = { 1, 2, 3, 4 }
Y = { 9, 10, 11, 12 }
H(X) = 2
H(Y) = 2
H(X,Y) = 2
I(X,Y) = 2
H(X,Y) / (H(X) + H(Y)) = 0.5
H(X,Y) / I(X,Y) = 1.0

Example 2

X = { 1, 2, 3, 4, 5, 6, 7, 8 }
Y = { 9, 10, 11, 12, 13, 14, 15, 16 }
H(X) = 3
H(Y) = 3
H(X,Y) = 3
I(X,Y) = 3
H(X,Y) / (H(X) + H(Y)) = 0.5

H(X,Y) / I(X,Y) = 1.0
Comment: Article by Reshef might answer this question "Quite a lot, really." See Reshef, D., Reshef, Y., Finucane, H., Grossman, S., McVean, G., Turnbaugh, P., Lander, E., Mitzenmacher, M., and Sabeti, P. (2011). Detecting novel associations in large data sets. *Science*, 334(6062):1518–1524.
Answer: Yes, you can use mutual information to determine how much one variable tells you about another – as a matter of fact, that's the exact purpose of mutual information.
The reason for the different results in the two examples is that in the second example you have more possible values for each variable, which means there is more information to possibly gain about it.
I assume you used the base-2 logarithm, such that information is measured in bits. In example 1 there are 4 possible values, each occurring equally likely, which means knowing the value of $X$ or of $Y$ constitutes an amount of information of $\log_2 4 = 2$ bits. Because there is a one-to-one-correspondence, knowing the value of $X$ means exactly knowing the value of $Y$ (and vice versa), which means the value of $X$ does not only constitute 2 bits about $X$, but also 2 bits about $Y$ (and vice versa).
In example 2 there are 8 possible values, which means knowing $X$ or $Y$ constitutes $\log_2 8 = 3$ bits, and again there is a one-to-one correspondence, so that knowing $X$ tells you 3 bits about $Y$ (and vice versa).
The formal MI result therefore makes perfect sense.
Answer: Relative entropy (or Kullback-Leibler divergence ) is the appropriate quantity to measure how much one variable tells about another one. Relative entropy is, unlike the mutual information, unsymmetrical. That means, it is possible that knowing a variable A tells everything about another variable B, even knowing B does not tell anything about A.
Comment: If you know one's SIN (social insurance number), you will know the persons gender (at birth at least). If you know one's gender, you basically know nothing about the person's SIN.
Comment: Well, "basically nothing" is not the same as "not anything". Let's simplify matters and assume that the number of issued SINs is the same for both genders, say $n$. In that case the SIN tells you the value of the variable "gender", which has two equally likely values, so you get 1 bit of information. The other way round, knowing the gender reduces the number of possible SINs from $2n$ to $n$ – again an amount of information of 1 bit.
Comment: Of course the total amount of information in a SIN is $\log 2n$, which is much more, so relatively speaking you gain "basically no" information. In absolute terms though you gain exactly the same amount either way, 1 bit.
Comment: Excellent analysis. You are right, MI is the right the measure for mutual information, not the KL divergence.
Comment: Kullback–Leibler divergence only tells you how different the distributions of two variables are. It has no relation to their joint distribution, only their separate (marginal) distributions, so it is unable to detect any structured relation between the variables which is the basis of one telling something about the other. The correct measure of information between variables is mutual information. It is related to KL divergence insofern as it is the divergence between the joint distribution of two variables and the product of the marginal distributions.
Comment: "it is possible that knowing a variable A tells everything about another variable B, even knowing B does not tell anything about A": I don't believe that is actually possible. Can you give an example?


Best practices to messages feedback after POST in ASP.NET MVC

Question: I'm tring to find a good approach to feedback messages after a POST.
for example, I have these methods
    public ActionResult Index(int id)
    {
        Model model = getModel(id);

        return View(model);
    }

    [HttpPost]
    public ActionResult Save(Model model)
    {

        Result result = saveModel(result)

        if (Result.Status != Status.SUCCESS)
        {
            ...
        }
        else if(Result.Status != Status.FAILURE)
        {
            ...
        }else
        {
            ...
        }
        return RedirectToAction("Index");
    }

and Result class has a message property
        public class Result
        {
           ...
           public string Message{get;set;}
        }

So, When I call .../Controller/Index/1 my index view is shown and this view has an submit button that call Save Action, but if something goes wrong on business layer, I would like to present error/warning message.
so, the doult is What is the best approach to solve this situation?
I have read some article that recommend TempData, others recommend ViewData, to transfer the message to Index View.
Comment: If you want to persist the data between across the request, you must use TempData. ViewBag wont work if you are redirecting to another action...
Comment: `TempData` for redirects, `ViewBag` for same page.
Answer: Let me share an example of a ChangePassword action of one project that I developed:
[HttpPost]
public ActionResult ChangePassword(LoginModel model)
{
    if (ModelState.IsValid && _userService.ChangePassword(model.Password, model.NewPassword))
        ViewBag.SuccessMessage = UI.PasswordChanged;
    else
        ModelState.AddModelError("Password", ErrorMessages.InvalidPassword);
    return View(model);
}

Just remember that ViewBag wont work if you are redirecting to another action. You must use TempData if you want to persist the information across one request.
Answer: For failures you should use
ModelState.AddModelError("", "The error message");

since it will automatically be picked up by Html.ValidationSummary(true) (which is automatically added to all views)
For notifications you can use either TempData or the ViewBag.
I however discourage you from redirecting for everything. Instead give the user a chance to correct any errors (even if it's the business layer which generates them). Use something like:
[HttpPost]
public ActionResult Save(Model model)
{
    if (!ModelState.IsValid)
        return View(model);

    Result result = saveModel(result)
    if (Result.Status == Status.SUCCESS)
        return RedirectToAction("Index");

    if (Result.Status != Status.FAILURE)
    {
        ModelState.AddModelError("", "Ooops, failed");
    }
    else
    {
        ModelState.AddModelError("", "Some other error");
    }

    return View(model)
}