the correct way to input images for testing the tensorflow model

Question: I have been trying to test the fcn implementation posted here. The only thing I changed is the way for setting up the input images for test against the model. My modification is marked with red curve in the following figure.

However, running the program caused the following error message TypeError: The value of a feed cannot be a tf.Tensor object. Acceptable feed values include Python scalars, strings, lists, or numpy ndarrays.
, happened at down, up = sess.run(tensors, feed_dict=feed_dict). I am curious to know what are something wrong in my implementation, and how to modify it.  In the original post, the author uses img1 = skimage.io.imread("./test_data/tabby_cat.png")
  to input the image.
If I change batch_images=tf.expand_dims(images,0) to batch_images=tf.expand_dims(img1,0)  the program will output the following error messages.
Answer: As the error indicates, the types of values that you can use as feeds are Python scalars, strings, lists or numpy arrays. What you are trying to use as feed is img1, the output of tf.image.decode_png, which is of type tf.Tensor. Hence the error. You have two options :
1) Convert img1 to a numpy array before feeding it. You can do that by simply evaluating img1 as follows :
feed_dict = {images:img1.eval()}

2) Use img1 itself as the input to the rest of the model. You can do that as follows :
batch_images = tf.expand_dims(img1, 0)

Comment: Hi, Keveman, thanks for the reply. I tried the first approach previously, but the program just stop at line of  feed_dict = {images:img1.eval()} forever without moving forward or exit with some error messages. It just stopped there.

I just tried the second approach, the result is included in the original one. Looks like model building process does not even go through.
Comment: For the second approach, try `batch_images = tf.expand_dims(tf.cast(img1, tf.float32), 0)`
Comment: Hi keveman, thanks for your suggestion, I tried it, then the error message o f "TypeError: The value of a feed cannot be a tf.Tensor object. Acceptable feed values include Python scalars, strings, lists, or numpy ndarrays.
" appears again.


Clarification on the Math of Logistic Regression

Question: I am trying to get a better understanding of the math behind logistic regression.
Logistic regression is looking to give a prediction based on data likelihood.
From my understanding it is making a posterior prediction.
Suppose we have a two-class model. So this becomes the equation
\begin{align}
p(C_1|x) &= \frac{p(x|C_1)p(C_1)}{p(x|C_1)p(C_1)+p(x|C_2)p(C_2)}\\
&=\frac{1}{1+\exp(-a)} = \sigma(a)
\end{align}
where $a= \ln \frac{p(x|C_1)p(C_1)}{p(x|C_2)p(C_2)}$
My questions are what does $a= \ln \frac{p(x|C_1)p(C_1)}{p(x|C_2)p(C_2)}$ intuitively mean? Does it imply that the $\theta x$ of our model, is trying to estimate the ratio in $a$? And why in the first step is the equation equivalent to the same as the naive bayes classifier; how does logistic regressor distinguish itself from the naive bayes if the background math is the same
Comment: Try substituting $a$ in the expression for $\sigma(a)$.  You will see it's just definitional given the first line, there isn't any intuition in it except as a transform of the first expression.  The logistic regressor is equivalent to this by definition; we could use other expressions (see probit regression) for an expression.  It gets its name from the Logistic distribution, the standardized form of which has cumulative density function equal to $1/(1+\exp(-x))$.
Answer: Basic workflow of logistic regression is:

Calculate log-odds of success (success means that your observation belongs
to the target class of interest)
Convert log odds to percentage likelihood
Apply some threshold to your percentage likelihood in order to apply a
predicted class label of "success" or "failure"

Your "a" term are known as the log-odds.  These are the log of the odds that a sample belongs to a given class given that it possesses a set of predictors, to the odds that it does not belong to a certain class given its predictors.  Think of it as the raw output from a logistic regression method in your native statistical software package (although sometimes you could also be given a probability, or even the class label directly, the log odds are driving all of these secondary outputs).
We cannot directly apply a linear regression to a classification problem since it violates the assumption that the response variables be continuous (i.e. exists between - infinity and + infinity).  What we can say is that the odds in favor of an event occurring (in this case that your observation belongs to a class of interest), are positive and (usually) non-zero numbers.  This means you can also calculate the log of those odds (the log odds from earlier).  The beauty of the log odds is that they are continuous values that we can apply linear regression to.
The remainder of the work comes from finding the weights of your coefficients that maximize the probability of an observation belonging to the success class if it is indeed a success, and minimize the probability that are assigned to non-successes.  This is controlled through the coefficient weights.
If you exponentiate the log-odds (your 'a' term) it is straightforward to calculate the probability.
<code>           Probability = (odds)/(odds+1)
</code>
Given that we have log-odds:
<code>           Probability = exp(a)/(exp(a)+1)
</code>
Divide out the exp(a) term and the probability statement reduces to your
<code>            sigma(a) = 1/ 1+exp(-a)
</code>
Comment: Hi thanks for your answer. So basically the log odds is the unoptimized version of the logistic regression?
Comment: It's not that log-odds are "unoptimized",  they just need to be converted to a probability and then to a class prediction.
Comment: makes sense, but what is the relation of the log odds to our weights?
Comment: Your initial intuition was close to correct, except the theta*X terms are not just estimating the ratio:  They are estimating the log of that ratio.  The thetas are the weights, and the choice of weights will generate different values of log-odds for each observation depending on the value of the predictors associated with an observation.  Larger value of log odds translates to higher probability of target class membership.
Comment: Hi thanks for the reply.

We could write the same equation on the posterior probability using the naive bayes classifier, correct? Does estimating the log likelihoood with an optimized theta, using gradient descent, differentiate this from the naive bayes model as this will not double count correlated features..


Outlook Add-in to always run

Question: I am trying to create an add-in which runs all the time even when outlook is closed or the user is using OWA and needs to be able to act off the data in the headers of incoming emails. I have already created the VSTO (in c#) for this. However, obviously this stops when outlook is closed. Is it possible to use the code for an add-in that runs continuously? And is the best Add-in for this the outlook web add-in?
Answer: By it definition, and addin is only running when its host app is running. That aplies to both COM and web addins. If you need something running continously, you need to create a standalone app. It can either use the Outlook Object Model (make sure the profile it uses is online rather than cached) or EWS.
Is your problem that NewMailEx event does not fire on messages received while your VSTO addin was not active? In that case you can check on startup for the unread messages in the Inbox folder (MAPIFolder.Items.Restrict)) and process Items.ItemAdd event (it will fire when the cached profile synchronizes server emails to the cached Inbox folder).
Comment: Thanks, thought that would be the case but hoped it wouldn't be.

Outlook Add-in to always run


nginx rate limit based on location & header of post

Question: Nginx has this useful module called ngx_http_limit_req_module
where you can limit requests to the server based on IP or number of requests.
Is it at all possible to rate limit the location that includes the custom header? e.g.  I tried this but it fails to start nginx.
location /myservice {
    if ($http_x_custom_header) {
      limit_req zone=one burst=5;
    }
}

The specific error is:

nginx: [emerg] "limit_req" directive is not allowed here in
  /etc/nginx/sites-enabled/default

i.e. it doesn't like the limi_req directive inside the if block.  It's fine outside the if block.
I may be able to live with different set of limit parameters outside the location though.
Answer: You are correct, limit_req can not be used at that level.
The way to get around that is as follows.
error_page 418 = @limitreq;

location /myservice {
    if ($http_x_custom_header) {
      return 418;
    }
}

location @limitreq {
    limit_req zone=one burst=5;
    <rest of config>
    }

This will return 418 if the custom_header is present, which nginx will intercept and send to the custom location block, which will the apply the limit_req
Comment: Awesome! works perfectly.


How would I investigate the convergence of the following series

Question: $$\frac{1}{2}-\frac{2}{20}+\frac{3}{38}-\frac{4}{56}+\frac{5}{74}$$
Denominators increase by 18. That's most far I've gotten. Couldn't apply the diritchlet test either - any hints?
Comment: So, you're looking at $\sum_{n=0}^\infty(-1)^n\frac{n+1}{18n+2}$?  Shouldn't the $n$-th term not converge to zero?
Comment: As far as it seems $a_n \not\to 0$, thus the sum diverges.
Answer: Because of $$\lim_{n\rightarrow\infty} \frac{n+1}{18n+2}=\frac{1}{18}\ne 0$$ , the alternating sum is not converging. I am not sure whether it diverges to $\infty$ or whether it oscillates.
Comment: The sum of each pair leaves a remainder that is $O(1/n)$ so the partial sum to $N$ diverges at $O(\log(N))$.
Comment: Since $(n+1)/(18n+2)>1/36$ for all but finitely many $n$, the sum diverges to $\infty$.If the terms converge to a positive limit $L$ then only finitely many of the terms are less than $L/2$...... In general, the sum of an infinite sequence of non-negative terms either converges or goes to $\infty.$


Confidence intervals on predictions for a non-linear mixed model (nlme)

Question: I would like to obtain 95% confidence intervals on the predictions of a non-linear mixed nlme model. As nothing standard is provided to do this within nlme, I was wondering if it is correct to use the method of "population prediction intervals", as outlined in Ben Bolker's book chapter in the context of models fit with maximum likelihood, based on the idea of resampling fixed effect parameters based on the variance-covariance matrix of the fitted model, simulating predictions based on this, and then taking the 95% percentiles of these predictions to get the 95% confidence intervals?
The code to do this looks as follows :
(I here use the 'Loblolly' data from the nlme help file)
library(effects)
library(nlme)
library(MASS)

fm1 <- nlme(height ~ SSasymp(age, Asym, R0, lrc),
    data = Loblolly,
    fixed = Asym + R0 + lrc ~ 1,
    random = Asym ~ 1,
    start = c(Asym = 103, R0 = -8.5, lrc = -3.3))

xvals=seq(min(Loblolly$age),max(Loblolly$age),length.out=100)
nresamp=1000
pars.picked = mvrnorm(nresamp, mu = fixef(fm1), Sigma = vcov(fm1)) # pick new parameter values by sampling from multivariate normal distribution based on fit
yvals = matrix(0, nrow = nresamp, ncol = length(xvals))

for (i in 1:nresamp)
{
    yvals[i,] = sapply(xvals,function (x) SSasymp(x,pars.picked[i,1], pars.picked[i,2], pars.picked[i,3]))
}

quant = function(col) quantile(col, c(0.025,0.975)) # 95% percentiles
conflims = apply(yvals,2,quant) # 95% confidence intervals

Now that I have my confidence limits I create a graph:
meany = sapply(xvals,function (x) SSasymp(x,fixef(fm1)[[1]], fixef(fm1)[[2]], fixef(fm1)[[3]]))

par(cex.axis = 2.0, cex.lab=2.0)
plot(0, type='n', xlim=c(3,25), ylim=c(0,65), axes=F, xlab="age", ylab="height");
axis(1, at=c(3,1:5 * 5), labels=c(3,1:5 * 5))
axis(2, at=0:6 * 10, labels=0:6 * 10)

for(i in 1:14)
{
    data = subset(Loblolly, Loblolly$Seed == unique(Loblolly$Seed)[i])
    lines(data$age, data$height, col = "red", lty=3)
}

lines(xvals,meany, lwd=3)
lines(xvals,conflims[1,])
lines(xvals,conflims[2,])

Here's the plot with the 95% confidence intervals obtained this way:

Is this approach valid, or are there any other or better approaches to calculate 95% confidence intervals on the predictions of a nonlinear mixed model? I am not entirely sure of how to deal with the random effect stucture of model... Should one average perhaps over random effect levels? Or would it be OK to have confidence intervals for an average subject, which would seem to be closer to what I have now?
Comment: @Roland Not sure the normality assumption for parameters in linear models would more likely be satisfied than those in nonlinear models. Would there be any theoretical rationale to believe that to be the case?
Comment: There isn't a question here.  Please be clear about what you are asking.
Comment: I tried to formulate the question more precisely now...
Comment: As I've commented when you asked this previously on Stack Overflow, I'm not convinced a normality assumption for non-linear parameters is justified.
Comment: I haven't read Ben's book, but he doesn't seem to refer to mixed models in this chapter. Maybe you should clarify this when referencing his book.
Comment: Yes this was in the context of maximum likelihood models, but idea should be the same... I've clarified it now...
Answer: What you've done here looks reasonable. The short answer is that for the most part the issues of predicting confidence intervals from mixed models and from nonlinear models are more or less orthogonal, that is, you need to worry about both sets of problems, but they don't (that I know of) interact in any strange ways.

Mixed model issues: are you trying to predict at the population or the group level? How do you account for variability in the random-effects parameters? Are you conditioning on the group-level observations or not?
Nonlinear model issues: is the sampling distribution of the parameters Normal? How do I account for nonlinearity when propagating error?

Throughout, I will assume you're predicting at the population level and constructing confidence intervals as the population level - in other words you're trying to plot the predicted values of a typical group, and not including the among-group variation in your confidence intervals. This simplifies the mixed-model issues. The following plots compare three approaches (see below for code dump):

population prediction intervals: this is the approach you tried above. It assumes the model is correct and that the sampling distributions of the fixed-effect parameters are multivariate Normal; it also ignores uncertainty in the random-effects parameters
bootstrapping: I implemented hierarchical bootstrapping; we resample both at the level of groups and within groups. The within-group sampling samples the residuals and adds them back to the predictions. This approach makes the fewest assumptions.
delta method: this assumes both multivariate Normality of sampling distributions and that the nonlinearity is weak enough to allow a second-order approximation.

We could also do parametric bootstrapping ...
Here are the CIs plotted along with the data ...

... but we can hardly see the differences.
Zooming in by subtracting off the predicted values (red=bootstrap, blue=PPI, cyan=delta method)

In this case the bootstrap intervals are actually narrowest (e.g. presumably the sampling distributions of the parameters are actually slightly thinner-tailed than Normal), while the PPI and delta-method intervals are very similar to each other.
library(nlme)
library(MASS)

fm1 <- nlme(height ~ SSasymp(age, Asym, R0, lrc),
            data = Loblolly,
            fixed = Asym + R0 + lrc ~ 1,
            random = Asym ~ 1,
            start = c(Asym = 103, R0 = -8.5, lrc = -3.3))

xvals <-  with(Loblolly,seq(min(age),max(age),length.out=100))
nresamp <- 1000
## pick new parameter values by sampling from multivariate normal distribution based on fit
pars.picked <- mvrnorm(nresamp, mu = fixef(fm1), Sigma = vcov(fm1))

## predicted values: useful below
pframe <- with(Loblolly,data.frame(age=xvals))
pframe$height <- predict(fm1,newdata=pframe,level=0)

## utility function
get_CI <- function(y,pref="") {
    r1 <- t(apply(y,1,quantile,c(0.025,0.975)))
    setNames(as.data.frame(r1),paste0(pref,c("lwr","upr")))
}

set.seed(101)
yvals <- apply(pars.picked,1,
               function(x) { SSasymp(xvals,x[1], x[2], x[3]) }
)
c1 <- get_CI(yvals)

## bootstrapping
sampfun <- function(fitted,data,idvar="Seed") {
    pp <- predict(fitted,levels=1)
    rr <- residuals(fitted)
    dd <- data.frame(data,pred=pp,res=rr)
    ## sample groups with replacement
    iv <- levels(data[[idvar]])
    bsamp1 <- sample(iv,size=length(iv),replace=TRUE)
    bsamp2 <- lapply(bsamp1,
        function(x) {
        ## within groups, sample *residuals* with replacement
        ddb <- dd[dd[[idvar]]==x,]
        ## bootstrapped response = pred + bootstrapped residual
        ddb$height <- ddb$pred +
            sample(ddb$res,size=nrow(ddb),replace=TRUE)
        return(ddb)
    })
    res <- do.call(rbind,bsamp2)  ## collect results
    if (is(data,"groupedData"))
        res <- groupedData(res,formula=formula(data))
    return(res)
}

pfun <- function(fm) {
    predict(fm,newdata=pframe,level=0)
}

set.seed(101)
yvals2 <- replicate(nresamp,
                    pfun(update(fm1,data=sampfun(fm1,Loblolly,"Seed"))))
c2 <- get_CI(yvals2,"boot_")

## delta method
ss0 <- with(as.list(fixef(fm1)),SSasymp(xvals,Asym,R0,lrc))
gg <- attr(ss0,"gradient")
V <- vcov(fm1)
delta_sd <- sqrt(diag(gg %*% V %*% t(gg)))
c3 <- with(pframe,data.frame(delta_lwr=height-1.96*delta_sd,
                             delta_upr=height+1.96*delta_sd))

pframe <- data.frame(pframe,c1,c2,c3)

library(ggplot2); theme_set(theme_bw())
ggplot(Loblolly,aes(age,height))+
    geom_line(alpha=0.2,aes(group=Seed))+
    geom_line(data=pframe,col="red")+
    geom_ribbon(data=pframe,aes(ymin=lwr,ymax=upr),colour=NA,alpha=0.3,
                fill="blue")+
    geom_ribbon(data=pframe,aes(ymin=boot_lwr,ymax=boot_upr),
                colour=NA,alpha=0.3,
                fill="red")+
    geom_ribbon(data=pframe,aes(ymin=delta_lwr,ymax=delta_upr),
                colour=NA,alpha=0.3,
                fill="cyan")

ggplot(Loblolly,aes(age))+
    geom_hline(yintercept=0,lty=2)+
    geom_ribbon(data=pframe,aes(ymin=lwr-height,ymax=upr-height),
                colour="blue",
                fill=NA)+
    geom_ribbon(data=pframe,aes(ymin=boot_lwr-height,ymax=boot_upr-height),
                colour="red",
                fill=NA)+
    geom_ribbon(data=pframe,aes(ymin=delta_lwr-height,ymax=delta_upr-height),
                colour="cyan",
                fill=NA)

Comment: So if I understand correctly this would be the confidence intervals on a typical group. Would you also have any idea how one would include the among-group variation in your confidence intervals? Should one average over the random effect levels then?
Comment: @ben-bolker, I'm using the bootstrap approach you outline here in a paper, and would like to cite you. Do you have a paper or book where this approach is described?
Comment: Not offhand: if I wanted a citation for this I would have to do a literature search myself.
Comment: @BenBolker, the method seems to be called "Nonparametric case bootstrap coupled with individual residual bootstrap" according to this paper. https://doi.org/10.1002/pst.1561
They do not seem to provide an original reference for it, but I'll cite that paper and mention this StackExchange answer in the supplement.


Finding the Aspect Ratio for Stacked Boxes

Question: I am trying to figure out how to calculate the ideal aspect ratio for a collection of boxes. All of the boxes always maintain the same aspect ratio, but can be scaled to fit the following layouts:

larger box on top, two smaller boxes taking up half of the width of the larger box arranged side-by-side beneath.
larger box to one side, two smaller boxes each taking up half the height of the larger stacked on the other side.

The total width is an unknown, but constant number for both layouts. The total height can vary.
example: https://i.stack.imgur.com/mNLJW.png
Any help would be appreciated!
Answer: There is no "ideal" aspect ratio as the layouts can always be achieved.
If the aspect ratio is $a:b$ and the width is $w$, then in the first layout the big box is $w\times \frac{bw}{a}$ and the small boxes are $\frac w2\times \frac{bw}{2a}$, so the height is $\frac{3bw}{2a}$. In the second layout  the big box is $\frac{2w}3\times \frac{2bw}{3a}$ and the small boxes are $\frac{w}3\times \frac{bw}{3a}$, so the height is $\frac{2bw}{3a}$.
Comment: Doh, that makes sense. You are absolutely right, I was asking for the wrong thing. Your equations for figuring out the correct final sizes are what I needed. Thank you!


How can I make two chords on the ukulele sound interesting for a whole song?

Question: I'm accompanying my friend on one of her own songs with my uke. However, the song only has two chords the whole way through!
The chords are Dm and Am - eight even beats of each. It's mostly slow and steady then it speeds up at the end to a fast climax.
Any ideas for different chord configurations or strumming patterns that could make this more interesting for me to play?
Comment: You might compare with "What I Got" by Sublime, which is just two chords over and over again. If the rhythm, lyrics, and singing are good enough, you don't need interesting chords.
Comment: Also check out [Wagner's prelude to *Das Rheingold*](https://www.youtube.com/watch?v=cjkjF9OfMe0). It's a very different style, but it only uses *one* chord!
Comment: There are several songs that only use two chords. One is 'Dance the Night Away' The Mavericks. I've played it with many different bands, and the truth is, without the rhythmic chords - the same all through intro, verses, choruses, solo - it doesn't sound as good. Yes, boring to play, but it WORKS! If your rhythm pattern is solid, people will listen more to the whole song, especially the lyrics. And let's face it, the uke is only an accompaniment. Don't get in the way, but provide a solid backing like expected! Unless the song is tedious and then you need some rhythmic spice!
Answer: You could try lots of things. Changing the rhythm pattern is one, but you could also try things that work with tone (timbre) and dynamics too. Here are some ideas that come from guitar, but I think should work on ukulele too:

Palm mute. Creates a softened, quieter sound. You can increase/decrease the amount suddenly or gradually.
Change the position where you strum across the strings. Closer to the bridge becomes brighter, away from the bridge becomes softer.
Change speed and direction of the strum. You get a kind of brushing effect with a slow strum. Usually you strum down on the down beat, but strumming up creates a subtle difference in sound.
You can slide into the proper chord from a few different position - above or below by one or two frets. Just move all your fingers in one locked position at the same time. You can place the slide at different points metrically - full beat before the first beat, half beat before beat three, etc. Try to see if any work with the song.
Mix in some arpeggiation with strumming the full chord, or maybe a tremolo effect by alternating two strings rapidly.
Play the chords in various positions up the fretboard. With 8 bars per chord you have a lot of time to move through two or three positions for each chord.

With all of these you can control how much change you add to a basic pattern. This could mirror the tempo increase the song already has. Start out with a little bit of variation to a basic strumming pattern, then add bit more as the song progresses.
Another approach is to add space. Can you add some rests or hold a chord for a longer time? Maybe only add the changes or extra ukulele parts when the voice comes to a rest. The possibilities depend on what the vocal part is doing.
Answer: A simple way to add some movement to the chords without departing too far from the songwriter's prescription is to use the 'step up/step down' principle. Stepping up from D minor you would go to E minor, and stepping up from A minor you would get B minor. The most subtle approach is to step up on a 'weak' beat of the bar ( beats two or four or the second in a pair of quavers). Stepping down from D minor you would go to C major, and from A minor you would go to G major. Of course, when you've stepped up you need to return to the original chord. If you think you can get away with something a little more audacious, you can go an extra step up. For D minor you would play: D minor E minor F major, then either E minor or straight back to D minor. For A minor you would play: A minor B minor C major then either B minor or straight back to A minor. The important thing is to sneak these extra chords in without detracting from the main game, which is still D minor and A minor.
Comment: If you are in D minor, Em might clash with the B♭ of D minor.
Comment: D minor triad:DFA, E minor triad: EGB. I see no problem.
Comment: The B♭ of the *key* of D minor.
Answer: You've been offered a comprehensive list of ukulele techniques. And suggestions on elaborating a simple harmony.  But maybe they aren't necessary.  Maybe they'd just get in the way of the song.
You ask how to  "make this more interesting for ME to play".  But what does the SONG need?  Try strumming along gently and listen to the words of the song. Maybe this one just isn't about you.
Comment: To clarify, you're saying that the accompaniment doesn't need to be spiced up?
Comment: I've edited my answer to, hopefully, make this even more clear!
Answer: If it were me,I'd make my vocalist friend write a better arrangement. Seriously.
However, that's probably not an option, so start out by figuring out what she really wants for your "continuo" part.  Maybe she wants the voice to be all the audience notices, and your simple backing is there just for ambiance.  Maybe you can try doing a modal analysis of the tune and then suggesting additional chords.
Hey it could be worse: you might have to   play   Pachelbel  :-)
Comment: @CarlWitthoft Better phrase might be "vamp"? Although when I read "continuo" I immediately understood what you were saying, so there's no problem...
Comment: There's no continuo involved here. Is it possible you've replied to the wrong question?
Comment: @neilfein  What would you call an interminable 2-chord strum other than "continuo" ?
Comment: In my experience a continuo is a repeated bass part. Never heard the word used to describe the chord structure.
Comment: @user45266  I've always used "vamp" to define a few measures which are repeated an indeterminate number of times until something happens to kick off the next theme.  Most often  used in  theatre musicals to cover conversation in the middle of a song


How to show and hide menu tabs in module

Question: I have created tabs in module for different functions. As per code shown below, all the tabs are shown simultaneouly when I login to the module. But, I want to show specific modules depending on the play formats, how can I do that? I tried hook for menu_alter and created custom module but all hides specific menu directly and not condition wise.
Play URLs
<code>$items['abc/play'] = array(
    'title' => t('Welcome Player'),
    'access arguments' => array('administer users'),
    'page callback' => 'drupal_get_form',
    'page arguments' => array('score_entry_table'),
    'access callback' => TRUE,
    'file' => 'scoreentry.inc',
);

$items['abc/play/cricket_score'] = array(
    'title'         => t('Cricket Scoreboard'),
    'page callback' => 'cricket_score_table',
    'page arguments' => array(1),
    'access arguments' => array('administer users'),
    'access callback' => TRUE,
    'type' => MENU_LOCAL_TASK,
    'file' => 'cricketscore.inc',
    'weight' => 2
);
$items['abc/play/soccer_score'] = array(
    'title'         => t('Soccer Scoreboard'),
    'page callback' => 'soccer_score_table',
    'page arguments' => array(1),
    'access arguments' => array('administer users'),
    'access callback' => TRUE,
    'type' => MENU_LOCAL_TASK,
    'file' => 'soccerscore.inc',
    'weight' => 1
);
</code>
Now it shows both tabs after login, I want to show only one tab as per format. I go to scoreentry page, where I get the format code and I want to hide the tab based on that format.
Answer: Have a look at the Tab Tamer Module.
Answer: If you want to avoid showing both tabs simultaneously then you will need to manage path hierarchy. In a path hierarchy if sibling items are typo of MENU_LOCAL_TASK then they always appear in tabs simultaneously.
This is an example to break you path hierarchy in more deeper way.
<code>$items['abc/play'] = array(
    'title' => t('Welcome Player'),
    'page callback' => 'drupal_get_form',
    'page arguments' => array('score_entry_table'),
    'file' => 'scoreentry.inc',
    'type' => MENU_NORMAL_ITEM,
);

$items['abc/play/cricket'] = array(
    'title' => t('Welcome Player'),
    'page callback' => 'drupal_get_form',
    'page arguments' => array('score_entry_table'),
    'file' => 'scoreentry.inc',
    'type' => MENU_NORMAL_ITEM,
);

$items['abc/play/soccer'] = array(
    'title' => t('Welcome Player'),
    'page callback' => 'drupal_get_form',
    'page arguments' => array('score_entry_table'),
    'file' => 'scoreentry.inc',
    'type' => MENU_NORMAL_ITEM,
);

$items['abc/play/cricket/score'] = array(
    'title'         => t('Cricket Scoreboard'),
    'page callback' => 'cricket_score_table',
    'page arguments' => array(1),
    'access arguments' => array('administer users'),
    'type' => MENU_LOCAL_TASK,
    'file' => 'cricketscore.inc',
);

$items['abc/play/soccer/score'] = array(
    'title'         => t('Soccer Scoreboard'),
    'page callback' => 'soccer_score_table',
    'page arguments' => array(1),
    'access arguments' => array('administer users'),
    'type' => MENU_LOCAL_TASK,
    'file' => 'soccerscore.inc',

);
</code>


Problema de referências

Question: Estou estudando .net-core e estou tentando fazer um sistema para um hospital utilizando padrão DDD e .net-core.
Logo quando comecei a criar meus modelos e views, me deparei com um problema de referência quando fui utilizar a tag helper "asp-items"...
O que eu estou esquecendo?
Model

View
Comment: Inclua o código e não por imagem, só com as informações presentes na pergunta não é possível concluir "nada". Apresente um [MCVE]
Comment: Ps.: Experimente colocar o mouse em cima do trecho sublinhado e ler o que aparece no "Quick Actions" (lampadinha)
Comment: conseguiu resolver?
Comment: Sim, consegui! Eu havia esquecido de importar a model somente... Obg!
Answer: Felipe,
Veja a documentação:
https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.mvc.viewfeatures.htmlhelper.getenumselectlist?view=aspnetcore-3.1
Sexo é uma entidade/model pelo modelo.

Problema de referências


Nakayama's Lemma for local rings

Question: I have to prove this form of Nakayama's lemma:
Let $M$ be a finitely generated module over a local ring $R$ with maximal ideal $I$. If there is $m_1,…,m_K \in M$ such that $\bar{m_1},…,\bar{m_k}$ generate $M/IM$ as an $R/I$ vector space, then $m_1,…,m_k$ generate $M$ as an $R$ module.
I know the version:
Let $M$ be a finitely generated $R$-module and $I$ an ideal of $R$ with $IM=M$. Then there is an element $a \in I$ with $am=m$ for all $m \in M$.
And with this version I proved another version for local rings
Let $M$ be a finitely generated module over a local ring $R$. If there is a proper ideal $I$ of $R$ with $IM=M$ then $M=0$.
So I have that
$$M/IM= <\bar{m_1},…,\bar{m_k}>_{R/I}$$
then
$$M=IM +N$$
when $N=\sum_{i}^{k}Rm_i$. Now we can consider the quotient
$$M/N=(IM+N)/N=(IM)/N=I\cdot (M/N)$$
and for the second form that I wrote, we have
$$M=N.$$
My question is about
$$M/IM= <\bar{m_1},…,\bar{m_k}>_{R/I} \implies M=IM +N$$
why is it licit?
I tried with a direct computation: if I take $m+IM \in M/IM$ We have
$$m+IM=(r_1+I)(m_1+IM) +…+(r_k+I)(m_k+IM)$$
with $r_1,…,r_k \in R$.
But I don't know how to work with the right part of this equality.
Is it lecit to write:
$$m+IM=(r_1+I)(m_1+IM) +…+(r_k+I)(m_k+IM)=\\=(r_1m_1+…+r_km_k +IM)+I(m_1+…+m_k)=\\=(r_1m_1+…+r_km_k)+IM?$$
How can I prove the implication? And how can I work with this type of sum (quotient of modules + quotient of rings)?
Answer: I think working directly with ideal arithmetic makes things more confusing (it's not wrong, though).
It's simpler to reason this way: take an element $m\in M$. Then its residue in $M/IM$ can be written as
$$\overline{m} = \sum_i \overline{r_i}\overline{m_i}$$
which means that $m = \sum_i r_im_i + a$ with $a\in IM$, so $m\in IM+N$.

Nakayama’s Lemma for local rings


High CPU Usage (95-99%) for mssql on the server

Question: our server is running win2003 with mssql 2005 on the same box with 2GB ram. the website itself is based on dnn 4. We are seeing these high cpu usage spikes (mssql) of 95-99% even with no traffic.sometimes they appear with regular intervals with no traffic.
What can possible cause this? how do find the root cause?
Answer: Do you have content indexing turned on in DNN?  It could be DNN running an index of your site content.  This indexing is performed in the background and will often happen during times when you have no other traffic activity hitting the database.
I would recommend setting up a SQL Server Profiler trace on your database and see what is going on during the spike times.
Answer: During a period of high CPU usage, you need to either jump on the box and use Activity Monitor, or type in something like sp_who2. Also check to see what SQL Server agent jobs are scheduled, when - and see if this correlates. This will give you an idea of who is doing what.
Squillman's suggestion to use Profiler is good advice, especially to monitor when unattended - and check back later to see what was going on.
Comment: I HAVE RUN THE SP_WHO2 COMMAND AND HERE IS THE RESULT: https://docs.google.com/leaf?id=0ByVTgLUAxb56NDQ1Yzc5MWYtNjllMC00NTE4LTljNTEtMDY2YjJjMGVkMzgw&hl=en&authkey=CMCom5AH
Answer:
Check for SQL Agent jobs running.
Check to make sure you don't have auto-shrink enabled for any of your databases.
If the server is not dedicated to SQL, make sure memory configuration is set to leave RAM for other processes.
Profiler/sp_who as others have suggessted. Careful with Profiler, it can stress a stressed out server even more if you are capturing too much info.
Comment: 1. no sql agent jobs are running. 2. auto shrink is NOT enabled. 3: how do i allocate ram for other processes and how much should be allocated for sql? and how do i make these settings and where?


Apóstrofo (aspa simples) dentro de aspas em javascript e php

Question: Queria uma maneira mais elegante de escrever o código abaixo. Ele está passando os parâmetros via GET mas vai com \' e posso até usar uma função do PHP de replace mas queria uma solução sem gambiarra. Repara que fica ' dentro de ' ai coloco o \' mas ele vai no parâmetro e se coloco \" do php não funciona. Segue o código
onClick=\"window.parent.document.getElementById('webcam_show').src='webcam_html5/index.php?text_area=\'ta_proprietario_altera_foto64\'&frame_destino=\'frame_proprietario_foto_altera\'';window.parent.div_mostra('webcam')\"

Comment: Tem como postar mais do seu código, desde a variável que recebe a string que você quer escapar?
Do jeito que esta, confesso que para mim, esta um pouco difícil de entender
Comment: Não existe nada de gambiarra em escapar uma string
Comment: Blza, até concordo que não seria o fim do mundo, mas estou muito chato com este projeto, queria saber se tem uma maneira de não fazer isso. obrigado.
Comment: mas pq esse \' na linha? Se é no php não é necessário. O valor da variavel não necessita de aspas. Mas se contudo, todavia, mesmo assim quiser colocar aspas simples substitua \' por ".chr(39)."  Veja como fica:  ..... index.php?text_area=".chr(39)."ta_proprietario_altera_foto64".chr(39)."&frame_destino ..... Mas o melhor mesmo é atender a solicitação do MarceloBoni para que possamos responder com certeza.
Comment: o \' na linha aparece pois não posso passar uma parametro texto via get sem aspas. Obrigado pela ajuda, parece uma boa solução.
Answer: Uma possibilidade seria mover o código para uma função.
Desse modo seu HTML poderia ficar assim:
onClick="myOnClick()"

e no JavaScript:
function myOnClick(e) {
    window.parent.document.getElementById('webcam_show').src="webcam_html5/index.php?text_area='ta_proprietario_altera_foto64'&frame_destino='frame_proprietario_foto_altera'";
    window.parent.div_mostra('webcam');
}

Comment: Verdade, muito obrigado, bem mais elegante.


Confidence interval in a nonlinear model

Question: Thanks for your suggestions.Actually i have the following model that i explains to you.
Suppose i have observations structure like $$\begin{align} y_1 &=&v_1+e_1 \\  y_2 &=&v_2+e_2 \\  y_3&=&v_1^2-v_1v_2(\cos(x_1-x_2)+\sin(x_1-x_2))+e_3 \end{align}$$, and so on......
Every error term assume to follow normal distribution with mean zero but different variance. In model form above observations can be written as
Y=XB+e.(heterscedastic and nonlinear model) Here v1,v2 x1,x2....are unknown values and surely model is nonlinear in unknown parameters. We use Newton type method to estimates the unknown parameters. Guass Newton method work well to find estimates in this problem. I can find Hessian matrix or fisher Information matrix by this method that can be used to find standard errors of the estimates.
Now my question is related to confidence intervals for unknown parameters. i construct the confidence intervals for unknown parameters by using the standards that are calculated from Hessian matrix .I construct individual confidence intervals for each parameter by using t statistics?(Wald Type).When i simulate this model by knowing the actual values of parameters i found that coverage probabilities are very low special for unknown parameters angles.
Appreciate if you could suggest the technique to tackle this problem.
Comment: What model is it? There is a very general class of models that allows you to construct a theoretically sound (albeit valid only asymptotically) confidence interval by inverting a Wald test, or by using the delta method
Comment: Dear Harvey Motulsky:Could you please comments now to the problem
Answer: Sounds like you want to do a simulation. In that case you know the population parameter and the proportion of confidence intervals that include that true parameter value is your coverage probability.

To quantify the the comment by @HarveyMotulsky: In a given scenario (sample size, number, distribution and correlation of independent/explanatory/right-hand-side/x-variables, strength of effects, marginal variance of the dependent/explained/left-hand-side/y-variable, ...) you would expect about 5% of the replications to fall outside your 95% confidence interval. So if you have 100 replications, you would expect to base your estimate of the coverage on only 5 replications. That is obviously not enough. For 95% confidence intervals I tend to use 20,000 replications; that way I expect to base my estimate of the coverage on 1,000 replications in which the true hypothesis is rejected.
Once you have done so for one scenario, you change one of the characteristics (e.g. change the sample size), and do the simulation again. You continue doing that until you have a good idea of under what conditions your test fails and under what conditions your test is fine.
You would not expect a simple conclusion like "the confidence interval is right", because any statistic will fail if you use it on extreme enough data. Instead, the purpose for a simulation study should be to find the conditions under which the test performs acceptable, the conditions under which the test will not perform acceptable, and the "grey zone" where the user should be extra careful when using this test.
Comment: For the sake of others reading this: You need to run lots (thousands) of simulations. Running one simulation won't help.
Comment: @Harvey A very worthwhile clarification; a lot of people (myself included) say 'a simulation' as shorthand for something like 'a simulation study' ... which will itself have many, many sets of simulated data in it.


Is it meaningful to test the interaction effect if there is no significant effect in the main effect model?

Question: Suppose we have a linear regression model with two covariates, $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2$.
There are three possible scenarios:

Both $\beta_1$ and $\beta_2$ are significant.
Either $\beta_1$ is significant and $\beta_2$ is not, or $\beta_2$ is significant and $\beta_1$ is not.
Neither $\beta_1$ nor $\beta_2$ is significant.

My question is, statistically (not theoretically), under which case (scenario) is further investigation of the interaction between $x_1$ and $x_2$ meaningful? For example, investigating a model $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_1 x_2$. I have seen many people further investigating interaction effects under scenarios 1 and 2, but I am not sure if we should also try to evaluate the interaction effect for scenario 3.
Answer: First, I wouldn't base any sort of decision about model building on the significance of earlier phases of the model building. Statistical significance is tied up with sample size.  I would use effect sizes.
But, second, even if both main effect sizes are very close to 0, the interaction can be hugely important. This can happen when the two effects are crossed, as in the example that Jeremy Miles gave in his answer.  Or one can dilute the other without the two being crossed.
Third, it can make sense to investigate the interaction because it is part of your hypotheses or research questions. I know you said you wanted something "statistical" not "theoretical" but I don't think that dividing the issues that way makes sense here.
Comment: Based on substantive knowledge, as with any kind of effect size.  Even a small effect could be important in some fields (e.g. airplane manufacturing).
Comment: Substantive knowledge of how big an effect is "big" is common. The statistician can show what different interactions would look like.  "Under an interaction like XXX, 100,000  more people are predicted to  get pimples." or whatever.
Comment: That sounds like a good approach!  But doesn't that imply you're really testing all three parameters (two slopes and the interaction) simultaneously, rather than the interaction alone?  Or would you really be asking the expert(s) to compare two sets of predictions: one with and another without the interaction?
Comment: I would present a variety of models. First one with only the main effects. Then ones with different interactions. But always in terms of what the model would predict. The parameters for interactions are, indeed, hard to interpret, even for statisticians!
Comment: How would you establish a minimal effect size for an interaction?
Comment: "Substantive knowledge" of interactions is rare.
Comment: That seems to ignore a basic problem: the effect size of an interaction depends on how the original variables are expressed.  For categorical variables that's not an issue, but when one or both variables are numerical, it's a big deal.  In some circumstances there may be a convention (concerning the units of measure), but generally, the "effect size" of an interaction *has no intrinsic meaning.*
Comment: If I was asking a substantive expert, I'd present them with predictions from the model not with the parameter estimates.
Answer: Yes, it makes sense to test for interactions regardless of main effects .
b1 might have a positive slope if b2 is positive, and a negative slope if b2 is negative. That means that the average effect of b1 is 0.
Real life example: When people watch you doing something, it can increase your nervous system arousal. If it's something you're very good at, that might make you better. If it's something you're bad at, it might make you worse. An audience watching people play pool makes the good players better, and the bad players worse. On average, the effect of the audience is zero. That doesn't mean that an audience doesn't have an effect.
Comment: Testing the interaction coefficient alone is questionable, though: one ought to test the *group* of variables with an omnibus F test.  See https://stats.stackexchange.com/questions/65898 for a related discussion.
Answer: A large driving force behind modeling interactions is theory, but I realize that your question says you are not looking for that specifically, so I will not dive into that further (though I discuss this in the first part of my answer here for those interested).
To your main question:

My question is, statistically (not theoretically), under which case (scenario) is further investigation of the interaction between $x_1$ and $x_2$ meaningful?

Gelman and his colleagues suggest that "When inputs have large main effects, it is our general practice to include their interactions as well" (p.246-247 in this book). When are the effects large? Plotting the interactions can go a long way to understanding if there is indeed an interaction worth noting. Using the second part of the linked answer above as an example, we can clearly see that the bottom interaction between two continuous variables is minimal and probably not worth investigating further:

Whereas the plot from this answer shows a strong continuous by categorical variable interaction that is probably important to look at further (the left plot doesn't include an interaction in the model, whereas the right plot does). We can see that if we don't include the interaction, we are effectively ignoring the change in the data points and assuming that only the conditional mean varies by group and that the slope is generally negative:

Care must be taken here if you had not considered an interaction model in advance. I would say that this advice can potentially lead to HARKing if its not clearly made explicit a priori that you 1) had a previously conceived main effects-only model first and 2) you tested this only after discovering that the magnitude of main effects is large.
I will add that if you include an interaction, it is my opinion that you should always include both main effects and interactions together. By only entering the interaction, you are not disentangling the independent main effects on the outcome, which may be important to investigate.


Outlier detection for heavy-tailed data

Question: Applying modified z-score for outlier elimination on some data (Iglewicz and Hoaglin, 1993), I discovered that a big proportion of the data (~10%) was outside the range abs(z)>=3.5. Further investigation showed that the data is heavy-tailed. I assumed that the Bienaymé–Chebyshev inequality would hold for the median absolute deviation MAD, too, but obviously does not.
.sum.bool  <- function(x) c('TRUE'=sum(x),'FALSE'=sum(!x),
                            'TRUE %'=round(sum(x)/length(x)*100,1), length=length(x))

rrn <- rnorm(10000)
rrt <- rt(10000,1)

# simplified z-score for demonstration purposes
mad.outlier <- function(x)abs(x-mean(x))/mad(x) > 3
sd.outlier <- function(x)abs(x-mean(x))/sd(x) > 3

rbind(mad.n=.sum.bool(mad.outlier(rrn)),
      sd.n=.sum.bool(sd.outlier(rrn)),
      mad.t=.sum.bool(mad.outlier(rrt)),
      sd.t=.sum.bool(sd.outlier(rrt)))

On the heavy-tailed t-distribution with 1df, 14% of the data are outside 3 MADs.
      TRUE FALSE TRUE % length
mad.n   29  9971    0.3  10000
sd.n    29  9971    0.3  10000
mad.t 1381  8619   13.8  10000
sd.t    33  9967    0.3  10000

Can someone shed light on this property of the MAD/z-score in the presence of heavy-tailed distributions? What are recommendations for outlier detection for heavy-tailed data?
Answer: Ignoring the tails, the Gaussian and Cauchy (T-dist w/ DF=1) look pretty similar in their meaty center.  The MAD only looks at the meaty center (more-or-less).  The MAD estimates will be pretty similar, which will give a pretty similar range of "acceptable".  The Cauchy, with it's fat tails, will violate that acceptable range more often.
I'm not sure what your intentions are with this experiment though.  In my experience, most real world data would lie somewhere in the middle of the Gaussian to Cauchy spectrum.  When I apply robust statistics, I really just want to focus on capping the influence of any single point.  There isn't really a "correct" answer.  Robust statistics are more focused on reasonable estimates that match the bulk of your data (but not all of it).  In the real world, all definitions of "outlier" are subjective.  You need to tune acceptance levels to fit your own needs.
Comment: In general, I'd always use a robust measure of scale to identify outliers.  If you don't, extreme outliers tend to mask moderate outliers.  I'd also recommend moving past the MAD to something more efficient like Qn or a 2-stage Tau estimate of scale if you're serious about this.  Package `robustbase` in R provides many of these alternatives and more.
Comment: Thanks for your answer. The data at hand is also 'somewhere in the middle of the Gaussian to Cauchy spectrum' (and less like the standard Cauchy of the example). What I am not sure, however, is if I should use a robust measure of the variation or the standard deviation to defined outlier cutoffs. For the sd I can assure that I will not disregard too many measurements as it does not just model the meaty center in heavy-tailed distributions.
Answer: You need to look at this problem in terms of what you mean by a univariate outlier.  The traditional theory of univariate outlier detection is that the data are normally distributed and points extreme with respect to normal observations are outliers.  That is the basis of Grubbs' test which has some optimality properties and Dixon's ratio test.  As I mentioned on previous posts about outliers, my paper in the American Statistician "A Note on the Robustness of Dixon's Ratio test in Small Samples"  shows that for really small sample size 3 to 5 the test maintains its significance levels for a variety of non-normal distributions.  It has often been used with running samples of size 3 as a screening procedure.  But there are a lot of subtleties with outlier detection including the masking problem and the issue of heavytailed distributions.  I recommend that you look at the two excellent books on outliers
(1) Barnett and Lewis "Outliers in Statistical Data" and
(2) Douglas Hawkins monograph on outliers.
Most importantly though is how to interpret outliers once you found them.  An outlier found by Grubbs' test for example could be due to a contaminated normal model or simply because the distribution is heavytailed.  There are other possibilities too.  Is the outlier an error, just a chance occurrence or an indication of an important change in the process?  These questions require understanding of the data and cannot be answered on purely statistical grounds.
Now if I am to suppose from your question that you know that your data represent iid observations from a heavytailed distribution, then if you can assume a parametric form for that distribution you can derive or approximate the distribution of the maximum and minimum for a sample of size n and use that as the null distribution to test for an outlier on a sample.  If you have to estimate the parameter(s) of the distribution then you would need to convince yourself that a given set of data is outlier free and use that data to fit the parameter(s).  Then that fitted model could be used to test on an independent set of data.
Comment: Thank you for your reading recommendations. The outliers which I want to capture are really erroneous readings, and the sample sizes are between a couple and a few dozens (1000s of variables are in the same data set, each variable has variable number of readings!). I am a bit hesitant to model a distribution for outlier detection, but I will certainly investigate if the measurements distribute the same across the variables.

Outlier detection for heavy-tailed data


Como realmente o XCache trabalha?

Question: Como realmente o XCache trabalha com o PHP? É preciso necessariamente usar as funções do XCache para conseguir usar as suas "vantagens"? Como por exemplo:
<?php echo xcache_inc('count'); ?>

Ou a partir do momento que habilito a extensão extension=php_xcache.so e adicionar as configurações no php.ini eu já teria tais "vantagens"?
Answer: Bem, acredito você já saiba o propósito do XCache então não vou falar sobre a questão conceitual aqui.
Alguns fatos a respeito: utilizei o XCache a muito tempo daquela época até agora ele pouco evoluíram a questão documental, existe muito pouco conteúdo a respeito. Muitos acharam que o projeto estava morto até a última atualização no final de 2014.
Sobre o funcionamento: funciona dessa forma mesmo que você falou, basta instalar e efetuar as configurações do arquivo ini respectivo. Mas você pode sobrescrever assim como faz com algumas diretivas do php para algum determinado script/aplicação (como você fez no exemplo).
nesse post foi explicado detalhadamente as diretivas mais comuns do XCache https://stackoverflow.com/questions/20087650/configuring-xcache-cacher

Como realmente o XCache trabalha?


Creating an if-statement without hard-coding

Question: I have an array that I want to check, for example:
 var str = ["a","b","c","d","e","f"];

but I want check if this array's string values are in a database or of some sort. I want to find out which one matches and store the found matches inside an array.
 var dataFound = [];
 var count = 0;
 for(var stringData in someDatabase){
    if(stringData == str[0] || stringData == str[1] ...etc){
         dataFound[count] = stringData;
         count++;
    }
 }

 //someDatabase will consist of strings like random alphabets
 from a-z

I don't want to hardcode the if-statement because a query array can be anywhere from a, b, c .. (n) as n can represent any amount of strings. I tried to use a for-loop from 0 to str.length but that can take quite a while when I can use or operator to search in one go with all the string values, is there a way around this?
Comment: look for array#includes or array#indexOf or array#find ... and many other methods
Comment: Possible duplicate of https://stackoverflow.com/questions/237104/how-do-i-check-if-an-array-includes-an-object-in-javascript?rq=1
Comment: You can `break` from your loop just like `||` shortcuts when it found something, so that it won't take a while.
Comment: I don't want to break though, I want to find all possible values by using the | | operator and return the ones that have been found
Answer: if (str.indexOf(stringData) > -1) {
    //found
}

Array.prototype.indexOf() (MDN)
Comment: `str.indexOf("c")` will return `2` because `"c"` is the third element in the array (zero-based so the third item is located at index `2`), whereas `str.indexOf("c") > -1` will return `true` because we're asking "does `"c"` exist in `str`?"
Comment: For some reason, I even tried this in console in Chrome but it's only returning the actual index found of the array instead of returning either 0 or -1 when found
Comment: Glad I could help.
Answer: If str is an array, can you do this?
var dataFound = [];
var count = 0;
for(var stringData in someDatabase){
 for (i++; i < str.length; i++) {
  if(stringData == str[i]){
     dataFound[count] = stringData;
     count++;
  }
 }

}
Comment: I can do that but that's going to take (n)th time to search for where as I can use or operator **||** for each str value to be found in the database so it will only iterate once since we're looking for several values at once for each array in someDatabase


Recurrent event data modelling - Cox PH or AFT?

Question: I am attempting to model recurrent event data in Stata. My data is dataset of patient records and I am modelling a patients first delivery of a mobility aid and the subsequent deliveries of new mobility devices as their previous device will have broken/failed or the patient no longer has it. This means the time resets when a person is given a new mobility device. Patients can have a minimum of 2 rows of data, up to a maximum of 50 depending on how many replacements to mobility aids they have had.
I have explored different methods of conducting survival analysis, with the main two being a Cox PH model and Accelerated Failure Time model. For the Cox model the PH assumption is violated no matter what I do, I have tried creating smaller discrete time intervals, stratification and interactions between covariates. Therefore, I believe this suggests using a different method.
Instead I have been following the advice in https://jdemeritt.weebly.com/uploads/2/2/7/7/22771764/parametric.pdf and in the book Cleves et al (2010) An introduction to Survival Analysis in Stata for the AFT model. From this I believe an AFT with a Weibull distribution to be the most appropriate for my data.
So, my questions are, is an AFT with a Weibull distribution suitable for recurrent events data such as mine? Many of the examples I have read have a more simple data structure.
Someone has recommended to me that a discrete-time event model might be more appropriate, however, upon reading about them I'm not sure whether this is the case due my time data being in days rather than years/months. Is this correct?
Also, a final question, all my covariates are categorical and I tested for association between them using Pearsons Chi2 and Cramers V. Some of my covariates have moderate associations, how would I address this in either a Cox PH model or a AFT model?
Here is the results of the AFT Weibull regression in Stata for reference/context.
Many thanks in advance for any advice/help!
Comment: Discrete models work in days; they just may not be the most computationally efficient approaches because of the large number of records per subject.  Discrete time state transition models are very much easier to interpret than recurrent event analysis especially if you have other events getting in the way, e.g., a terminating event such as death.  A lot of information about these models may be found at https://hbiostat.org/proj/covid19.  One of the outputs of discrete time Markov processes is the mean time in a given state.
Comment: A Weibull AFT model also assumes proportional-hazards (PH)--it's the only distribution family that is both AFT and PH--so if the PH assumption doesn't hold in Cox analyses a Weibull model probably isn't the best choice of an AFT family.
Answer: First, as Frank Harrell noted in a comment, the main drawback with discrete-time models for this type of data is computational efficiency. What you call the units (days, weeks, months, years...) isn't really important. Yes, you are likely to have a large number of discrete times for evaluation, but I don't think that will pose a problem for data of your scale if you want to go in that direction.
Second, as I noted in a comment, if your model fails to meet the proportional hazards (PH) assumption then a Weibull model also won't work well, because a Weibull model also assumes PH. If you are using an accelerated failure time (AFT) model because of a failure of PH in a Cox model, you need to use a different distribution family and check the fit of the model.
Third, I wonder if some of your problems are arising from the following treatment of your data:

the time resets when a person is given a new mobility device.*

That implicitly assumes that the time course and covariate associations following both the initial delivery and new deliveries are the same for any of the reasons for delivery that you cite. Does that really make sense, based on your understanding of the subject matter? Would a patient-specific frailty term, representing individual tendencies to need more deliveries, help? Might this situation be better handled with a multi-state model, perhaps taking into account the reason for each new delivery?
See this paper for alternate approaches to modeling recurrent events.

*It's not completely clear to me that this is actually how the software is handling your data, as I'm not familiar with Stata syntax. The references to tstart and tstop in your display suggest that this is being modeled as a counting process over time  from the initial delivery rather than resetting time to 0 at each delivery. In any event, think carefully about the assumptions underlying the model you chose.


How do I use qemu-bridge-helper to get a virtual server running on my local network?

Question: I would like to access my virtual server from anywhere on my local network. I'd prefer to use qemu-bridge-helper so as to simplify the networking, since i'm weak in that area. After reading the Arch wiki, attempting to comprehend the Gentoo wiki, and watching an unsuccessful tutorial on YouTube, i figured i've boiled down my needs to a few simple commands--but i still don't know where i'm going wrong. Below is my configuration:
Host Configuration (arch linux)
# Create the bridge
ip link add name br0 type bridge
ip link set dev br0 up

# Not sure if this is necessary. Causes loss of connectivity on host.
ip link set eth0 master br0

# Assign an IP address to the bridge
dhcpd br0

# Firewall passthrough?
sysctl net.ipv4.ip_forward=1

# /etc/qemu/bridge.conf
allow br0

qemu script
# `qemu-bridge-helper` creates & configures a `tap0` interface, and also sets it as the bridge's master
qemu-start-script -net nic,model=virtio -net bridge,br=br0

Guest/Server Configuration (arch linux)
systemctl start dhcpcd

Basic network connectivity from VM-to-router is all i'm after. What am i missing?
Answer: The issue was a conflict with using both netctl and NetworkManager to connect the Host to the internet. Switching exclusively to NetworkManager resolved the issue.
Comment: And how exactly dido you switch to `NetworkManager`? Which commands did you use?


Central limit theory

Question: I know that when you keep adding Sn it will tend to approximate a normal curve as n gets bigger and bigger,but what happens if you change the expectancy ?
Answer: Roughly speaking, if we only change the expectation, then CLT will still hold, but each $S_n$ will be centered around $\bar{\mu}_{n} = \frac{1}{n} \sum_{i=1}^{n} \mu_{i}$. This is since CLT actually refers to the centered random variables. Note that if $\bar{\mu}_{n} \to \infty$, then anyway $S_n \overset{P}{\to} \infty$.
A more challenging extension is to change the variance. Then, the CLT doesn't always hold. A sufficient condition for it to hold is the Lyapunov condition. In essence, you need to add existence of a moment higher then second, and for it to converge slowly enough. There is some basic details in  Wikipedia.

Edit: After the debate in the comment, I changed the note to convergence in probability. I wish to emphasize again that we deviate from the normal CLT assumptions  only by changing the expectation. I provide here a proof for the claim:
et $Y_n$ be i.i.d with mean 0, variane $\sigma^2$ and assume third absolute moment exists ($\mathbb{E}\left[\left|X\right|^3\right] < \infty$). Let $\mu_i$ s.t. $\frac{1}{n}\sum_{i=1}^{n} \mu_i \to \infty$. Let $X_i = Y_i + \mu_i$. For every $M>0$, by Berry Essen theorem we have
$$
\lim_{n\to\infty}P\left(\frac{1}{n}\sum_{i=1}^{n}X_{i}\ge M\right) =\lim_{n\to\infty}P\left(\frac{1}{\sqrt{n}\sigma}\sum_{i=1}^{n}Y_{i}\ge\frac{\sqrt{n}}{\sigma}\left(M-\frac{1}{n}\sum_{i=1}^{n}\mu_{i}\right)\right)
 \\ =\lim_{n\to\infty}P\left(-\frac{1}{\sqrt{n}\sigma}\sum_{i=1}^{n}Y_{i}\le\frac{\sqrt{n}}{\sigma}\left(\frac{1}{n}\sum_{i=1}^{n}\mu_{i}-M\right)\right)
 \\ \ge\lim_{n\to\infty}\Phi\left(\frac{\sqrt{n}}{\sigma}\left(\frac{1}{n}\sum_{i=1}^{n}\mu_{i}-M\right)\right)-\frac{C_{1}}{\sqrt{n}}
 \\ =\lim_{n\to\infty}1-\Phi^{C}\left(\frac{\sqrt{n}}{\sigma}\left(\frac{1}{n}\sum_{i=1}^{n}\mu_{i}-M\right)\right)-\frac{C_{1}}{\sqrt{n}}
 \\ \ge\lim_{n\to\infty}1-C_{2}e^{-\frac{n^{2}}{2\sigma}\left(\frac{1}{n}\sum_{i=1}^{n}\mu_{i}-M\right)^{2}}-\frac{C_{1}}{\sqrt{n}}=1
$$
Where the last inequality hold since $\frac{1}{n}\sum_{i=1}^{n}\mu_{i}-M$ is positive eventually.
It seems possible to prove the almost surely convergence for sub-exponential random variables (using Borel-Cantelli lemma) but I'm not sure it holds in the general case.
Comment: Re your note: this looks false.  As a counterexample, let $Z_n$ be iid random variables of zero mean and unit variance.  For $i=0, 1, 2, \ldots$ and $j=0, 1, \ldots, 2^i-1$ (representing $n=2^i+j$), define $X_n=i+2^{-i} Z_n.$ The $X_n$ are independent, the mean of their partial sums is asymptotically near $i$ and so diverges, yet the variance of their partial sums converges to $3.$ Thus if by "$S_n$" you mean the sample standard deviations of the partial sums, they will almost surely *not* diverge.
Comment: @whuber $S_{n}$ marks the partial sums (not the sample sd of the partial sums, which i'd denote as lower case $s_n$). I meant $S_n = \sum_{k=1}^{n} X_{k}$ where $\left\lbrace X_{k} \right\rbrace_{k=1}^{\infty}$ are the series of r.v in question. I think that what OP meant by $S_{n}$ (judging from the wording)
Comment: In statements of the CLT, "$S_n$" often refers to the SDs used to standardize the partial sums.  Your reference to "centered around" suggests this could be your meaning, too.  This ambiguity will certainly lead to confusion.  So that readers do not misunderstand you, you at least need to explain in your post what *you* mean by this symbol and why you think it's relevant to the question (which is so ambiguous it has been closed in the meantime). BTW, even in this changed sense your note is still incorrect!
Comment: Here's a counterexample where $S_n$ are the partial sums.  Let the $X_n$ be independent, taking on the value $3^n$ with probability $n/3^n$ and the value $0$ with probability $1-n/3^n.$  The partial sums of the means $\mu_n=n$ are $\bar\mu_n=(n+1)/2,$ which diverges, but there is a positive chance that *all* the $X_n$ are zero, implying $S_n=X_1+\cdots+X_n$ does not almost surely diverge.
Comment: I think you are still confusing with the lower case $s_n$ that is often used as the normalized constant.  Most references I encountered denoted $S_n$ just as I defined it. What references where you refering to? If you really want to get technical, CLT refers to normalizing by the *true* standard deviation (or mean standard deviation) *not* the sample mean. If I'm not mistaken, normalizing by the sample mean should converge to t-distribution.
Comment: As for you second counter example, your variance isn't fixed in the series of random variables, while I explicitly stated that from the conditions of the classical CLT, we *only* change the requirement for the mean to be identical.
However, it seems that I didn't think about the proof well enough, and I'm not certain anymore it holds in the general case. I updated the claim and added proof.
Comment: I cannot "confuse" anything that you haven't defined!  The notation is not universal by any means.  It is crucial that you explain the symbols you use.

Central limit theory


How to list files content diffs for particular TFS changeset via command line?

Question: Changeset consists from files (let's consider only file changes, adding, removing). I want specify changeset number and obtain text output with all files modifications against previous version similar to TF difference command:
full file name 1
....
line 1;
- old line;
+ new line;
line 3;
....
full file name 2
....
line 1;
- old line;
+ new line;
line 3;
....

I don't want restrict directory scope. How to list such diffs for concrete changeset in command line?
Comment: this works on small test project: "tf difference $/TeamProjectRoot /version:C4~C5 /noprompt /r". But on real project it is so slow that I was waiting for a whole day and didn't encountered any useful output - only empty directories (I compared from root of team project):
Comment: Maybe there are no changes. You can view the history in soucre control explorer. Double check the difference between the two changeset of the project. Or you can try to define a specific file (ex. xx.sln) which  has obviously changed under the projcet during two versions.
Comment: There were changes - rerun was successful. I run several diffs concurrently. I obtained 3 kinds of output 1) normal with proper changes 2) zero length files 3) files of equal size with all directories listed recursively - but with no changes. The problem was insufficient disk space for TFS temp files. Console has errors but files (I redirected output to) have no errors. So, the command is correct but performance is slow. **It seems that all work is performed on client side**. On first stage (during several hours) processors load is 100%; after that network traffic increases.
Answer: tf difference $/TeamProjectRoot /version:C4~C5 /noprompt /r

when we want to list diffs for the whole project tree for changeset 5.


Way to update specific values in Bootstrap cards

Question: I apologize for the amateur question here, however, I am experiencing some issues with a quick demo website I am creating for a friend.
The purpose of the site is to take the total of all "credit card values", and then divide it by the total amount of "points" there are to arrive at the "points per dollar"
Once that is done, I'd like to multiply the "points per dollar" by the total amount of "points" that are specified in the dropdown menu by the user, and then update all of the cards that have something selected to be the proper math with their point values.
I've successfully done all the math and created functions, but I am unsure how to actually apply it to the specific cards that need it.
I've created functions to do the math, and update the header numbers, just not apply it to the respective cards.
My code can be found here
My website is running on an express.js server with e.js as the templating engine, with bootstrap and jquery.
Any suggestions/help would be very much appreciated.
Comment: Can you provide us with some code you are working on, the link is empty!
Comment: @AwatITWork Sorry about that, you can find the updated code [here](https://www.toptal.com/developers/hastebin/papibanisu.xml)
Comment: can you share response object code
Comment: You can render ejs tamplate and send an object with response. In the ejs file you can loop over that object.
Comment: @Bhaskar There is no data coming from the web server. All the math is done in the browser. The commented code snippet is for something else I am working on. I need to filter all the cards by what they have selected in the dropdown. Then apply a specific value to the card heading
Answer: Easiest way to do this is to assign a unique id to each card so that you can write the corresponding data.
Then you can simply do something like:
$(`#${id} .whatever-class-name-for-points-element`).html(points);

It's a bit hard to tell what's going on in your code since it's not a minimum, reproducible example, but assuming you're trying to write one "points" value per employee, you could add id="<%= sm._id %>" or similar to each employee card.
FWIW, you probably want to use the id higher up (like on the card) and then use a data attribute (e.g. data-id="<%= sm._id %>") on the input.
Happy to give you further advice if you create a working, minimal CodePen/CodeSandbox/etc.
Comment: Basically if the points value change, it needs update all of them. So I could update specifically by the ID but that would not update all of cards. If the total amount changes the points per dollar changes, hence there needs to be an update of all cards since the amounts are based multiplication of the number of points selected in the dropdown by the point per dollar.


Name server change, impact analysis and down time - worst case

Question: The company I am working for is changing the name servers our registrar points to and wants a complete impact analysis in worst case scenario to impacted systems.
I know if we update the TTL's and wait approx 48 hours for replication and then duplicate all entries on old and new name servers update registrar to point to new servers and leave both servers up and running until replication from old to new has occurred there should be no down time but they are interested in worst case.
The worst case scenario is that there is a typo when updating the name server and it points to a none existent name server and is not detected until 48 hours later.
Now their biggest concern is how this will impact Microsoft Office 365 services... specifically how often does Microsoft check for the txt ownership of domain record and is there a possibility all of business could become locked out of O365 etc. because Microsoft checked and decided you no longer own the domain while the name servers couldn't be found. Does any one have experience with that kind of outage and can say if O365 would go down? I can't find specific Microsoft docs on this topic which Business would prefer to see. Has anyone else come across Microsoft docs specifying these details?
Comment: "I know if we update the TTL's and wait approx 48 hours for replication" There is no hardcoded value like that in the DNS. If you change nameservers, what counts are the TTL on the `NS` records at the parent of your zone, hence something you don't control, but indeed need to be taken into account.
Answer:
how often does Microsoft check for the txt ownership of domain record

Never. The TXT record verification is used one time to verify domain ownership when you add your custom domain to your Office 365 tenant. It's never used after that and can be deleted from your DNS.
If you're concerned about how your name server change might impact your Office 365 services (such as email) then open a support case from your Office 365 tenant and ask them the question. Office 365 support is always free.
Answer: Worst case is you mess up and don't notice it then you have a outage of whatever you messed up until you fix it and TTL expires.  This can be prevented by careful checks and testing the new server before the change-over.
If you can deploy a clone of the old records on the new server worst case is the registrar blocks your update for some unexpected reason, you're then stuck with the old server until the registrar problem is solved.
Actually the worst case is probably that you put the wrong domain name for the new servers, you then have a partial outage lasting
until the registrar TTL ends.
Be sure to update the glue records to point to your new DNS.


Pig: Accessing inner tuples' fields for filtering

Question: I have a custom Pig loader like this:
A = LOAD 'myfile' USING myudf_loader()

A contains:
((key1, val1), (key2, val2), (key3, val3), ...)

That is A has an outer tuple that contains key-value pairs stored in inner tuples.
I am not using maps because maps require that key values within a relation must be unique. The keys I have do not necessarily have to be unique.
The keys are chararrays, while the values can be chararrays, ints, and floats.
I would like to access A's inner tuples, as well as the (key, value) pairs within those tuples.
For example, I want to FILTER the keys of A such that the only fields remaining are key = "city" and value = "New York City".
Example input:
DUMP A;
(("city", "New York City"), ("city", "Boston"),
 ("city", "Washington, D.C."), ("non-city-key", "non-city-value"),
 ("city", "New York City"), ("non-city-key", "non-city-value"))

Example output of the filtering, which is stored into B:
DUMP B;
("city", "New York City")
("city", "New York City")

Answer: I dont have your full pig latin script.
But you can achieve using the below idea
grouped_records = GROUP records By Key;

filtered_records = FILTER grouped_records By group='CITY'

Dump filtered_records

Cheers
Nag
Comment: Thank you, but this answer doesn't show how to access the inner tuples. Like key1 in ((key1, val1), (key2, val2), ..., (keyn, valn)).


How to use computeDistanceBetween()?

Question: I am using Google Maps Android API Utility Library to calculate distance between two points ,I have seen the docs but I don't know where I am doing wrong. The distance measured between two points are much greater . Below is my code
ParseGeoPoint requesterLocation = object.getParseGeoPoint("requesterLocation");
LatLng from = new LatLng(userLocation.getLatitude(),userLocation.getLongitude());
LatLng to = new LatLng(requesterLocation.getLatitude(),requesterLocation.getLongitude());

//Double distanceInKm = userLocation.distanceInKilometersTo(object.getParseGeoPoint("requesterLocation"));
Double dis = SphericalUtil.computeDistanceBetween(from,to);
Double disOneDp = (double) Math.round(dis * 10) / 10;
listViewContent.add(String.valueOf(disOneDp) + " Km");

Result is 12627.6 km where it should be 29.4km
Comment: @pskink this is aerial distance and it is much shorter than road distance and btw i want road distance
Comment: then,I dont know whats the use of this. Can you please tell me how to get minimum distance?
Comment: what do you see on the logcat if you call `Log.d(TAG, "from " + from + ", to " + to)`?
Comment: I think the problem is with your coordinates as you've used `computeDistanceBetween` in the correct way and as result is in metres, still `12627.6/1000 != 29.4`.
Comment: i have double checked i am doing all good
Comment: @pskink  this is what my logs are showing  27.498242,90.12489
Comment: Either this is not the complete log or you've errors, because this is just a single latLng, where's the other one?
Comment: call `Log.d` i wrote
Comment: by the way can anyone of you please check that am i doing the conversion right ie  from GeoPoint to LatLng
Comment: D/geo: lat/lng: (27.58583053,90.20794596) lat/lng: (27.498242,90.12489)
Comment: Your latlng seems fine.
Comment: see https://developers.google.com/maps/documentation/distance-matrix/intro or https://developers.google.com/maps/documentation/directions/start then
Answer: Here getting distance in kilometers (km)
private double distance(double lat1, double lon1, double lat2, double lon2) {
double theta = lon1 - lon2;
double dist = Math.sin(deg2rad(lat1))
                * Math.sin(deg2rad(lat2))
                + Math.cos(deg2rad(lat1))
                * Math.cos(deg2rad(lat2))
                * Math.cos(deg2rad(theta));
dist = Math.acos(dist);
dist = rad2deg(dist);
dist = dist * 60 * 1.1515;
return (dist);}

private double deg2rad(double deg) {
return (deg * Math.PI / 180.0);}

private double rad2deg(double rad) {
return (rad * 180.0 / Math.PI);}

Comment: I tried this and i think this will return aerial distance and i want minimum roadDistance
Comment: @dhami_ji You should have mentioned that in your question.
Answer: Here getting distance in meters (m)
try these methods will help you find radius based distance and if you are looking for distance by road try to use google map API for distance GOOGLE MAP MATRIX API , ROAD BASED DISTANCE AND TIME TUTORIAL.!
private float distance(double currentlatitude, double currentlongitude, double originLat, double originLon) {

    float[] results = new float[1];
    Location.distanceBetween(currentlatitude, currentlongitude, originLat, originLon, results);
    float distanceInMeters = results[0];

    return distanceInMeters;
}

Here getting distance in Meters(m)
Location startPoint=new Location("locationA");
startPoint.setLatitude(17.372102);
startPoint.setLongitude(78.484196);

Location endPoint=new Location("locationA");
endPoint.setLatitude(17.375775);
endPoint.setLongitude(78.469218);

double distance=startPoint.distanceTo(endPoint);

Comment: is it aerial or road distance?
Comment: @dhami_ji did you see the links i posted?
Comment: @pskink yes i did but I am still struggling on how to use it, but i think this will give me the road distance
Comment: @dhami_ji what "this"? this answer?
Comment: pass it your current location and then final location.@dhami_ji .? and if you have any query then do let me know.!
Comment: Pretty sure this is direct distance, not along roads
Comment: @cricket_007 thanks mate for this knowledge, could you please explain how to get road distance then.?
Comment: You would use the Distance matrix API and parse out the sum of the distances https://developers.google.com/maps/documentation/distance-matrix/intro
Answer: I think you are actually doing it wrong.
According to the official Google map utils documentation ;

computeDistanceBetween(LatLng from, LatLng to)

Returns the distance between two LatLngs, in meters.

The result is returned in meters, so you should convert it to kilometers like so;
Double disInMeters = SphericalUtil.computeDistanceBetween(from,to);
Double disInKilometers = disInMeters / 1000;
// Double disOneDp = (double) Math.round(dis * 10) / 10;
listViewContent.add(String.valueOf(disInKilometers) + " Km");

How to use computeDistanceBetween()?


Keyboard Expression Pedal Position

Question: I have a keyboard setup using a keyboard, a sustain pedal and an expression pedal. An image of my keyboard stand is below for reference. I am having trouble finding a good position for the expression pedal, because my knee hits the stand when I put my foot on the expression pedal. I saw some images of organs with expression pedals, which seem to be a bit higher and more angled, than the usual floor pedals.
I think I have my keyboard and seat positions currently very similar to normal piano heights
Should setup my stand and seat different to fix this problem?
Are organ manuals (and seats) higher than piano manuals?
Comment: Your knee should be nowhere near hitting the stand for this kind of keyboard stand. Sounds like you're sitting way too high.
Comment: Does your knee hit the stand when you put your foot on the sustain pedal?
Comment: No. I added the expression pedal later, had no problems before and did not change anything so far.
Answer: I use exactly the same stand - very good! But it's on its lowest height, so maybe 3 or 4" lower than yours. I have no problem with a damper pedal and expression pedal, in fact, I mounted those two along with a volume pedal, on a piece of ply, making it easier to set up and break down. And, they don't wander around. My k'bd is 3-4" high, so I can't really see what the problem is, except your seat is too high, or your foot to knee measurement is quite long.
My damper is on the right, central is the expression, and left is the volume (swell) pedal. It all works well, for me, but if not for you, the stand needs changing for maybe a 'Z' stand, giving more room underneath. When I use more than one k'bd, the higher ones are more tilted than yours appears to be, as well.
Comment: It seems like you're recommending either a higher stand or a lower seat, correct?
Comment: @ToddWilcox - can't see another alternative, short of a different stand.
Comment: Agreed. Just wasn't sure that was your point. I'm terrible at reading between the lines, though.
Comment: @ToddWilcox - they're quite simple: let's face it, in treble clef, they're FACE..!

Keyboard Expression Pedal Position


Is every map between connected CW-complexes homotopic to a map sending $x$ to $y$?

Question: Let $f \colon X \to Y$ be a continuous map between connected CW-complexes $X$ and $Y$.
Choose points $x \in X$ and $y \in Y$. Is $f$ homotopic to a map sending $x$ to $y$?
Comment: Do you have any results at your disposal that you think might help? As written, this just looks like a homework problem. Please provide some context.
Comment: Good to know. Just so you know, these kinds of things should be included in the question otherwise the question may be closed for lacking context.
Comment: I'm reading this [paper](http://arxiv.org/pdf/0902.2480.pdf) on aspherical manifolds where the statement is used implicitely after Theorem 1.1. Sadly, I do not know much techniques about CW-complexes.
Comment: Here's a necessary condition: $f(x)$ and $y$ must be in the same path component
Comment: @AlexG. Note that $Y$ is path-connected as a connected CW-complex.
Answer: In short: Suppose $f: X\to Y$ is any map with $f(x) = y^{\prime}$ and let $\gamma:[0,1]\to Y$ be a path between $\gamma(0)=y^{\prime}$ and $\gamma(1)=y$. Then $f$ and $\gamma$ together constitute a continuous map $$X\times [0,1]\supset X\times\{0\}\cup_{\{x\}\times \{0\}} \{x\}\times [0,1]\to Y.$$
Since the inclusion $\{x\}\hookrightarrow X$ is a cofibration, this map extends to a map $X\times [0,1]\to Y$, which can be interpreted as a homotopy from $f$ to a map $X\to Y$ sending $x$ to $y$.
More details: First, without loss of generality, we can assume that $x$ is a $0$-cell of the given CW-structure by potentially passing to a subdivision. It is therefore sufficient to show that the any inclusion $A\hookrightarrow X$ of a subcomplex $A$ of a CW-complex $X$ is a cofibration. The corresponding extension problem
$$(\ddagger)\qquad X\times [0,1]\supset X\times\{0\}\cup_{A\times \{0\}} A\times [0,1]\to Y$$ can then inductively be reduced to the special case $X={\mathbb D}^n\supset A={\mathbb S}^{n-1}$, where it holds since the inclusion on the LHS of $(\ddagger)$ is a deformation retract.
Reference: Hatcher, Algebraic Topology, Proposition 0.16
Answer: Yes, this is true and to prove this, just follow the proof of the Cellular approximation theorem.


Doubt about the definition of the normal plane of a curve

Question: I was trying to solve the following exercise from Shifrin's textbook (ex. 9 of section 1.2)

Prove that if all the normal planes of a curve pass through a particular point, then the curve lies on a sphere

My attempt was the following: consider $\alpha:I \longrightarrow \mathbb{R}^3$ an arclength-parametrized curve. The normal plane at point $s \in I$ is the plane $\langle N(s),B(s) \rangle$. From the hypotesis we have that $\exists P \in \mathbb{R}^3$ such that for every $s$ we have that
$$\langle N(s),B(s) \rangle =\{x \in \mathbb{R}^3 \mid T(s) \cdot (x-P)=0\}
$$
So I wanted to show that $\alpha (s) \in \langle N(s),B(s) \rangle$, because this implies that $\alpha '(s) \cdot (\alpha (s)-P)=0$, but $2\alpha '(s) \cdot (\alpha (s)-P)=f'(s)$ where $f(s)=(\alpha (s)-P) \cdot (\alpha (s)-P)=\parallel \alpha(s)-P \parallel ^2$, and so we can conclude that $\alpha$ lies on a sphere because its norm is constant.
The problem is that I don't know how to show that $\alpha(s) \in \langle N(s),B(s) \rangle$ and the second answer to this question also made me think if the normal plane is $\langle N(s),B(s) \rangle$ or the translated $\alpha(s) + \langle N(s),B(s) \rangle$.
So my question is: what is the exact definition of the normal plane? And is my attempt correct? How to conclude it?
Edit: maybe I figured out how to solve it. Since, from what I understood, the normal plane is the plane spanned by $N(s),B(s)$ that  pass through $\alpha(s)$, we can describe it with
$$\Pi _s=\{x \in \mathbb{R}^3 \mid T(s) \cdot (x-\alpha(s))=0\}$$
Since we know from hypotesis that $P \in \Pi _s$ we have that $T(s) \cdot (P-\alpha(s))=0$ for every $s$. Thus we can conclude as I said before. Is it correct?
And another question: does $\Pi _s = \alpha(s) + \langle N(s),B(s) \rangle$?
Answer: We assume that all the normal planes to the curve $\alpha(s)$  pass through a the origin $0$ of the euclidian space.
As $\alpha'(s)$ is tangent top the curve, it perpendicular to $\alpha (s)$. Then $<\alpha (s). \alpha(s)>=F(s)$ satisfies $F'(s)= 2<\alpha (s) . \alpha'(s)>=0$. In other words the square of the distance of $\alpha (s)$ to the origin is constant. And the curve lies on a sphere.
If the special point is not the origin, just change the origin.
Comment: If course : this plane is the affine plane through $\alpha(s)$ which is orthogonal to $\alpha'(s)$. Btw you do not use $N, B$ and this is perfectly defined for a $C^1$ curve..
Comment: Why is $\alpha '(s)$ perpendicular to $\alpha(s)$?
Comment: The vector $\alpha(s)$ is the vector through the origin. It belongs to the normal plane by hypothesis.
Comment: So the normal plane is $\alpha (s) + \langle N(s),B(s) \rangle$ and not $\langle N(s),B(s) \rangle$? This is my only doubt, the rest of the proof is clear to me


Merge Dataframes on basis of coordinates having no common columns

Question: INPUT :
df1
Pg  x0      y0      x1      y1      Text
1   521.3   745.92  537.348 754.097 word1
1   538.982 745.92  580.247 754.097 word2
1   527.978 735.253 572.996 747.727 word3
2   268.985 732.36  341.59  746.636 word4
2   344.443 732.36  390.175 746.636 word5

df2
Pg  x0      y0      x1      y1      Text                T   R   C
1   507.6   730.8   593.76  754.8   word1 word2 word3   1   1   2
2   334.56  732.36  401.34  746.636 word5               2   3   1

Expected OUTPUT :
Pg  x0      y0      x1      y1      Text    T   R   C
1   521.3   745.92  537.348 754.097 word1   1   1   2
1   538.982 745.92  580.247 754.097 word2   1   1   2
1   527.978 735.253 572.996 747.727 word3   1   1   2
2   268.985 732.36  341.59  746.636 word4
2   344.443 732.36  390.175 746.636 word5   2   3   1

I need to find which all words in df1 are present in df2 on basis of coordinates(overlap) and not Text based approach. After this I need to copy the values of columns [T, R, C] from df2 to df1.
For eg : First row of df2 has coordinates that overlap the coordinates of the word1, word2, word3 of df1. Overlap here means the bbox(x0, y0, x1, y1) of a row in df1 should lie inside the bbox(x0, y0, x1, y1) of a specific row of df2.
My Approach :
I am iterating each row in df2 and then comparing each row coordinate from df1 to find any overlaps and then merging the dataframes.
for i, r in df2.iterrows():
    df1.loc[
                (df1.x0 >= r.x0) &
                (df1.y0 >= r.y0) &
                (df1.x1 <= r.x1) &
                (df1.y1 <= r.y1) , 'flag'] = 1

    df1.loc[df.flag == 1, ['T', 'R', 'C']] = r.T, r.R, r.C

Problem is the whole process is working properly as expected but takes a lot of time to run. It takes around 90 seconds to run df1 = 20,000 rows and df2 = 3500 rows.
Comment: Can you post your working code that merges the dataframes so we can make specific recommendations on how to improve it?
Answer: You can use apply and masking. Example:
def compare(row):
    mask = df2[
        (df2['x0'] <= row['x0']) &
        (df2['x1'] >= row['x1']) &
        (df2['y0'] <= row['y0']) &
        (df2['y1'] >= row['y1'])
    ]
    if mask.empty:
        return row
    row['T'] = mask['T'].tolist()[0]
    row['R'] = mask['R'].tolist()[0]
    row['C'] = mask['C'].tolist()[0]

return row

result = df1.apply(compare, axis=1)


Improving SVM classification

Question: I have a classification problem (bioinformatics domain) where I have around 333 features. Currently, I am first selecting features (using importance feature of random forest) and then pushing the same through RBF kernel for SVM. On the results front, I have section of data which gives bad results on the classifier.
There is feature rscindex, which separates the data into well classified data and incorrectly classified data,i.e data points with rscindex < 0.8 are performingly badly during classification stage. Please give me ideas on how to improve my results on this badly performing section of data. Will using Fisher kernel help (if so is there a library I can use) or is there a sampling technique that I should be looking into.? Please help. Thanks
Comment: How large is your training dataset in number of samples? The ratio of #samples/#features can help on deciding if a non-linear kernel is needed, and if feature selection makes any sense at all.
Comment: @iliasfl My positive set is around 2000 and negative set is around 3000. I have reduced the number of features to less than 100 using feature selection.
Comment: Not sure if it matters with the number of data examples that you have, but you may need to use a weight parameter in your SVM to correct for imbalanced classes. How bad is the performance actually and what do you mean with "a section" of your data?
Answer: Two suggestions:

make sure your parameters are properly tuned (in the case of SVM you will probably have $C$ and $\gamma$ for the kernel)
try without feature selection (SVM is fairly robust to useless features anyway).
Comment: To use SVM without feature selection, it is *vital* to tune the kernel and regularisation parameters very carefully (I use leave-one-out cross-validation and LS-SVMs and have generally found feature selection makes things worse for almost all of the problems I have looked at).  Also try a linear SVM first, rather than an RBF SVM, many problems with many features are best classified by a linear classifier (even if the underlying problem is non-linear, because of the difficulty in tuning the kernel)
Comment: I tried using SVM without feature selection. the performance degrades without it.

Improving SVM classification