Using brute force to get exact p-value instead of Chi-Squared Test of Independence

Question: Background
I have the following contingency table
+------+-----------+-----------+
|      |    X      |  Not X    |
+------+-----------+-----------+-------+
|  Y   |    60     |     20    |  80   |
+------+-----------+-----------+-------+
| notY |    15     |     40    |  55   |
+------+-----------+-----------+-------+
       |    75     |     60    |  135  |
       +-----------+-----------+-------+

where I've included the row and column marginals as well as the table total.
I would like to know if the rows and columns are independent or not.  I could just use a Pearson's Chi-Square Test for Independence to get the p-value, which in R is
our_table <- matrix(c(60,20,15,40),nrow=2,byrow=T)
pvalue <- chisq.test(our_table, correct=F)$p.value

This gives a p-value of $4.17\times10^{-8}$, which seems reasonable since this table does not look independent.
My Goal
As an exercise, I attempted to calculate the exact p-value for my contingency table by looking at every possible contingency table constrained to have 135 observations and see how many are as or more extreme than mine.  Note: I don't fix the row or column marginals, just as in the Chi-Squared Test for Independence.
Pearson's Chi-Square is just an approximation which was derived from a (generalized) likelihood ratio test[1] so I will be working with this test directly.
Likelihood Ratio
The marginal probabilities for the rows and columns are
$$\pi_{i\cdot}=\sum_{j=1}^2\pi_{ij}$$
and
$$\pi_{\cdot j}=\sum_{i=1}^2\pi_{ij}$$
respectively, where $\pi_{ij}$ is the probability of an observation being in the i-th row and j-th column.
The null hypothesis is that the rows and columns are independent, i.e.
$$\text{H}_0:\;\; \pi_{ij}=\pi_{i\cdot}\pi_{\cdot j},\;\; \forall i,j\;\;,\tag{1}$$
and under the null hypothesis, the MLEs for the $\pi_{ij}$ are given by[2]
$$\hat{\pi}_{ij}=\hat{\pi}_{i\cdot}\hat{\pi}_{\cdot j}
=\frac{n_{i\cdot}}{n}\frac{n_{\cdot j}}{n} \tag{2}$$
where $n=135$ is the total number of observations, $n_{i\cdot}=(75,60)$ are the column marginal counts and $n_{\cdot j}=(80,55)$ are the row marginal counts.
Under the alternative hypothesis, the cell probabilities are free, but must all sum to 1.  Under this hypothesis, the MLEs for the cell probabilities are:
$$\tilde{\pi}_{ij}=\frac{n_{ij}}{n}$$
The likelihood ratio is then given by
$$\Lambda = \frac{\frac{n!}{n_{11}!n_{12}!n_{21}!n_{22}!}
\hat{\pi}_{11}^{n_{11}}
\hat{\pi}_{12}^{n_{12}}
\hat{\pi}_{21}^{n_{21}}
\hat{\pi}_{22}^{n_{22}}}
{\frac{n!}{n_{11}!n_{12}!n_{21}!n_{22}!}
\tilde{\pi}_{11}^{n_{11}}
\tilde{\pi}_{12}^{n_{12}}
\tilde{\pi}_{21}^{n_{21}}
\tilde{\pi}_{22}^{n_{22}}}
= \prod_{i=1}^2\prod_{j=1}^2 \left(\frac{\hat{\pi}_{ij}}{\tilde{\pi}_{ij}}\right)^{n_{ij}} \;\;.$$
Simplifying and taking the logartihm, we have
$$\ln \Lambda = \sum_{i=1}^2\sum_{j=1}^2 n_{ij} \ln\left[\frac{n_{i\cdot}n_{\cdot j}}{n_{ij}n}\right]\;\;.\tag{3}$$
and for our table, $\ln\Lambda = -15.53$.
p-value
(edited)
I have two nuisance parameters, $\pi_{1 \cdot}$ and $\pi_{\cdot 1}$.  According to @Scortchi, the "the p-value is defined as the maximum of the sum of the probabilities of the more extreme tables over all values of both these nuisance parameters".  In other words,
$$\text{p-value}= \;\max_{\pi_{1 \cdot}, \pi_{\cdot 1}} \;P\left(\ln \Lambda < -15.53 | \pi_{1 \cdot}, \pi_{\cdot 1} \right)$$
where the number -15.53 was calculated above and $\ln \Lambda$ is given by Eq (3).
This can be rewritten as
$$\text{p-value}=\,\max_{\pi_{1 \cdot}, \pi_{\cdot 1}}\;\left\{ \sum_{n_{11},n_{12},n_{21},n_{22}} \;\;
\frac{n!}{n_{11}!n_{12}!n_{21}!n_{22}!}
\pi_{11}^{n_{11}}
\pi_{12}^{n_{12}}
\pi_{21}^{n_{21}}
\pi_{22}^{n_{22}}
\right\}$$
where the summation is over all possible contingency tables which have $\ln \Lambda < -15.53$, and where the $\pi_{ij}$ are given by Eq (1).
The Python script below will calculate the above p-value, but only if you tell it what the nuisance parameters are.
Correct me if I'm wrong, but I don't think there's a way to analytically get the nuisance parameters.  We could use a computer to numerically maximize the p-value. I wonder if that would be a convex optimization...
I didn't implement the optimization but I tested out a couple different choices of nuisance parameters and the highest p-value I got was from $\pi_{1 \cdot}=0.8, \pi_{\cdot 1}=0.5$, with p-value = $4.72\times10^{-8}$, which is in good agreement with our original p-value of $4.17\times10^{-8}$.

Script
(edited)
import numpy as np
from math import log, factorial, exp

our_table = np.array([[60,20],[15,40]])

# I found these particular choices for the 2 nuisance parameters to be a good
# choices to maximize p-value (implementing a proper numerical optimization would
# be better of course)
row_probs = np.array([0.8,0.2])
col_probs = np.array([0.5,0.5])
cell_probs = np.outer(row_probs, col_probs)

num_rows = our_table.shape[0]
num_cols = our_table.shape[1]
n = our_table.sum(axis=(0,1))

def binomial(N,k):
    '''N choose k'''
    f = factorial
    return f(N) / (f(k) * f(N-k))

def log_likelihood_ratio(table):
    '''Find log likelihood ratio under the setup of a chi-squared test for
    independence

    table:  array representing a contingency table
    '''
    row_marginals = table.sum(axis=1)
    col_marginals = table.sum(axis=0)
    n = table.sum(axis=(0,1)) # (same as n in the enclosing scope)

    # Now we take the double-summation in the formula for log likelihood ratio
    log_lik_ratio=0
    for row in range(num_rows):
        for col in range(num_cols):
            # if table[i,j] is zero, it is understand that this term is zero
            if table[row,col] != 0:
                log_lik_ratio += table[row,col] * log((row_marginals[row] * col_marginals[col]) \
                                / (n * table[row,col]))

    return log_lik_ratio

def prob_of_table(table, cell_probs):
    '''Given the particular settings of our nuisance parameters, calculate
    the probability of getting a given table, which is just given by a
    multinomial distribution'''
    n = table.sum(axis=(0,1)) # (same as n in the enclosing scope)
    num_rows = our_table.shape[0]
    num_cols = our_table.shape[1]

    # Now build up the log of the multinomial porbability one cell at a time
    # I use logarithms to make this computationally feasible
    log_prob = log(factorial(n))
    for row in range(num_rows):
        for col in range(num_cols):
            log_prob += table[row,col] * log(cell_probs[row,col]) \
                    - log(factorial(table[row,col]))

    return exp(log_prob)

our_log_lik_ratio = log_likelihood_ratio(our_table)

# Find out how many contingency table have equal or smaller likelihoods ratios
# variable to hold cumulative probability for tables more extreme than ours
# by probability, we mean the likelihood of a table under given nuisance parameters
cumul_prob = 0.0

# 3 nested for loops for all possible ways to split points into cells (3 degrees of freedom here)
for count11 in range(0, n+1):
    for count12 in range(0, n-count11+1):
        for count21 in range(0, n-count11-count12+1):
            count22 = n - count11 - count12 - count21
            cont_table = np.array([[count11, count12], [count21, count22]])
            #current log likelihood ratio
            cur_log_lik_ratio = log_likelihood_ratio(cont_table)
            print('__')
            if cur_log_lik_ratio <= our_log_lik_ratio:
                cumul_prob += prob_of_table(cont_table, cell_probs)
                print(cont_table, ' has log_likelihood of ', cur_log_lik_ratio, ' which is <= ', our_log_lik_ratio,'\n')

p_value = cumul_prob
print('For the table, ', our_table, ', the p-value is: ', p_value)

[1]: See Mathematical Statistics and Data Analysis, 3rd ed. by John Rice, Section 13.4.
[2]: Rice, p. 522.
Comment: You're counting the more extreme  tables rather than summing their probabilities under the null
Comment: Note also the null is composite.
Comment: @Garret: In the case of a fixed total but no fixed margins  you have two nuisance parameters to consider. So the p-value is defined as the *maximum* of the sum of the probabilities of the more extreme tables over all values of both these nuisance parameters.
Comment: Garrett - now I've seen what you're trying to achieve - note that **both** the usual chi squared tests for an $m\times n$ table (homogeneity and independence) condition on both margins.
Comment: @Glen_b, are you referring to the fact that the Pearson's Chi-Squared statistic approximately follows a $\chi ^2$ distribution with df=1?  If so, my understanding was that the test for independence only fixes the total.  The df=1 comes from $\text{H}_0$ having 2 df, $\text{H}_A$ having 3 df, and so the $\chi ^2$ distribution has 3-2=1 df.
Comment: @Garrett: That's right. There's no analytic way to find where the candidate p-value attains its maximum over the two nuisance parameters. You can get rid of one by conditioning on either margin, or both by conditioning on both margins; or take a Baysian approach & integrate them out of the posterior density for the odds ratio.
Comment: The issue of calculating the "exact p" value is not so much an issue of how the test statistic is derived, but finding the appropriate small-sample distribution of the test-statistic under the null hypothesis. This is usually surprisingly negligible.
Comment: Secondly, the likelihood ratio test is certainly the "uniformly most powerful" test, but the Pearson Chi-Square test is a score test which is asymptotically equivalent to the LRT.
That means that they usually give very nearly identical inference.
Comment: Lastly, the Fisher's Exact test is a nice test for $2 \times 2$ contingency tables that has an exact $p$-value (although the test is not of correct size). I wonder if you have encountered that in the course of your deliberation over this topic and what you thought about the major differences between it and other (model based) tests.
Comment: Note that R will get this probability for you via simulation --
`chisq.test(our_table,simulate.p.value=TRUE,B=100000000)` run 3 times gave 6, 6 and 8 $\times 10^{-8}$ as p-values, suggesting a p-value something on the order of $7\times 10^{-8}$ for a Pearson chi-square.
Comment: @Scortchi, right, I see I was mistaken in assuming all tables have equal probability.  So under $\text{H}_0$, how would I calculate the probability of getting a given table?  Having a composite null hypothesis is throwing me off.
Comment: @AdamO, thanks.  My motivation for working through this exercise was to prove to myself that the Chi-Squared Tests for Independence and the Chi-Squared Test for Homogeneity gave different p-values (in the non-asymptotic case where the sample size is small).  See [here](http://stats.stackexchange.com/questions/77928/how-does-chi-squared-test-of-homogeneity-differ-from-other-chi-squared-tests?lq=1).  That's why I'm not using Fisher's Exact Test or Chi-Squared tests.
Comment: @Glen_b, good idea! I thought I might gain some insight into the probabilities for contingency tables by looking at the source for `chisq.test(our_table,simulate.p.value=T)`.  But it turns out that to run the MC simulation, it generates tables with fixed row and column marginals (given by your particular marginals), generating exactly the same tables as `fisher.test(our_table, simulate.p.value=T)`.  (the function generating the tables for both is `rcont2` which I found in `R-3.0.1/src/library/stats/src/rcont.c`)  I don't know what to make of that...
Comment: @Garret: Perhaps it'd be helpful to look at code from the `Exact` package, which is for *unconditional* tests in 2x2 tables.
Comment: @Glen_b: Not sure what you mean: if "the usual chi-squared tests" involve the asymptotic approximation of Pearson's statistic to the chi-squared distribution, they work whether you condition on one or both margins, or only the total. Yule's correction is supposed to make the result closer to that of Fisher's exact test for smaller samples.
Comment: @Scortchi If you only condition on 1 margin you'll have 2df, and if you only condition on the total, you'll have 3df, unless I've gotten muddled on this (which is definitely possible)
Comment: @Glen_b: If the expected frequencies are pre-specified (the G.O.F. test) there are indeed 3 d.f.; if they're estimated from the marginal totals (the test of association) there's only one. Pearson originally used 3 df, but Fisher corrected him (& not by saying you should condition on marginal totals as with his exact test): [Stigler (2008), "Karl Pearson's theoretical errors & the advances they inspired", *Statistical Science*, **23**, 2](http://projecteuclid.org/euclid.ss/1219339117).
Comment: @Scortchi, thanks!  I've edited my question to incorporate your definition of the p-value.  The problem seems to be solved now.  Do you mind just taking a quick glance at the (edited) section on the p-value to make sure I interpreted what you said correctly?  Also, I was wondering if there's an analytic way of determining the best choice of nuisance parameters. If this question is solved, could you copy and paste your last comment into an answer so I can mark the question as solved?
Comment: @Scortchi -- you now seem to be making he point I was trying to make.
Comment: @Glen_b: Don't think so: I'm saying that for tests of independence or homogeneity the exact distribution of Pearson's test statistic depends on the conditioning scheme (on the overall total, on one or other marginal totals, or on both marginal totals), but that asymptotically it tends to the chi-squared distribution with **one** degree of freedom under **any** of those conditioning schemes. For a goodness-of-fit test to a pre-specified multinomial distribution the test statistic is different (expectations are not derived from marginal totals) & asymptotically it tends to the chi-squared ...
Comment: ... distribution with **three** degree of freedom. You *seem* to be saying that, for tests of independence with multinomial sampling, if you only condition on the overall total, the distribution of Pearson's test statistic (with expected values derived from marginal totals rather than pre-specified) will asymptotically tend to a chi-squared distribution with three degrees of freedom. That was an error first made by Pearson himself.
Comment: @Scortchi, thanks!  It seems solved then, can you please turn your comment into an answer?
Answer: First, rather than counts of more (or as) extreme tables that need to be summed, it's their probabilities under the null hypothesis. Second the null hypothesis is composite, with $\pi_{1.}$ and $\pi_{.1}$ as nuisance parameters: the p-value is defined conservatively as the maximum of the sum of the probabilities of the more extreme tables over all values of both these nuisance parameters. There's no analytic solution, but the approximation of both the likelihood ratio and Pearson's score statistic to a chi-squared distribution with one degree of freedom is often close enough. If you want an exact test then, even for a multinomial sampling scheme, conditioning on either row or column totals (and thus removing one nuisance parameter) can be justified/urged on the grounds that either is ancillary; conditioning on both (and thus removing both nuisance parameters) can be justified/urged on the grounds that together they're approximately ancillary. In a Bayesian analysis you have priors for the nuisance parameters and simply integrate them out to get the posterior density of the odds ratio. Berger & Boos devised some tests that carry out the maximization over a confidence interval for the nuisance parameters, and still maintain the correct size.†

# specify marginal totals
n1. = 80
n2. = 55
n.1 = 75
n.2 = n1. + n2. - n.1
n.. = n1. + n2.

# specify nuisance parameters
pi1. = 0.6
pi.1 = 0.5

# specify no. simulations to perform throughout
no.sims <- 100000

# define function to calculate Pearson's test statistic
pearson.test.stat <- function(x){chisq.test(matrix(x,c(2,2)), correct=F)$statistic}

# simulate conditioning on overall total
multinom.obs <- rmultinom(no.sims,n..,c(pi.1*pi1., (1-pi.1)*pi1., pi.1*(1-pi1.), (1-pi.1)*(1-pi1.)))
apply(multinom.obs , pearson.test.stat, MAR=2) -> multinom.sim.stats

# simulate conditioning on row totals
matrix(NA, 4, no.sims) -> binom.row.obs
binom.row.obs[1, ] <- rbinom(no.sims,n1., pi.1)
binom.row.obs[2, ] <- n1. - binom.row.obs[1, ]
binom.row.obs[3, ] <- rbinom(no.sims,n2., pi.1)
binom.row.obs[4, ] <- n2. - binom.row.obs[3, ]
apply(binom.row.obs, pearson.test.stat, MAR=2) -> binom.row.sim.stats

# simulate conditioning on column totals
matrix(NA, 4, no.sims) -> binom.col.obs
binom.col.obs[1, ] <- rbinom(no.sims,n.1, pi1.)
binom.col.obs[2, ] <- n.1 - binom.col.obs[1, ]
binom.col.obs[3, ] <- rbinom(no.sims,n.2, pi1.)
binom.col.obs[4, ] <- n.2 - binom.col.obs[3, ]
apply(binom.col.obs, pearson.test.stat, MAR=2) -> binom.col.sim.stats

# simulate conditioning on both row & column totals
matrix(NA, 4, no.sims) -> hypergeom.obs
rhyper(no.sims, n.1, n.2, n1.) -> hypergeom.obs[1, ]
n1. - hypergeom.obs[1, ] -> hypergeom.obs[2, ]
n.1 - hypergeom.obs[1, ] -> hypergeom.obs[3, ]
n.. - hypergeom.obs[1, ] - hypergeom.obs[2, ] - hypergeom.obs[3, ] -> hypergeom.obs[4, ]
apply(hypergeom.obs, pearson.test.stat, MAR=2) -> hypergeom.sim.stats

† Berger & Boos (1994), "P-values maximized over a confidence set for the nuisance parameter", JASA, 89, 427


Why are some major trombone solos written for the 2nd player rather than the Principal?

Question: Does anyone happen to know why it is that several major trombone solos (Rimsky-Korsakov's Russian Easter Festival Overture and Scheherazade for example) are indicated in the 2nd Trombone Part rather than the Principal?  By that point, alto trombones were no longer in general use, right?
Answer: This has long been a subject of discussion among trombone players.
Here's an attractive theory:
"I just have read in magazine that why the solo was on 2nd...
Because Russian Empelor at the Rimsky age loved music and he played the trombone on the 2nd position in certain Russian(Sankt-Peterburg?) orchestra.
So you can guess the ansewer easily, Rimsky dedicated for him play such a solo."
http://tromboneforum.org/index.php?topic=2410.5;wap2
Comment: The classic Trombone 2 solo is Tuba Mirum in the Mozart Requiem.  This was a straightforward decision to use tenor rather than alto of course.   As a jobbing trombonist I was sometimes engaged to play Trombone 1 (all the high notes) AND the solo.
Comment: I had heard a similar story, but with Rimsky-Korsakov's best friend as the second trombone player in some orchestra.  It sounds a bit like a trombonist-specific urban legend to me, to be honest.
Comment: It's an interesting explanation, although I could have sworn that I'd seen this done in other scores by non-Russians as well.  I just went back and checked on the Liszt-Doppler and saw that I was wrong; the solo was in the 1st part after all.
Comment: It just always struck me as such an odd mystery.  That, and some other strange assignments of parts, like the 4th horn solo in the 3rd movement of Beethoven 9.
Comment: There's an opinion that Beethoven's 4th horn may have owned an early valved instrument.  This is hotly disputed!  Plenty of information available from the obvious Google search.
Comment: Is it usual to honour these assignments today?  Or are the parts swapped around?
Comment: @badjohn:  Usually they are kept as written;  the voicing of the chords among the trombones is usually done conventionally (first highest, third lowest), and it'd be weird to swap these sections around.
Answer: I was just talking about this with a student this week. The answer is very straight forward and logical. The 2nd trombonist in the St Petersburg Orchestra was generally known to be having an affair with his wife. Rimsky's way of taking revenge was to write very difficult solos for the gentleman who was not particularly skilled and humiliate him on a regular basis each concert season!
Comment: Hey, welcome to the site. This answer might be improved with a source or citation?
Comment: No, it couldn't be improved!   If it turns out not to survive investigation, we are all the poorer.
Comment: Note the that there's an important difference between this version of the story and the version I posted as a comment on [Laurence Payne's answer](https://music.stackexchange.com/a/47911/21557) a couple years ago.  Probably at least one version is an urban legend.
Answer: When I recorded the Eastern Russia Festival Overture solo with Svetlanov and the Philharmonia, Svetlanov said that the 2nd trombone player in the St Petersberg Orchestra was a friend of the composer.
Comment: Welcome to Stack Exchange!


Get the value from HTML form using jquery

Question: I want to get the values if HTML form using jquery.
In my form here is two div info1 and info2 and in the jquery I have two variable store and store2
value of info1 is save in variable store and value of info2 is save in store2.
But here is the some error in the code due to this reason this is not run. see the code below.
Here is code:
<form>

         <input type="text" />
         <input type="text" />
         <input type="text" />

         <input type="text" />
         <input type="text" />

   <input type="button" id="btn" value="click" />
 </form>

JQUERY:
$(document).on('click','#btn',function(){
    var store = [];
    var store2 = [];
     $('#info1 :input[type="text"]').map(function(){
        var values = $(this).val();
       store.push({values:values});
          });
     $('#info2 :input[type="text"]').map(function(){
        var values = $(this).val();
       store2.push({values:values});
          });
     });

This code is not working. I think the error is here $('#info1 :input[type="text"]').
Please solve this.
Thankx.
Comment: Nobody is here to _solve this_. SO is not made of _freelancers_ that would code for your - and for free.
Comment: Did you try _$('#info1 input[type="text"]')_ ? (without the ":")
Comment: Aside from not using `store2` Your code appears to work just fine: http://jsfiddle.net/TrueBlueAussie/u6c4wojb/2/. What is the problem you are having?
Answer: Your code is working, to check the values in console, try this:
 $(document).on('click','#btn',function(){
     var store = [];
     var store2 = [];

     $('#info1 :input[type="text"]').map(function(){
        var values = $(this).val();
       store.push({values:values});
          });
     $('#info2 :input[type="text"]').map(function(){
        var values = $(this).val();
       store2.push({values:values});
          });

     console.log(store);
     console.log(store2);
 });


When to use non-parametric regression?

Question: I am using PROC GLM in SAS to fit a regression equation of the following form
$$
    Y = b_0 + b_1X_1 + b_2X_2 + b_3X_3 + b_4t
$$
The QQ plot of the resulting redsiduals indicate deviation from normality. Any transformation of $Y$ is not useful in making the residuals normal.
At this point, can I safely switch to non-parametric methods such as PROC LOESS.
I have already used PROC LOESS, and the fit looks better than PROC GLM. But I do not have much knowledge in non-parametric regression. I do not know when to choose non-parametric regression over parametric regression.
Can someone help me with this?
I will go ahead and add another question. Following are the description of my variables in the model.
$$
Y =\text{cost of medical care}\\
X_1 =\text{number of injections}\\
X_2 =\text{number of surgeries}\\
X_3 =\text{number of physical therapies}\\
t =\text{time}
$$
Sometimes I get negative predicted cost. This does not make sense. How can I address this issue?
Comment: You can of course avoid predicting negative costs by modelling the log of it: $log(Y) = b_0 + b_1X_1 + b_2X_2 + b_3X_3 + b_4t$
Answer: Before looking on QQplots of residuals, you should assess the quality of fit, by plotting residuals against the predictors in the model (and possibly, also against other variables you have which you did not use). Non-linearity should show up in this plots. If the effect of variable $x$ really is linear, you expect the plot of residuals against $x$ to be "horizontal", without visible structure:
                                                                   *
*                 *
      *                               *
        *
                                                  *
--------------------------------------*------------------------------x
   *
           *

                                     *
       *                                                    *
                                *

That is, a random horizontal "blob" of points, centered around the line resid=0.
If the effect is non-linear, you expect to see some curvature in this plot.
(and, please, ignore the QQplots until you got non-linearities sorted out, using plots as above!)
You should also think about possible interactions (modelled usually by product terms), that is, the effect of one variable depends on the levels of another, (If all your three variables have high values at the same time, maybe that shows some particularly difficult patient? If so, interactions could be needed).
If you go for some non-linear model, after having tried for interactions and transformations (did you try log(Cost)?) Did you try some box-cox-transformations?  Since you have multiple regression, I don't think that loess is what you need, you should look for gam (generalized additive models, SAS should have that, in R it is in package mgcv).
Comment: Thanks for the valuable information. I did try the log(cost) with the linear regression, but it did not help much. I will add the  interaction effects and observe what happens. I will also experiment with transformations again. I will keep everyone posted with my developments and findings.
Answer: A LOESS will always give a better fit than regression, unless the data truly lie along a straight line. LOESS is a locally linear approximation that is designed to pass close to the data. These methods are basically exploratory. And while it is dangerous to extrapolate a linear model beyond the limits of the fit, extrapolation would be reckless in the case of LOESS.
If your model gives you negative costs, that's a pretty good sign that a linear regression is not appropriate on the variables you have.  You say that you tried transformations. Did you take the log of cost against your predictors?
In the nature of things, it is unlikely that there is a simple relationship between cost and the variables you mention. Sometimes the purpose of a linear regression is simply to demonstrate that some sort of correlation exists, and perhaps to select a sensible set of predictors.
Comment: It makes so much sense when u mentioned that negative costs indicates that linear regression might not be appropriate. I will continue my analysis and add some interactions. Thank you.
Answer: Bravo for doing residual analysis. Puts you way ahead of the typical analyst. (Your description of the model is deficient in not describing the error structure, though.)  You should be considering transformations of the X's as well as looking at transformations of the Y's. I realize that SAS is behind R in modeling with spline fits but I understand that recent versions have offered that capacity. Consider adding restricted cubic spline fits for the X terms. As a reference Frank Harrell's text "Regression Modeling Strategies" is hard to beat. It has solid statistical arguments for this approach. It is a parametric approach that allows discovery of structure in the data that would other wise be missed.
Comment: Thanks for the appreciation DWin. I just graduated, and this is my first job as an analyst. Coincidentally, this kind of analysis is new to the company too. So, I am just trying to come up an analysis that is not completely nonsense. I will take your suggestion and try transformations on both $Y$ and $X$ variables. I will also go through the reference. I just found a pdf version online. Thanks for your input.
Comment: There is something wrong with log transforming in and outputs that intuitively depend linearly. If you model $log(Y) = b_0 +  b_1log(X_1) + b_2log(X_2)$  where $X_1$ are injections in the right arm and $X_2$ are injections in the left arm, you predict completely different costs for someone having all injections in the same arm and someone who having half of them at each side
Comment: Your comment seems rather tangential to my response (and to the question since partitioning of injections by arm was never mentioned)  I hope you don't think that spline functions are equivalent to log transformations. The log transformation of the Y creates a model where the model is multiplicative in the predictors when transformed back to the cost scale. That's a fairly big change and one whose problems you have not adequately describe to the questioner.
Answer: I think kjetil has given you some good suggestions.  I would add that non-normal residuals does not mean you have to jump from linear or nonlinear regression to nonparametric regression.  By going to nonparametric regression you give up the structure of a functional form. There are robust regression alternative to OLS regression that you could go to first.  Then generalized linear models and generalized additive models if next steps are needed.  LOESS should in my view be your last resort.  I think that I agree with kjetil on that.

When to use non-parametric regression?


Msysgit GPG error during commit signing

Question: I'm getting the following error on Windows with msysgit. Doesn't matter whether or not I am doing it from the command prompt or the BASH bundled with Git:

gpg: error loading `iconv.dll': The specified module could not be found.

Google hasn't yielded any good results yet. Any idea of how to get around this? I see libiconv2.dll in the same bin folder as gpg.exe.
Answer: See this page for the dll and how to install it
The directions are as follows:

Installing iconv.dll
To support different languages and character sets, Microsoft Windows
  versions of GnuPG make use of a DLL called "iconv.dll". This DLL has
  been written by the GNU project too and is used by several
  applications ported from GNU/Linux to MS Windows systems. If this
  library has not yet been installed on your system, an error message
  might have you taken to this page. Please follow the instructions
  below to install this library.
GnuPG has a fallback mechanism to work even without this DLL. However
  some texts may not show up correclty and when creating a new key it
  might not be encoded correctly. Thus unless you are using the Latin-1
  or UTF-8 character set, you should definitley install the DLL before
  you create a new key or add a new user ID to a key.
If you have used the Gpgwin package to install GnupG, you should never
  see this message because this DLL will get installed. However, it
  might happen that such a DLL has been insatlled on your system by
  another application and later be deinstalled. To rectify this, simply
  re-install Gpg4win and your are done.
The DLL is available as a ZIP file at the address
  ftp://ftp.gnupg.org/gcrypt/binary/libiconv-1.9.1.dll.zip. You should
  also get the corresponding ".sig" file and verify the integrity . This
  DLL has been built from the source package libiconv-1.9.1.tar.gz
  available at the same FTP sites. We distribute it separately for
  convenience and because GnuPG only requires this DLL and not the other
  files from the binary package. For copying conditions see the included
  file "COPYING.LIB".
To install this DLL, unzip it and copy the file "iconv.dll" to a
  directory where you usually keep DLLs. If you don't know such a
  directory, it is safe to copy it into the same directory as the
  "gpg.exe" file is stored.

Msysgit GPG error during commit signing


Check if a tree is a BST using a provided higher order function in OCAML

Question: So let me start by saying this was part of a past homework I couldn't solve but as I am preparing for a test I would like to know how to do this. I have these implementations of map_tree and fold_tree provided by the instructor:
let rec map_tree (f:'a -> 'b) (t:'a tree) : 'b tree =
match t with
| Leaf x -> Leaf (f x)
| Node (x,lt,rt) -> Node (f x,(map_tree f lt),(map_tree f rt))

let fold_tree (f1:'a->'b) (f2:'a->'b->'b->'b) (t:'a tree) : 'b =
  let rec aux t =
    match t with
     | Leaf x -> f1 x
     | Node (x,lt,rt) -> f2 x (aux lt) (aux rt)
  in aux t

I need to implement a function that verifies a tree is a BST using the above functions, so far this is what I've accomplished and I'm getting the error:
 Error: This expression has type bool but an expression was expected of type
     'a tree

This is my code:
 let rec smaller_than t n : bool =
 begin match t with
 | Leaf x -> true
 | Node(x,lt,rt) -> (x<n) && (smaller_than lt x) && (smaller_than rt x)
 end

 let rec greater_equal_than t n : bool =
 begin match t with
 | Leaf x -> true
 | Node(x,lt,rt) -> (x>=n) && (greater_equal_than lt x) && (greater_equal_than rt x)
 end

 let check_bst t =
 fold_tree (fun x -> true) (fun x lt rt -> ((check_bst lt) && (check_bst rt)&&(smaller_than lt x)&&(greater_equal_than rt x))) t;;

Any suggestions? I seem to have trouble understanding exactly how higher order functions work in OCAML
Comment: Look the type of the functions `f1`, `f2` of `fold_tree`. The way of calling this last function (in `check_bst`) replace the type `'b` by `bool` in `f1` but also in `f2` so the two arguments `'b` in `f2` are infered to `bool` and not a `Node` like excepted. Also add `rec` to your function `check_bst` because this function makes recursives calls.
Comment: The parameters for the "non-leaf" function of the fold are the value of the current node and the results of folding the subtrees - not the subtrees themselves.
Comment: I can't change the provided functions. Yes, I understand that the values are the results of folding the subtrees but how do I verify that the subtree it is a BST? I can obtain a bool indicating it but then I wouldn't be able to verify the other higher trees and will get an error of type mismatching
Comment: You'll need an intermediate step where you fold over something fancier than `bool` and then extract the result from the fold's result. I think it's possible to do it only knowing whether the subtrees are BST's, the greatest value in the left subtree, and the smallest value in the right subtree.
Comment: @AndresGalaviz do you have to use `fold_tree`? The algo is pretty simple without folding
Answer: What is the specification of a BST? It's a binary tree where:

all the elements in the left subtree (which is also a BST) are strictly smaller than the value stored at the node
and all the ones in the right subtree (which is also a BST) are bigger or equal than the value stored at the node

A fold is an induction principle: you have to explain how to deal with the base cases (the Leaf case here) and how to combine the results for the subcases in the step cases (the Node case here).
A Leaf is always a BST so the base case is going to be pretty simple. However, in the Node case, we need to make sure that the values are in the right subtrees. To be able to perform this check, we are going to need extra information. The idea is to have a fold computing:

whether the given tree is a BST
and the interval in which all of its values live

Let's introduce type synonyms to structure our thoughts:
type is_bst      = bool
type 'a interval = 'a * 'a

As predicted, the base case is easy:
let leaf_bst (a : 'a) : is_bst * 'a interval = (true, (a, a))

In the Node case, we have the value a stored at the node and the results computed recursively for the left (lih as in left induction hypothesis) and right subtrees respectively. The tree thus built is a BST if and only if the two subtrees are (b1 && b2) and their values respect the properties described earlier. The interval in which this new tree's values live is now the larger (lb1, ub2).
let node_bst (a : 'a) (lih : is_bst * 'a interval) (rih : is_bst * 'a interval) =
  let (b1, (lb1, ub1)) = lih in
  let (b2, (lb2, ub2)) = rih in
  (b1 && b2 && ub1 < a && a <= lb2, (lb1, ub2))

Finally, the function checking whether a tree is a BST is defined by projecting out the boolean out of the result of calling fold_tree leaf_bst node_bst on it.
let bst (t : 'a tree) : bool =
  fst (fold_tree leaf_bst node_bst t)

Comment: Thank you for providing a detailed explanation!


c# Executing Multiple calls in Parallel

Question: I'm looping through an Array of values, for each value I want to execute a long running process. Since I have multiple tasks to be performed that have no inter dependency I want to be able to execute them in parallel.
My code is:
List<Task<bool>> dependantTasksQuery = new List<Task<bool>>();

foreach (int dependantID in dependantIDList)
{
    dependantTasksQuery.Add(WaitForDependantObject(dependantID));
}

Task<bool>[] dependantTasks = dependantTasksQuery.ToArray();

//Wait for all dependant tasks to complete
bool[] lengths = await Task.WhenAll(dependantTasks);

The WaitForDependantObject method just looks like:
async Task<bool> WaitForDependantObject(int idVal)
{
    System.Threading.Thread.Sleep(20000);

    bool waitDone = true;

    return waitDone;
}

As you can see I've just added a sleep to highlight my issue.  What is happening when debugging is that on the line:
dependantTasksQuery.Add(WaitForDependantObject(dependantID));

My code is stopping and waiting the 20 seconds for the method to complete.  I did not want to start the execution until I had completed the loop and built up the Array.  Can somebody point me to what I'm doing wrong?  I'm pretty sure I need an await somewhere
Comment: You don't have any asynchronous methods. `WaitForDependantObject` blocks synchronously. `async` doesn't make a method asynchronous, it allows you to use the `await` keyword inside its body.
Comment: Please post code that compiles - `WaitForDependantObject` have no chance to be compilable as shown. Note that `Sleep` should be used to imitate long CPU intesive task, use `Task.Delay` as replacement for long IO bound work.
Comment: Should compile now, thread.sleep is purely to force a delay. This is just a sample and not my actual code, it's only used to demonstrate my issue.
Comment: I've added answer showing proper "sample of async method" (which would work with original code, still would show delays for "synchronous part" and if using WPF/WinForms all results come back to original thread).
Answer: In your case WaitForDependantObject isn't asynchronous at all even though it returns a task. If that's your goal do as Luke Willis suggests. To make these calls both asynchronous and truly parallel you need to offload them to a Thread Pool thread with Task.Run:
bool[] lengths = await Task.WhenAll(dependantIDList.Select(() => Task.Run(() => WaitForDependantObject(dependantID))));

async methods run synchronously until an await is reached and them returns a task representing the asynchronous operation. In your case you don't have an await so the methods simply execute one after the other. Task.Run uses multiple threads to enable parallelism even on these synchronous parts on top of the concurrency of awaiting all the tasks together with Task.WhenAll.
For WaitForDependantObject to represent an async method more accurately it should look like this:
async Task<bool> WaitForDependantObject(int idVal)
{
    await Task.Delay(20000);
    return true;
}

Comment: Gotcha, Luke's answer accomplishes what I need so I've accepted it. I've Up-voted this answer as it is good to know for future reference.
Comment: @keitn Then I would suggest updating your title, as your question doesn't regard `async` calls
Answer: Use Task.Delay to make method asynchronous and looking more real replacement of mocked code:
async Task<bool> WaitForDependantObject(int idVal)
{
   // how long synchronous part of method takes (before first await)
   System.Threading.Thread.Sleep(1000);

   // method returns as soon as awiting started
   await Task.Delay(2000); // how long IO or other async operation takes place

   // simulate data processing, would run on new thread unless
   // used in WPF/WinForms/ASP.Net and no call to ConfigureAwait(false) made by caller.
   System.Threading.Thread.Sleep(1000);

   bool waitDone = true;

   return waitDone;
}

Answer: You can do this using Task.Factory.StartNew.
Replace this:
dependantTasksQuery.Add(WaitForDependantObject(dependantID));

with this:
dependantTasksQuery.Add(
    Task.Factory.StartNew(
        () => WaitForDependantObject(dependantID)
    )
);

This will run your method within a new Task and add the task to your List.
You will also want to change the method signature of WaitForDependantObject to be:
bool WaitForDependantObject(int idVal)

You can then wait for your tasks to complete with:
Task.WaitAll(dependentTasksQuery.ToArray());

And get your results with:
bool[] lengths = dependentTasksQuery.Select(task => task.Result).ToArray();

Comment: Even if the calls are not asynchronous you should still you `await Task.WhenAll` instead of blocking the calling thread.
Comment: @I3arnon So, I guess I've been doing asynchronous wrong in C#. I just read [this](http://stackoverflow.com/q/6123406/2479481). It seems to me that `Task.WaitAll` *could* be appropriate *if* you need the results of your `Task`s in order to continue. Is this correct?
Comment: `Task.WhenAll` returns the results, `Task.WaitAll` doesn't. The issue here is blocking vs asynchronously waiting for completion. blocking is usually bad as it uses up a thread without executing anything and so hinders scalability.

c# Executing Multiple calls in Parallel


Output filename in views

Question: I have an uploaded file my_cat.jpg
Upon upload, the file-entity's name is by default populated with the file's filename.
Try one: In views, adding a field file:name to my view first looks like the correct answer.
But that's the file-entity's name, which can, even after the upload, be changed to eg. My cat, while the filename on the disk will never change.
Try two: If i add a file:path field, i can decide if it should output the Download-URL or the internal Filestream-Adress.
But no way to leave out the path completely...
From the field's name, that's not a big surprise.
So, how can i get just the filesystem-filename my_cat.jpg?
Comment: Are you on Drupal 7 or 8?
Comment: I am using Drupal 7
Answer: In Drupal 8: in Relationships, add a relationship to your image (mine looks like: Image, (field_image:target_id), Content, Appears in: image). In the next step select "Require this relationship". Now, in Fields, you have the filename field available.

Output filename in views


yum package conflict installing mysqlclient package

Question: I'm trying to build and install Zabbix 1.8 on my CentOS 5.2 server. This server has MySQL 5.0 installed from the standard CentoOS 5.2 repository.
./configure ... is failing because of:

configure: error: Not found mysqlclient library

When I do yum list mysqlclient* yum says there are no matching packages.
So I did a yum install mysqlclient* but get this error:
Transaction Check Error:
  file /usr/lib/mysql/libmysqlclient.so.15.0.0 from install of mysqlclient15-5.0.91-1.ius.el5.i386 conflicts with file from package mysql-5.0.77-4.el5_5.4.i386
  file /usr/lib/mysql/libmysqlclient_r.so.15.0.0 from install of mysqlclient15-5.0.91-1.ius.el5.i386 conflicts with file from package mysql-5.0.77-4.el5_5.4.i386
How do I resolve this?
Answer: Install the mysql-devel package.
Comment: Thanks, this pointed me in the right direction. mysql-devel was already installed but it seems like the configure script didn't like it. I needed to do `yum update mysql-devel` instead which got me on my way.
Answer: The mysqlclient library (libmysqlclient) is--as the error implies--already installed.  Rather, it sounds like your paths are not setup correctly, which may be because the file is in a subdirectory of lib.
Try running configure with:
./configure --with-mysql=/usr/lib/mysql ...

You can also specify the path to mysql_config instead.  If this continues to fail, you can explicitly set the library paths for configure, by setting environment variables:
LDFLAGS=/usr/lib/mysql ./configure ...

Leave feedback if this doesn't work, and good luck!
Comment: I did try both of those options but configure still failed. Ignacio's answer pointed my in the right direction. thanks for the answer though. +1
Answer: A sidenote about yum saying nothing is matches. My bet is you have a file that matches mysqlclient* in the directory you are running yum from. Try
yum list 'mysqlclient*'

Use apostrophes instead of double quotes because these interpolate and you would have the same problem.


Erlang: How do you return an element of a list?

Question: For example: [{a,b},{c,d},{e,f}]. I want to pick on the list with an argument (being c) and it will return d.
Answer: You can use few approaches or use Erlang functions from the standard library(like lists etc.) or you can create your own, eg:
List Comprehensions
1> List = [{a,b}, {c,d}, {e,f}].
2> Being = e.
3> [Result] = [Y || {X, Y} <- List, Being =:= X].
4> Result.
f

Functions
1> GetVal = fun GetVal (_, [])                -> not_found;
                GetVal (X, [{X, Result} | _]) -> Result;
                GetVal (X, [_ | T])           -> GetVal(X, T)
            end.
2> List = [{a,b}, {c,d}, {e,f}].
3> Being = e.
4> GetVal(Being, List).
f

The simple way to use Pattern Matching and List Handling.
Answer:
Write a program that prints each tuple in the list.

Write a program that prints just the second element of each tuple in the list.

Write a program that takes an argument Target along with a List.  When you find the tuple {Target, Right}, print our Right.
Comment: yea thank you, i figured it out on how to do that. i got it working thanks!
Answer: If you don't mind using functions from the standard library, you can use lists:keyfind/2 or proplists:get_value/2,3.


How to calculate F-Measure from Precision Recall Curve

Question: I have a precision recall curve for two separate algorithms. If I want to calculate the F-Measure I have to use the precision and recall values at a particular point on each curve.
How is this point decided? For example on curve one there is a point where recall is 0.9 and precision is 0.87 and the other curve there is a point of recall at 0.95 and precision at 0.84.
Alternatively, should I plot a F-measure curve for every precision recall value?
Comment: I think this has been discussed before in the setting of AUC. In this analogue F-measure=Youden Index.
Comment: When you say precision/recall curve are you referring to ROCs where sensitivity/specificity are calculated over a range of thresholds? Youden's and F-scores are functions of a single sensitivity/specificity. You would have to choose a single threshold and report values for that threshold.
Comment: Regarding your question "How is this point decided?": I think an appropriate choice would be the EER (equal error rate). This is the point where TPR and FPR cross. Have a look on this:

http://security.stackexchange.com/questions/57589/determining-the-accuracy-of-a-biometric-system
Answer: Precision-Recall curve and ROC curve (doesn't matter they are just the mirror images of each other) are used to give you the sense of the quality of the binary classifier for the different values for some parameter that affects the performance of your classifier. Now, F1 are particular scores which combine both precision and recall into a single one, so that way you just need to select the configuration of your classifier which has the highest F score.
In your place for each pair of precision and recall I would calculate F score and then pick the configuration which has the highest F score.
Now, the tricky part is which F score. F1 is the score which values precision and recall the same, but sometimes the recall is more important than precision (for example, you don't mind having a lot of people falsely tested for some cancer if you know that all of the ones who have that cancer are tested). In that case you could use F2 measure.
I think it doesn't make sense to sum up all F measures for all combinations of precision and recall. After all, the idea is to pick a single model out of the
broader range of models, I would prefer to pick a model with the highest value of F score instead of the one with the biggest sum of all F scores.
Comment: PR and ROC curves are completely different, not mirror images. PR doesn't depend on the number of True Negative but ROC does. PR isn't always monotonic but ROC is.
Answer: It is hard to read off F1 (or any other weighted F-measure) directly from a Precision-Recall graph, because of needing to work with reciprocals (harmonic mean).
But if instead you plot the reciprocal Precision & Recall, then values of the F-measure form isobars (straight lines with equal values) with gradient depending on the tradeoff parameter.  In the case of F1 they will be isobars parallel to the diagonal, corresponding to equal weighting of success in terms of precision of the positive predictions and in terms of recall of the positive cases (the number predicted may not correspond to the real number of positives).
If you use isobars directly on the Precision-Recall graph, you optimize the arithmetic mean instead - the advantage of using harmonic over arithmetic is debatable and is discussed in the reference, and there is some evidence that geometric is better than either.
If you plot Precision & Recall logarithmically, then the isobars can be used to optimize this geometric mean.
Note that the logarithmic and reciprocal PR graphs are not defined at the points corresponding to no positives are present or none are predicted. At all other points these curves are equivalent to the ROC curve in that if a solution is better ranked in one it will be better ranked in the other.
ROC curves are however a bit different as the compare TPR (Recall) against FPR (Fallout) but again isobars are useful and those parallel to the diagonal correspond to equal weighting of positives and negatives. In particular Precision and Recall are independent of the number of True Negatives (correctly predicted negatives) but this is a complmenetary component of Fallout - in fact a mirror image graph is formed by plotting TPR vs TNR. The difference TPR-FPR is also known as Youden J or Informedness, and is linearly related to the area under the curve formed by a specific operating point.
This is discussed in detail in a report of mine I have uploaded, with the graphs I've discussed shown in Figure 3 for some real data relating to facial expression recognition:
https://www.researchgate.net/publication/273761103_What_the_F-measure_doesn%27t_measure
Discussion of ROC Area Under the Curve is here:
https://www.researchgate.net/publication/261155937_The_problem_of_Area_Under_the_Curve?ev=pub_cit


Unselect Radiobutton

Question: I can not remove the selection from radio buttons
As shown in the image, I tried the following :
Patch(GUAT, ThisItem, {Radio: Blank()}); UpdateContext({lclResetRadios:true})
as stated here:
https://powerusers.microsoft.com/t5/Building-Power-Apps/Reset-and-deselect-radio-buttons/td-p/668512
but it did not work
Answer: What I did was put a blank label around the button control (be careful not to cover the actual button where the user click to select the radio button, but outside that area), and used reset(radiobuttonname). Works beautifully. The user clicks the button and anywhere else outside the area it will deselect. Regards!

Unselect Radiobutton


Inline Template with Render Array in #context

Question: I'm using the https://api.drupal.org/InlineTemplate to render a string with a variable, and that variable is itself another theme function.
My reading of the documentation is that this is supported; based on the fact that the #context variable can be a string or render array.

#context: (array) The variables to substitute into the Twig template. Each variable may be a string or a render array.

However, my implementation is not working:
    $my_subtemplate = [
      '#type' => 'inline_template',
      '#template' => '{% trans %}my {{ foo }} here{% endtrans %}',
      '#context' => ['foo' => 'bar'],
    ];

    $my_string_with_embedded_themable = [
      '#type' => 'inline_template',
      '#template' => '{% trans %}Print this string and this subtemplate: {{ my_subtemplate }}.{% endtrans %}',
      '#context' => [
        'my_subtemplate' => $my_subtemplate,
      ],
    ];

The generated output is:

Print this string and this subtemplate: .

I've confirmed that my_subtemplate generates the correct output on its own:

my bar here

But when its placed into a variable of another inline template, it does not render.
Comment: Even if this worked it would not be a valid use case for interface translation, which is supposed to be used for string constants and not for strings after variable placeholders got replaced. And in inline templates defined in PHP you don't need Twig methods to translate because you can use t().
Comment: I get your point that you could use t(), but otherwise, this comment ^ doesn't make sense.  The example in the core documentation linked above uses the trans Twig function, and the trans Twig functions itself supports variables as placeholders, for position purposes.  IMHO the bug here is that the trans Twig function doesn't support render array variable expansion, while other Twig functions do, eg spaceless.
Comment: Regarding position purposes, I mean 'Hello {{name}}'  might be translated in another language by putting the name first, so the translation in some other language would be '{{name}} ASDF' (where ASDF is the translation of Hello).
Comment: To make it work I would use t() instead of renderRoot. Depending on where you place this code it is bad for caching and attachments or can cause a nested renderRoot error.
Answer: The issue not in the inline_template but the issue comes from {% trans %} if you place a render array variable into {% trans %}{% endtrans %} you will get the following error and it will not rendered:

Warning: htmlspecialchars() expects parameter 1 to be string, array given in Drupal\Component\Utility\Html::escape()

So you have two options:
1) get the variable out of {% trans %}{% endtrans %} like the following:
    $my_subtemplate = [
      '#type'     => 'inline_template',
      '#template' => '{% trans %} my {{ foo }} here {% endtrans %}',
      '#context'  => [
        'foo' => 'bar'
      ],
    ];
    $my_string_with_embedded_themable = [
      '#type'     => 'inline_template',
      '#template' => '{% trans %} Print this string and this subtemplate: {% endtrans %} {{ my_subtemplate }} .',
      '#context'  => [
        'my_subtemplate' => $my_subtemplate,
      ],
    ];

2) Pre-render the renderable array variable to a string first.
    $my_subtemplate = [
      '#type'     => 'inline_template',
      '#template' => '{% trans %} my {{ foo }} here {% endtrans %}',
      '#context'  => [
        'foo' => 'bar'
      ],
    ];

    $renderer = \Drupal::service('renderer');
    $my_subtemplate = $renderer->renderRoot($my_subtemplate); // this fixes it

    $my_string_with_embedded_themable = [
      '#type'     => 'inline_template',
      '#template' => '{% trans %} Print this string and this subtemplate: {{ my_subtemplate }}.{% endtrans %}',
      '#context'  => [
        'my_subtemplate' => $my_subtemplate,
      ],
    ];

Comment: wow,  that is very interesting.  Thanks for the quick response. I've added a comment to the documentation (link in OP) and expanded your answer to provide the other alternative -- to render the renderable to string first.
Comment: You are welcome, i'm glad that helped, great i read your comment in documentation, good job!
Answer: I like to also use inline_template, as a way to escape from the limitations of the drupal render arrays.
I would solve it with a single include to have templates as proper twig templates and not mixed with the logic (MVC approach + better syntax highlighting):
$my_string_with_embedded_themable = [
  '#type' => 'inline_template',
  '#template' => "{% include '@myTheme/path/my-template-file.html.twig' %}",
  '#context' => [
     'foo' => 'bar',
     'my_subtemplate' = 'my_subtemplate'
  ]
]

And then 'web/themes/myTheme/templates/path/my-template-file.html.twig':

{% trans %}Print this string and this subtemplate:{% endtrans %}
{% include '@myTheme/' ~ my_subtemplate %}.

The above example does not really require the include in the include, but the real approach is probably more complex anyway.
so 'web/themes/myTheme/templates/' + my_subtemplate:
{% trans %} my {{ foo }} here {% endtrans %}


Uploading a file to AWS S3 with ACL set to public_read

Question: In my Rails app I save customer RMA shipping labels to an S3 bucket on creation.  I just updated to V2 of the aws-sdk gem, and now my code for setting the ACL doesn't work.
Code that worked in V1.X:
  # Saves label to S3 bucket
  s3 = AWS::S3.new
  obj = s3.buckets[ENV['S3_BUCKET_NAME']].objects["#{shippinglabel_filename}"]
  obj.write(open(label.label('pdf').postage_label.label_pdf_url, 'rb'), :acl => :public_read)

.write seems to have been deprecated, so I'm using .put now.  Everything is working, except when I try to set the ACL.
New code for V2.0:
  # Saves label to S3 bucket
  s3 = Aws::S3::Resource.new
  obj = s3.bucket(ENV['S3_BUCKET_NAME']).object("#{shippinglabel_filename}")
  obj.put(Base64.decode64(label_base64), { :acl => :public_read })

I get an Aws::S3::Errors::InvalidArgument error, pointed at the ACL.
Comment: What's the full error message?
Comment: Aws::S3::Errors::InvalidArgument is the bulk of the message.  The console just points towards my Model at the obj.put line.  Can post a screenshot if that would be helpful.
Comment: I was wondering if it was some hint about the error, such as the format of the argument or that not being allowed or such. No worries.
Answer: This code works for me:
photo_obj = bucket.object object_name
photo_obj.upload_file path, {acl: 'public-read'}

so you need to use the string 'public-read' for the acl.  I found this by seeing an example in object.rb
Comment: You rock - thanks!  I can't believe it was just a hyphen instead of an underscore.  I actually switched back to the old API because of this - but now I know how to upgrade.  Thanks!
Comment: Ack!  Same here -- I had an underscore, too.  Double-checked the official docs three times without catching my error!
Comment: yep, the hyphen works, *thank you*. I'm using Python/boto3 and had the same problem, now mostly solved. still problem with mime-type, but I'll figure it out.
Comment: Why isn't this the accepted answer? Was stuck for hours until i stumbled across this answer.


Drupal Commerce Market Place

Question: I cannot find a link to add seller/vendor/store in the Commerce-Marketplace module.
I am using Drupal Commerce Kickstart.
Answer: As far as I know, there is not a direct solution for this. I have searched google for this,couldn't find a solution. But I will share what I think is correct.
The store in commerce market place is actually similar to an organic group, where you can add members and products.
The store is implemented as an entity. If you go to admin/structure/entity-type/commerce_store/store and add a store, you have created a store.
Now, if you try to add a product, you can type the name of the store that you had created, which will auto-fill for you.
Regarding the vendors, you may have to  create a role and give permission to create stores for that role.
Let me know how that goes.

Drupal Commerce Market Place


Local Server with external IP not reachable when in LAN

Question: I have a local debain server with external IP and my own domain. But I got following problem:

Using DNS/external IP from outside the LAN: Works
Using local IP from inside the LAN: Works
Using DNS/externa IP from inside: Does not work -> Timeout

I don't receive any message when connection to the domain using telnet. But ping and traceroute does work.
When visiting via browser no message is displayed. As the server logs nothing I don't know where to start searching.
Here is one of my nginx configs:
<code>server {
    listen 443;
    ssl on;
    server_name wiki.*;
    ssl_certificate     /etc/letsencrypt/live/admin.domain.de    /fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/admin.domain.de/privkey.pem;

    set $root_path "/home/wiki/dokuwiki";
    root $root_path;

    access_log /var/log/nginx/dokuwiki-access.log;
    error_log /var/log/nginx/dokuwiki-error.log;

    index index.php index.html doku.php;
    location ~ /(data|conf|bin|inc)/ {
         deny all;
    }
    location ~ /\.ht {
         deny  all;
    }
    location ~ \.php {
         fastcgi_index index.php;
         fastcgi_split_path_info ^(.+\.php)(.*)$;
         include /etc/nginx/fastcgi_params;
         fastcgi_pass unix:/var/run/php5-fpm.sock;
         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
   }
 }
</code>
Thank you!
Comment: If ping works, could it be a firewall issue?
Comment: The firewall on the server is empty. In the router all necessary ports are open. Connecting from the outside works
Comment: When you ping/traceroute does it resolve the Ip to the external ip address or the internal one. What is doing your Nat? Are you trying to access the server from another device on the same network, or from the server to the server
Comment: I tried accessing from another pc, laptop and mobile phone in the same local network. The IP resolving to the external one
Comment: Could you share your server's network configuration?
Comment: Here is one config I use https://pastebin.com/uR3E7XDD
Comment: Please edit your question to include the relevant portions of your config.
Comment: I second the request for the network configuration. Especially reading "FritzBox" makes me wonder if your server has an external IP at all or if you're talking about a port forward from the router's external IP address.
Answer: Your router either doesn't support hairpin NAT, or it does support it and it's not enabled.
Comment: I have an AVM fritz.box with OS 6.5. Do you now how to setup there? I found https://en.avm.de/service/fritzbox/fritzbox-7581/knowledge-base/publication/show/663_No-DNS-resolution-of-private-IP-addresses/ and set it up but no changes
Answer: It sounds like you have not set a route within your LAN to the public IP. A default gateway/route as the private IP would cover it usually as the packet would go to the private IP to be routed and the machine would look up in the routing tables where to route your packet next and since the public IP is a local interface IP on the gateway, then it would get routed straight to that interface. If the machine is not your default route then you should add a static route in your gateway or on an individual client for the public IP as the private IP of the machine with the public IP in question. You will also need IP forwarding enabled.
This would be the typical first place to look if you are sure it is not a firewall issue. "Is there a route to the IP?" If you just put a public IP on a machine on your network that is not the default gateway, there will be no knowledge in how to access it.


Forecast with auto Arima, with long term trend line, the 30 day forecast "jumps"

Question: I'm trying to create a 30 day forecast using auto.arima from the forecast package. I want to capture the long term trend, so I inserted it into the xreg argument.
The data:
dput(data)
structure(list(TKDate = structure(c(15706, 15707, 15708, 15709,
15710, 15711, 15712, 15713, 15714, 15715, 15716, 15717, 15718,
15719, 15720, 15721, 15722, 15723, 15724, 15725, 15726, 15727,
15728, 15729, 15730, 15731, 15732, 15733, 15734, 15735, 15736,
15737, 15738, 15739, 15740, 15741, 15742, 15743, 15744, 15745,
15746, 15747, 15748, 15749, 15750, 15751, 15752, 15753, 15754,
15755, 15756, 15757, 15758, 15759, 15760, 15761, 15762, 15763,
15764, 15765, 15766, 15767, 15768, 15769, 15770, 15771, 15772,
15773, 15774, 15775, 15776, 15777, 15778, 15779, 15780, 15781,
15782, 15783, 15784, 15785, 15786, 15787, 15788, 15789, 15790,
15791, 15792, 15793, 15794, 15795, 15796, 15797, 15798, 15799,
15800, 15801, 15802, 15803, 15804, 15805, 15806, 15807, 15808,
15809, 15810, 15811, 15812, 15813, 15814, 15815, 15816, 15817,
15818, 15819, 15820, 15821, 15822, 15823, 15824, 15825, 15826,
15827, 15828, 15829, 15830, 15831, 15832, 15833, 15834, 15835,
15836, 15837, 15838, 15839, 15840, 15841, 15842, 15843, 15844,
15845, 15846, 15847, 15848, 15849, 15850, 15851, 15852, 15853,
15854, 15855, 15856, 15857, 15858, 15859, 15860, 15861, 15862,
15863, 15864, 15865, 15866, 15867, 15868, 15869, 15870, 15871,
15872, 15873, 15874, 15875, 15876, 15877, 15878, 15879, 15880,
15881, 15882, 15883, 15884, 15885, 15886, 15887, 15888, 15889,
15890, 15891, 15892, 15893, 15894, 15895, 15896, 15897, 15898,
15899, 15900, 15901, 15902, 15903, 15904, 15905, 15906, 15907,
15908, 15909, 15910, 15911, 15912, 15913, 15914, 15915, 15916,
15917, 15918, 15919, 15920, 15921, 15922, 15923, 15924, 15925,
15926, 15927, 15928, 15929, 15930, 15931, 15932, 15933, 15934,
15935, 15936, 15937, 15938, 15939, 15940, 15941, 15942, 15943,
15944, 15945, 15946, 15947, 15948, 15949, 15950, 15951, 15952,
15953, 15954, 15955, 15956, 15957, 15958, 15959, 15960, 15961,
15962, 15963, 15964, 15965, 15966, 15967, 15968, 15969, 15970,
15971, 15972, 15973, 15974, 15975, 15976, 15977, 15978, 15979,
15980, 15981, 15982, 15983, 15984, 15985, 15986, 15987, 15988,
15989, 15990, 15991, 15992, 15993, 15994, 15995, 15996, 15997,
15998, 15999, 16000, 16001, 16002, 16003, 16004, 16005, 16006,
16007, 16008, 16009, 16010, 16011, 16012, 16013, 16014, 16015,
16016, 16017, 16018, 16019, 16020, 16021, 16022, 16023, 16024,
16025, 16026, 16027, 16028, 16029, 16030, 16031, 16032, 16033,
16034, 16035, 16036, 16037, 16038, 16039, 16040, 16041, 16042,
16043, 16044, 16045, 16046, 16047, 16048, 16049, 16050, 16051,
16052, 16053, 16054, 16055, 16056, 16057, 16058, 16059, 16060,
16061, 16062, 16063, 16064, 16065, 16066, 16067, 16068, 16069,
16070, 16071, 16072, 16073, 16074, 16075, 16076, 16077, 16078,
16079, 16080, 16081, 16082, 16083, 16084, 16085, 16086, 16087,
16088, 16089, 16090, 16091, 16092, 16093, 16094, 16095, 16096,
16097, 16098, 16099, 16100, 16101, 16102, 16103, 16104, 16105,
16106, 16107, 16108, 16109, 16110, 16111, 16112, 16113, 16114,
16115, 16116, 16117, 16118), class = "Date"), spend = c(7984.39,
11476.06, 6555.57, 3981.45, 3963.83, 4827.72, 6309.32, 13503.36,
17075.89, 33353.71, 29324.34, 7968.68, 5540.63, 12113.45, 15596.38,
19328.67, 20224.68, 18977.55, 16128.27, 10633.56, 11887.79, 17881.11,
12613.46, 11607.55, 38232.11, 7861.25, 9397.88, 12056.02, 15115.87,
12275.93, 14537.35, 9594.26, 8215.83, 9632.52, 9993.15, 13478.37,
28509.38, 12016.33, 8907.76, 8757.43, 9513.09, 10299.5, 10385.03,
12515.62, 9008.95, 17825.68, 9320.47, 11189.58, 12902.31, 13341.35,
18675.32, 16989.53, 10114.53, 9876.65, 11203.39, 11718.73, 26264.95,
12414.19, 12275.16, 9242.85, 8883.97, 10095.72, 11581.55, 14815.78,
25064.12, 9297.07, 8047.91, 6876.37, 8881.63, 10982.85, 9975.33,
24124.62, 8514.66, 15719.84, 5807.39, 8422.38, 15184.95, 14757.58,
11087.61, 11070.78, 10425.67, 15517.8, 11257.69, 11915.47, 11720.37,
34064.62, 6493.41, 5757.4, 4387.54, 6520.58, 7806.81, 6356.63,
10916.36, 9013.43, 9722.41, 6044.25, 7971.7, 23933.54, 8627.85,
9722.77, 18660.13, 13011.36, 11445.11, 14219.2, 17138.92, 16016.68,
11434, 31379.03, 8494.25, 12493.85, 7708.1, 21583.05, 9026.17,
9379.35, 8287.13, 7298.16, 6097.03, 8076.57, 12871.87, 11346.89,
9115.82, 7737.98, 15065.38, 5262.73, 6522.58, 12743.94, 23945.16,
16109.26, 6985.89, 6345.08, 6246.93, 6824.66, 8491.42, 9654.99,
18976.58, 19565.68, 8075.47, 7219.79, 8629.04, 12491.64, 11915.89,
27533.16, 13554.35, 10102.21, 20029.15, 11641.82, 15855.19, 14139.17,
15376.63, 14625.99, 9098.87, 9396.64, 12015.84, 17532.75, 15131.65,
15815.5, 16048.65, 9769.63, 9582.12, 11201.8, 12810, 18857.38,
11822.71, 19289.08, 8911.29, 9437.55, 10987.14, 12995.65, 16675.26,
9741.82, 9723.57, 10328.24, 7738.04, 8432.16, 23021.73, 10367.28,
8210.53, 10468.4, 8024.25, 7296.25, 7445.34, 8539.59, 12386.23,
15335.72, 9013.49, 7994.95, 7759.46, 8789.38, 11242.38, 28653.23,
9750.96, 14398.62, 9248.74, 6766.08, 8159.14, 9899.38, 9453.35,
17588.96, 8958.16, 8256.61, 6240.4, 7235.24, 23841.62, 9002.73,
11839.47, 8693.31, 7161.37, 7046.39, 9221.53, 10004.93, 8698.76,
7948.68, 9013.27, 18536.68, 7980.38, 8968.95, 23594.14, 17744.66,
12615.73, 13646.05, 10512.58, 9066.02, 9665.15, 13183.2, 23864.45,
12017.52, 10831.07, 8954.76, 7276.41, 7882.9, 16616.41, 15384.68,
11046.53, 10621.01, 8094.74, 5451.26, 6237.79, 10717.69, 7076.38,
7044.62, 7047.45, 7774.77, 6496.21, 6340.9, 7110.53, 7691.28,
17482.02, 5576.19, 3763.79, 11477.68, 5710.5, 6519.51, 20022.61,
13153.68, 6526.28, 5885.28, 5656.17, 6270.04, 9795.38, 6320.95,
5741.98, 10808.72, 5150.87, 5416.52, 6305.05, 20953.12, 6569.02,
6360.21, 9376.68, 4973.93, 5034.48, 6380.45, 15307.28, 14386.65,
17705.88, 4779.52, 4784.79, 4737.05, 5350.28, 12112.11, 13153.72,
6049.69, 5430.46, 4627.59, 3637.2, 5482.43, 16705.15, 12221.16,
13198.88, 6484.54, 5590.86, 4979.09, 5771.75, 7311.92, 16111.86,
8047.77, 11706.91, 6042.14, 5670.74, 6905.07, 11261.89, 9700.4,
6643.03, 5693.85, 14778.67, 9128.14, 3682.01, 7911.5, 17742.85,
5093.31, 7867.97, 3202.78, 2843.35, 2598.77, 10930.81, 11204.67,
7289.62, 4000.17, 4178.89, 4507.33, 6671.48, 10317.48, 9368.98,
6156.41, 8375.24, 2762.76, 2457.59, 4707.51, 4584.52, 3749.82,
11667.82, 4271.67, 3614.3, 3715.83, 4510.57, 4872.36, 21805.71,
4757.04, 6515.92, 2834.25, 2685.19, 3509.28, 4479.35, 17817.99,
10357.67, 3412.15, 3044.95, 2840.24, 3348.91, 13671.68, 2027.42,
1616.25, 1177.73, 995.25, 1062.25, 1578.07, 1649.8, 1410.06,
1592.03, 3995.24, 6489.87, 6895.21, 8298.58, 7698.68, 5782.07,
7671.08, 19539.4, 7023.84, 6509.9, 6643.28, 19850.3, 6856.67,
13142.15, 5524.75, 5063.2, 4916.81, 6117.54, 6717.86, 9393.95,
10462.44, 10511.15, 4497.94, 4038.31, 5503.91, 5554.82, 5801.11,
12992.82, 4778.61, 4067.41, 4359.53, 6148.1, 9236.51, 5773.16,
11313.13, 4702.37, 4167.3, 4067.75, 4469.11, 9278.41, 9911.18,
5161.13, 4477.78, 4459.53, 4080.14, 5084.67, 7735.34, 10676.6,
5507.86, 8286.12, 4332.23, 4737.52, 5952.09, 7134.44)), .Names = c("TKDate",
"spend"), row.names = c(NA, 413L), class = "data.frame")

The code:
library(forecast)
explaining<-rep(1:length(data$TKDate))
predic<-rep((length(data$TKDate)+1):(length(data$TKDate)+31))
modArima <- auto.arima(data[,2],xreg=explaining)
fit<-forecast(modArima,h=30,xreg=explaining,newdata=predic)
plot(fit)

I get this weird jump:

Can anyone explain to me this weird jump? Why doesn't the forecast continue from the last observed data point (or at least close to it)?
Answer: This is a hard to find error, I'll admit.
forecast.Arima() takes the new values of the external regressors not in a newdata parameter (as does predict.lm()), but in the xreg parameter. So instead of
fit <- forecast(modArima,h=30,xreg=explaining,newdata=predic)

where you forecasted using the values of explaining, not those of predic (unfortunately, forecast.Arima() does not throw a warning if you feed data to the nonexistent newdata parameter), do this:
fit <- forecast(modArima,h=30,xreg=predic)

and plot (with in-sample fits thrown in for good measure - EDIT: somewhat confusingly, the in-sample fits are not returned by auto.arima() or arima() as they are by lm(), but by forecast.Arima()):
plot(fit)
lines(fit$fitted,col="red")

Comment: I must say that inserting new data  into `xreg` and not into `newdata` is quit confusing, especially if your used to use `predict`, thanks again!


rails server not starting with cucumber/capybara

Question: I'm running test's in a new environment that we call as "stub". Using capybara with cucumber-rails. The rails server is not starting, by default. "config.run_server" is set to true.  I tried running in "test" environment too, but the server is not starting. Any help much appreciated.
Comment: same issue posted in capybara page, again redirected here https://github.com/jnicklas/capybara/issues/1202
Answer: I'm using cucumber-rails. I have to add the following lines in Cucumber environment file, "feature/support/env.rb"
<code>require File.expand_path('../../../config/environment', FILE)
require ("rspec/rails")
</code>
Only then capybara, gets hold of the application.


Why thwos this ERR: Unknown authentication strategy "loca"

Question: When I using passprot.js it throws this Error Unknown authentication strategy "local" !
config/configuration.js
var passport = require('passport')
  , LocalStrategy = require('passport-local').Strategy;
  var User = require('../model/students');

passport.use('local', new LocalStrategy(
  function(username, password, done) {
    User.findOne({ username: email }, function(err, user) {
      if (err) { return done(err); }
      if (!user) {
        return done(null, false, { message: 'Incorrect username.' });
      }
      if (!user.validPassword(password)) {
        return done(null, false, { message: 'Incorrect password.' });
      }
      return done(null, user);
    });
  }
));

index.js
router.get('/test', function (req, res, next) {
  res.render('test');
});

router.post('/test',
  passport.authenticate('local', { successRedirect: '/',
                                   failureRedirect: '/test',
                                   failureFlash: true })
);

Answer: Remove the local from the passport middleware.
Example:
passport.use(new LocalStrategy(
  function(username, password, done) {
    User.findOne({ username: email }, function(err, user) {
      if (err) { return done(err); }
      if (!user) {
        return done(null, false, { message: 'Incorrect username.' });
      }
      if (!user.validPassword(password)) {
        return done(null, false, { message: 'Incorrect password.' });
      }
      return done(null, user);
    });
  }
));

Hope this will solve the issue.


Como fazer duas exibições em uma view com angularJS?

Question: Tenho um app de chat e o problema que estou tendo é o seguinte:
Ao enviar uma mensagem, ela demorando um certo tempinho pra ser enviada e aparecer na tela, pois ela vai pra um banco de dados, via backend em php que eu fiz, e depois volta por um outro php que busca a mensagem e exibe na tela.
Então uma solução que pensei, pra quebrar o galho, foi a seguinte, exibir as minhas mensagens, direto na tela, como nesse exemplo e só depois envio pro backend..
Já tenho uma tela que exibe as mensagens vindas do banco de dados, mas como fazer para exibir, nessa mesma tela, as mensagens que eu envio?
Minha view:

{{mensagem.usuario}}

 {{mensagem.msg}}

{{mensagem.hora}}

Controller:
$scope.enviarMsgChat = function (mensagem) {
    //console.log(mensagem);
    $scope.disableButtonChat = true;

    var dia = moment().format(); //2016-02-16 T 16:05:52-02:00
    var diaP = dia.split('T');
    var dia = diaP[0];

    var horaP = diaP[1];
    var horaP2 = horaP.split(':');
    var hora = horaP2[0]+':'+horaP2[1];

    var idUsuario = $window.localStorage.getItem('idUsuario');
    var nome = $window.localStorage.getItem('nome');
    var usuario = $window.localStorage.getItem('usuario');
    var msgChat = {
        idUsuario: idUsuario,
        nome: nome,
        usuario: usuario,
        mensagem: mensagem,
        dia: dia,
        hora: hora
    }

    $http.post("http://www.vigilantescomunitarios.com/www/php/enviaMsgChat.php", msgChat).then(function (data) {
        //console.log(data);
        $scope.disableButtonChat = false;
        pegaMsgsChat();
        $scope.mensagem = {
          chat: ""
        }

    })
}

Comment: A função pegaMsgsChat() traz todas as mensagem do chat, ou somente a ultima?
Comment: Ambas... Todas e a última.
Comment: Já pensou em colocar a mensagem nova direto no seu array mensagem ? O array que está no loop é o mensagem, então o que vc deveria fazer é por a sua mensagem também no array.
Comment: Pois é @AndréVicente, mas to apanhando pra fazer isso.
Comment: Tipo $scope.mensagens.push(msgChat). Coloca a parte do código onde vc seta o array de mensagens. Se ela for global vc consegue usar na função que recebe os valores e depois usa para adicionar a nova mensagem nela.
Comment: Coloca a função `pegaMsgsChat() ` no problema que resolve isso ae rapidinho
Answer: Pega a ultima mensagem e adiciona no final do seu vetor de mensagens do angular.
Assim:
$scope.enviarMsgChat = function (mensagem) {
    var idUsuario = $window.localStorage.getItem('idUsuario');
    var nome = $window.localStorage.getItem('nome');
    var usuario = $window.localStorage.getItem('usuario');
    var idChat = 0;
    var dia = moment().format(); //2016-02-16 T 16:05:52-02:00
    var diaP = dia.split('T');
    var dia = diaP[0];

    var horaP = diaP[1];
    var horaP2 = horaP.split(':');
    var hora = horaP2[0]+':'+horaP2[1];

    var msgChat = {
        idChat: idChat,
        idUsuario: idUsuario,
        nome: nome,
        usuario: usuario,
        msg: mensagem,
        data: dia,
        hora: hora
    }

    $scope.mensagens.push(msgChat);

    $http.post("http://www.vigilantescomunitarios.com/www/php/enviaMsgChat.php", msgChat).then(function (data) {
    //console.log(data);
        $scope.disableButtonChat = false;
        pegaMsgsChat();
        $scope.mensagem = {
           chat: ""
        }

    })
}


py.test: how to forbid to load conftest files in upper directories

Question: I have structure like this:
conftest.py
dir/conftest.py
dir/test_my.py

When I change directory to dir and run py.test test_my.py, my tests start with configuration, declared in the outer conftest.py (which is out of dir). Is there any way to change pytest behaviour: to load conftest-file only from the local directory?
Answer: You can set the confcutdir option to specify where pytest should stop searching for conftest.py files.


How to: compare array with string and create hastags before tweeting

Question: Let's say I have the following array (which is the returned value of a database query):
Array ( [0] => PHP [1] => Webdesign [2] => Wordpress [3] => Drupal [4])

And the following string:

Working With Wordpress Shortcodes

How can I compare the array with the string to see if the string contains any word stored in the array? (hopefully that made sense to you :d )
When he finds a match (eg: Wordpress) it should create a hashtag like so:

Working With #Wordpress Shortcodes
Answer: Use preg_replace:
$tags = array('PHP', 'Webdesign', 'Wordpress', 'Drupal', 'SQL');
$text = 'Working With Wordpress Shortcodes and doing some NoSQL and SQL';

$regex = '/\b('.implode('|', array_map('preg_quote', $tags)).')\b/i';
$result = preg_replace($regex, '#$1', $text);

Would output:

Working With #Wordpress Shortcodes and doing some NoSQL and #SQL

Live example:

http://codepad.org/SxchtXmI
Comment: nice thanks, I'll have a look at it later. I did use preg_replace but not properly i think
Answer: Loop over the array and run preg_replace over the string for each value (making sure to check for word boundaries in the regular expression (e.g. so car doesn't match cartoon)
Answer: Simply loop through your array and use strpos() (or its case-insensitive version stripos()):
$tags = array('PHP', 'Webdesign', 'Wordpress', 'Drupal');
$message = "Working With Wordpress Shortcodes";

foreach ($tags as $tag) {
    $position = strpos($message, $tag);

    if ($position !== false) {
        // tag found! add # in front.
        $message = substr($message, 0, $position).'#'.substr($message, $position);
    }
}

Comment: Using strpos has the negative effect that it will match partial words, e.g. `SQL` would match `MySQL` and that would become `My#SQL`. Using regular expressions with word boundaries would work a lot better and produce the expected output.
Answer: <?
$working_array = array('PHP','Webdesign','Wordpress','Drupal');
$string = "Working With Wordpress Shortcodes";
foreach($working_array as $array_item)
{
    $the_pos = stripos($string,$array_item);
    if($the_pos !== false)
    {
        $string = substr($string,0,$the_pos) . '#' . substr($string,$the_pos);
    }
}
echo $string;
?>

Edited because I accidentally mixed some C syntax in!  Sorry!
Answer: May take much processing time (I guess) if your array is big
$hashtags = array('Wordpress', 'Twitter', 'Joomla', 'Google');
$content = "A tweet containing wordpress and joomla";

$content_breakdown = explode(' ', $content);
foreach ($hashtags as $hashtag) {

    foreach ($content_breakdown as & $word) {
        if (strtolower($word) == strtolower($hashtag))
            $word = '#' . $word;
    }
}

$content = implode($content_breakdown, ' ');
echo $content;

Output
A tweet containing #wordpress and #joomla

Answer: $hashtags = array('Wordpress', 'Joomla', /* ... */ );
$tweet = "This is a tweet containing Wordpress and Joomlazoom as hashtag";

$hashtags_tmp = implode('|', $hashtags);
$tweet = preg_replace("/\b($hashtags_tmp)\b/i", '#$1', $tweet);


Power calculation for two-sample Welch's t test

Question: The R function power.t.test does power calculations (outputs power, sample size, effect size, or whichever parameter you leave out) for t-tests, but only has a single parameter for sample size. The pwr package has a function pwr.t2n.test that performes calculations for a two-sample t-test with different sample sizes (n1,n2). Finally, this suite of stats functions includes a function for Welch's t test (used for samples with different variances), but only includes one parameter for the sample sizes.
I have not been able to find a formula for calculating the power for Welch's t.
Could someone help me out with the formula or an R function for this?
Comment: @COOLSerdash I link to that package in my question. It does not have options for different sample sizes.
Comment: The package [`MKmisc`](https://cran.r-project.org/package=MKmisc) has a function called `power.welch.t.test`.
Answer: Comment: First, I would suggest you consider carefully whether you have a really good reason to use different sample sizes. Especially if the smaller sample size is used for the group with the larger population, this is not an efficient design.
Second, you can use simulation to get the power for various scenarios. For example, if you use $n_1 = 20,\,
\sigma_1 = 15,\,$ $n_2 = 50, \sigma_2 = 10,$ then you have about 75% power for detecting a difference $\delta = 10$ in population means with a Welch test at level 5%.
n1=20; n2=50;  sg1=15; sg2=10;  dlt=10
set.seed(619)
pv = replicate(10^6,
       t.test(rnorm(n1,0,sg1),rnorm(n2,dlt,sg2))$p.val)
mean(pv <= .05)
[1] 0.753043

Because the P-value is taken directly from the procedure t.test in R, results should be accurate to 2 or 3 places, but this style of simulation runs slowly (maybe 2 or 3 min.) with
a million iterations.
You might want to use 10,000 iterations if you are doing repeated runs for various sample sizes, and then use a larger number of iterations to verify
the power of the final design.
Changing to $n_2 = 20$ gives power 67%, so the extra 30 observations in Group 2 are not 'buying' you as much as you might hope. By contrast, a balanced design with
$n_1 = n_2 = 35$ gives about 90% power (with everything else the same).
Comment: The reason I showed you a simulation is that I am not sure it is possible to give a simple formula for the exact power of the Welch t test. Given information has to be used to estimate the adjusted degrees of freedom to be used, which is used in turn to approximate the relevant noncentral t distribution. // Of course, the best time to do a power-sample size computation is _before_ you do the experiment. I suppose the main reason it is hard to find programmed power procedures for unbalanced Welch tests is that most investigators try to avoid unbalanced designs.
Comment: For us, what can happen is that we get a load of patients, some with the disease and some without. We can try to make sure we get enough balance of some groups (case/control, gender, age, etc.) But we will still end up with unbalanced samples. For example after we genotype 100 patients, we may find that in the case subgroup, for one risk factor, we end up with 10 homozygous risk, 3 homozygous non-risk, and the rest heterozygous. If we had funding and time to subscribe 1000 people in the study, we wouldn't have a sample size problem. But that's just how it can shake out.
Comment: thanks for the ideas. For my situation, the experiment has been done. We have 40 microbiota and a multitude of covariates which we test in various subgroups. After multiple testing, we might select the biota having the most significance. Sometimes the risk category for a disease might have 10 patients but the non-risk only 3. Power analysis will help me know if I can trust the p-value. In this situation, simulation would be very inefficient given that the calculation can be done directly--assuming one has the formula or a function.
Answer: The article "Optimal sample sizes for Welch's test under various allocation and cost considerations" from Show-Li Jan & Gwowen Shieh published in Behavior Research Methods in December 2011 has the following code in supplementary material A, slightly modified here for my own ergonomy.
ssize.welch = function(alpha=0.05, power=0.90, mu1, mu2, sigma1, sigma2, n2n1r, use_exact=FALSE)
 {
  mud<-mu1-mu2
  sigsq1<-sigma1^2
  sigsq2<-sigma2^2

  numint<-1000
  dd<-0.00001
  coevec<-c(1,rep(c(4,2),numint/2-1),4,1)
  intl<-(1-2*dd)/numint
  bvec<-intl*seq(0,numint)+dd

  #Z method
  za<-qnorm(1-alpha/2)
  zb<-qnorm(power)
  n1<-ceiling(((sigsq1+sigsq2/n2n1r)*(za+zb)^2)/(mud^2))
  n2<-ceiling(n1*n2n1r)

  if(use_exact) #Exact method
   {
    n1 = n1-1
    n2 = n2-1
    powere<-0

    while(powere<power)
     {
      n1<-n1+1
      n2<-ceiling(n1*n2n1r)
      sigsqt<-sigsq1/n1+sigsq2/n2
      hsigsqt<-sqrt(sigsqt)
      wpdf<-(intl/3)*coevec*dbeta(bvec,(n1-1)/2,(n2-1)/2)
      dft<-n1+n2-2
      p1<-(n1-1)/dft
      p2<-1-p1
      s1<-sigsq1/n1
      s2<-sigsq2/n2
      b12<-(s1/p1)*bvec+(s2/p2)*(1-bvec)
      r1<-(s1/p1)*bvec/b12
      r2<-1-r1
      dfevec<-1/((r1^2)/(n1-1)+(r2^2)/(n2-1))
      tdfea<-qt(1-alpha/2,dfevec)
      powere<-sum(wpdf*pt(-tdfea*sqrt(b12/sigsqt),dft,mud/hsigsqt))+1-sum(wpdf*pt(tdfea*sqrt(b12/sigsqt),dft,mud/hsigsqt))
      }
    }

  c(n1=n1,n2=n2)
  }

ssize.welch(0.05,0.9,85,105,10,20,3)
ssize.welch(0.05,0.9,85,105,10,20,3,TRUE)

The Z method is also the one used in https://clincalc.com/stats/samplesize.aspx which cites Rosner B. Fundamentals of Biostatistics. 7th ed. Boston, MA: Brooks/Cole; 2011. Weirdly, it forces you to input only one variance but the formula it gives can use two (and is the same as the paper above). It is in the spirit of the usual way of computing sample sizes, but I'm not too sure about when the underlying approximations start to be false.
After some fiddling around, both methods give similar results unless you have very unbalanced groups (but in that case, at some point you might want to just approximate the super large group as a known population).
Hope this helps.
Edit : just realized this does not exactly answer your question about power rather than sample size, but you can easily flip the Z method formula to compute power (exact method seems more hairy ; worst case, numeric trial and error should work since the relationship is monotonous).
Answer: It seems to me the power calculation is pretty straight forward. Let's consider a 2 tailed alternative.
The null is rejected if $t<t_{\alpha/2}$ or $t>t_{1-\alpha/2}$. So Power = $P(T'<t_{t_\alpha/2})+P(T'>t_{1-\alpha/2})$
The true effect is $\eta=\frac{\mu_1-\mu_2}{\sqrt{s_1^2/n_1+s_2^2/n_2}}$
The true sampling distribution of $T'$ is a noncentral T distribution centered at $\eta$. Let $T = T'-\eta$ follow a t distribution centered at zero. So
$$\text{power} = P(T<t_{t_\alpha/2}-\eta)+P(T>t_{1-\alpha/2}-\eta)$$
I just figured this out on my own, so I may have made a mistake, but It seems to make sense to me. This formula cannot be reversed to give you the necessary sample size though,  it would have to be solved by numerical means to get a pre-specified power given effect size (or given $\mu_1, \mu_2, \sigma_1, \sigma_2$. It would be easiest to solve it for $n_1=n_2=n$, but in theory you could specify (for example) $n_1$ and solve for $n_2$.


PHP Sistema pra curar usuário de x em x minutos

Question: Bom dia pessoal! Estou criando um joguinho de rpg e não estou descobrindo como fazer com que o ele cure o usuário de x em x minutos. Consegui fazer via ajax, mas não é seguro né? Facilmente o usuário pode burlar e curar toda a vida dele se ele quiser... Tem alguma forma de fazer isso sem ajax? No banco de dados já tenho todos os valores, vida, vida máxima etc... Me deem uma força! Valeu.
@EDIT com ajax embora até tenha dado certo, o usuário precisava estar logado pra funcionar, seria algo também que ele não precisasse logar... que o sistema funcionasse mesmo com ele desligado.
Comment: Pesquise sobre Cron, ou algo do tipo, os servidores geralmente disponibilizam "cronômetros" para realizar tarefas agendadas, você pode deixar um arquivo de php já feito para "curar" os players, e utilizar as tarefas agendadas para rodar esse arquivo. Só não tenho certeza o tempo mínimo que eles rodam..
Comment: Que bom que achou, é que eu já tive que fazer isso para um jogo de RPG que tinha criado. Na época essa foi a melhor solução que achei. Qualquer dúvida pode falar
Comment: Humm acho que já vi algo a respeito no meu host... Vou dar uma pesquisada.

@EDIT encontrei, meu host tem a opção tarefas cron avançadas, onde ele pode executar o arquivo php a cada 60 segundos, que é o tempo minimo. Obrigado pela ajuda, vou deixar a pergunta aberta por um tempo pra ver se dão algumas outras idéias! :)
Comment: Só para entender... vc vai enviar alguma coisa para php ou só vai rodar um arquivo que já está lá?
Comment: No momento estou utilizando o cron... Ele roda o php com o script que cura se a vida estiver abaixo do limite máximo...
Answer: Você pode ter um evento no banco de dados que a cada x e x tempo faça alguma ação para você, por exemplo:
Tabela personagem, com id, nome e vida.
CREATE TABLE IF NOT EXISTS personagem (
    id INT PRIMARY KEY AUTO_INCREMENT,
    nome VARCHAR(40) NOT NULL,
    vida INT NOT NULL DEFAULT 50
)

Evento, a cada 2 minutos acrescenta +10 a vida do personagem, se a mesma for melhor que 100 (tem de ter controle para se caso a vida estiver em 92, nao ir pra 102, pois no exemplo a vida maxima é 100, mad não vou fazer isso)
DELIMITER $$
--
-- Eventos
--
CREATE EVENT `realizaCura`
ON SCHEDULE EVERY 2 MINUTE
STARTS '2016-05-12 15:11:10' ON COMPLETION NOT PRESERVE ENABLE
DO UPDATE personagem SET personagem.vida = (personagem.vida + 10) WHERE personagem.vida < 100$$
DELIMITER ;

Desculpem se tiver erro de sintaxe, fiz pelo telefone.
Comment: Quanto ao fato de não ser seguro utilizando ajax, certo, ele poderia manipular, todavia poderia também trabalhar em uma "chave" e ou bloqueio que permita que tal ajax só possa ser rodado a cada X tempo e se a chave não for a válida ele não executa. Outra alternativa seria a utilização de CRON como já dito pelo Fleuquer Lima.

Como é feito quando ele "perde" vida? Se for feito através de ajax, você deve levar em consideração que isso também pode ser manipulável...
Comment: Bacana Ivcs, +1


Overlay image on right with background colors using :before and :after properties

Question: I have a header div with a background color with an image on the right. I also have :before and :after properties on the same div.
But after adding those properties, the image is being overshadowed by it.
I tried overflow:visible !important; but that does not seem to work.
Any suggestions?

  .header-one {
  font-size: 2rem;
  line-height: 2.5rem;
  letter-spacing: 0.2px;
  color: #FFFFFF;
  font-family: "Myriad Pro";
  font-weight: bold;
  background-color: #182561;
  padding: 25px 0px 45px 60px;
  margin-bottom: 0%;
  position: relative;
  img {
    max-width: 100px;
    top: 0;
    right: 20px;
    overflow: visible !important;
    position: absolute;
  }
  .header-one:before {
    content: '';
    border-left: 585px solid transparent;
    border-bottom: 20px solid #E0E3E1;
    border-right: 585px solid #182561;
    position: absolute;
    right: 0;
    bottom: 0;
    left: 0;
  }
  .header-one:after {
    content: '';
    border-top: 20px solid transparent;
    border-bottom: 20px solid transparent;
    border-right: 585px solid #293FA2;
    position: absolute;
    right: 0;
    bottom: 0;
  }
HTML:

  Heading 1

Comment: `z-index` is your friend here
Comment: `z-index` will resolve overlapping issue
Comment: `z-index` is your overlapping friend
Answer: So, you want the image to be in front?Try adding z-index: 1 to .header-one and z-index: -1 to your pseudo elements. This should work.