Query using Criteria

Question: How to write the following query using criteria
<code>StringBuilder queryString = new StringBuilder("from BorrowerEvaluationDTO b where b.hasEvaluation = TRUE"+" ");
                if(sector != null && !sector.trim().equals("")){
                    queryString.append (" AND b.sector = '" + sector + "'");
                }
queryString.append(" AND b.id in (select needProfile.id from Investment where investor.id = '"+investorId + "')");
</code>
Comment: I think your problem is the subselect. Here is a example: http://stackoverflow.com/questions/4483576/jpa-2-0-criteria-api-subqueries-in-expressions
Answer: i'm not sure how your model exacly looks like, but i think your query will be more or less like that:
<code>        CriteriaBuilder cb = em.getCriteriaBuilder();
        CriteriaQuery<Object> criteria = cb.createQuery(Object.class);
        Root<Object> root = criteria.from(Object.class);
        List<Predicate> predicates = new ArrayList<Predicate>();
        predicates.add(cb.equal(root.get("hasEvaluation"), cb.literal(true)));

        if (sector != null && !sector.trim().equals("")) {
            predicates.add(cb.equal(root.get("hasEvaluation"),
                    cb.literal(sector)));

        }
        Subquery<Object> sub = criteria.subquery(Object.class);
        Root<Object> subRoot = sub.from(Object.class);
        sub.select(subRoot.get("needProfile").get("id"));
        sub.where(cb.equal(subRoot.get("investor").get("id"), investorId));
        predicates.add(root.get("needProfile").in(sub));

        criteria.where(predicates.toArray(new Predicate[predicates.size()]));
</code>

Query using Criteria


Google Apps Directory Sync search rule returns incomplete results

Question: I have successfully set up Google Apps Directory Sync (GADS) for one of the Active Directory forests I manage. This forest was created with a functional level of 2012R2. Now I am setting up GADS for a second forest. This forest used to be at the 2003 functional level but was later raised to a 2008 functional level.
In GADS I'm using a pair of User Account search rules that suspend Google Apps users who are Disabled in AD and un-suspend users who are Enabled in AD. This arrangement works flawlessly in the first forest I set up. But in my new forest it returns incomplete data.
Here are the queries I'm using:
Do Not Suspend
(&(objectCategory=person)(!(userAccountControl:1.2.840.1135220.127.116.113:=2)))
Suspend
(&(objectCategory=person)(userAccountControl:1.2.840.113522.214.171.1243:=2))
Here's the problem: If I run "Test LDAP Query" in the GADS interface on that Suspend query, it returns me only two users. But there are 36 users total who should be returned. Because the query is not finding the complete list of Disabled users, when I disable someone in AD and run a sync, GADS does not disable the user in Google Apps.
When I run the same query in a non-GADS LDAP query tool (VTLDAPQuery), it returns me the complete list.
The two users that GADS returns when I test the query were created back during the 2003 days. The other 34 were, I believe, created after the functional level was raised.
Comment: Okay I figured it out, and now I'm so angry with myself. The problem was insufficient permissions for the service account being used to run the LDAP query.

In a forest that had not gotten as screwed up as our is, special permissions would probably not be required, but in my particular domain, adding the LDAP query account to "Domain Admins" immediately resolved the problem. I will have to continue experimenting to find the minimum level of permissions required, but it is clearly the permissions that were at fault.
Comment: Glad you got it figured out! Please go ahead and add this as an answer and mark it as the answer to better help other people.
Comment: Follow-up: I have done some further testing by moving an older user into the scope of the query. This user was created at functional level 2003 but **still** does not get returned by the query in GADS.
Comment: I have determined that some of my pre-2008-level users have the adminCount attribute set in AD, but others do not. (For the ones who have it, it is set to 1.) So far all the users I have tested who have this attribute set will be retrieved by my query. I have not identified any users who do not have this attribute set but who do get retrieved by the query. Could this have something to do with ACLs or Security Group membership?
Comment: Unsurprisingly, my domain had some screwed-up nested groups and users in protected groups. I've scrubbed as much as I could find.

Then I cleaned the permissions of one of the older users who showed up in the query. I reset her adminCount value to 0 and her permissions to default. After that, the query no longer retrieved her.
Comment: I added a new users to a protected group. After a few minutes her adminCount was set to 1, and she was no longer retrieved by the query.

So the problem seems to be that the GADS LDAP query method will not retrieve users with inheritable permissions.
Answer: Okay I figured it out, and now I'm so angry with myself. The problem was insufficient permissions for the service account being used to run the LDAP query. In a forest that had not gotten as screwed up as ours is, special permissions would probably not be required, but in my particular domain, adding the LDAP query account to "Domain Admins" immediately resolved the problem. I will have to continue experimenting to find the minimum level of permissions required, but it is clearly the permissions that were at fault.
Answer: I just encountered a similar issue, where an LDAP query in Google Cloud Directory Sync would return 109 users but the same query in Microsoft's LDP tool returned 118 users. The BaseDN was the same in both cases, logged in with the same BindDN user account. The query is as simple as it gets: <code>(objectclass=user)</code>
This turned out to be the easiest solution imaginable - GCDS automatically filters results to only select those that have a properly filled out "mail" attribute. If your user does not have an e-mail address in their AD properties, or the e-mail address is not a valid format, then GCDS will drop the user from the search results.
This may not be the problem with incomplete results in every case, but it was the problem in my case, so I hope this answer is helpful to someone.
Comment: I have also discovered the same thing. (It actually is very useful for proactively selecting accounts that you don't want to sync.)

This turned out not to be the source of the issue for me, though. I thought I had posted the answer already, but I hadn't.


Efficient handling of super wide, but not so tall, bitmap?

Question: Is there any way to create a more space/resource efficient bitmap?
Currently I try to render a file, approx 800px high but around 720000px wide.
It crashes my application, presumably because of the share memory-size of the bitmap.
Can I do this more efficiently, like creating it as an gif directly and not later when I save it?
I try to save a series of lines/rectangles from a real world reading, and I want it to be 1px per 1/100th of a second.
Comment: For assistance on google open id, check meta.stackoverflow.com
Comment: You may want to consider storing the data in plain text format, and rendering whatever portion the user requests as an image (with provided export/import functionality).  You might fear this to be more frustrating since the user needs to use your program to work with the data, but working with gigantic images is also hard for users, so you're really in trouble either way.
Comment: I'd love to know why you need such an image.
Answer: You have to remember that any image you load into memory regardless if it's a GIF or JPEG or whatever on disk will be turned into a 32 bit bitmap which means four bytes per pixel.
This means that the image you're creating will be:
<code>4 bytes * 800 pixels high * 720,000 pixels wide = 2,304,000,000 bytes
</code>
You're basically blowing your memory by trying to create an image that large.
For whatever you're trying to accomplish the answer is tiling and caching your image.
Answer: Your image is about 2.3 gig, and the biggest .Net object you can have is 2 gig regardless if the machine is 32 or 64 bit.
You're going to have to break the bitmap up in chunks to handle an image that size.
Answer: You're going to have to either:

Force x64 environment and get a stack load of RAM.
Change your architecture

Your image is going to be a little over 2 GB.
Comment: Even switching to x64 will not solve the issue immediately. The maximum object size in the GC Heap is still at 2GB, even in the x64 version of the .NET 2.0 runtime.
Answer:
Can i do it more efficient, like creating it as an gif directly and not later when i save it?

You can compress the image as you write it. It won't be in (uncompressed/unencoded) "bitmap" format anymore. Examples of compression algorithm include "run-length encoding" and "huffman".
Also, use the least possible color-depth: preferaby black-and-white, i.e. 1-bit-per-pixel.
Also perhaps save it in several, smaller, discontinuous memory chunks: instead of a single huge memory chunk (so huge that it can't even be allocated in the first place).
Answer: If you create it as 720000px high and 800px wide as a bmp and rotate it when actually displaying it(*),
you can stream the data directly to file in bitmap form.  Maybe use RLE instead of raw bitmap; streaming in this manner should still be possible in that case.
(*)Displaying it is left as an exercise to the reader.  You'll need tiling or something.


Meaning of MATLAB logistic regression coeficients

Question: Suppose that I'm using this function to implement logistic regression in MATLAB R2015a:
[coefs,~,stats] = glmfit(TrainInputs,TrainTargets,'binomial','logit'); % Logistic regression

I have ten continues inputs and a binary output (1 and 0). This is the coefs output of above function:
coefs =
2.04869088486007
-4.66230054501349
0.355690380077187
-1.87230109681715
6.85763253382301
3.46437006313075
-0.282766819896174
-1.08763387246821
-1.94250600133904
1.46621342829667
-2.83661766065114

What is meaning of the coefs in logistic regression of MATLAB? Suppose that input names are X1 to X10. If we have 1 % increase in X1, We have -4.66% increase in probability of being class 1?
Comment: The duplicate includes logistic regression as a special case. The first part of my answer addresses it directly.  It concludes, "Thus, $\beta_m$ is the marginal change in log odds with respect to $X_m$."
Comment: Maybe that's correct--but all your parentheses and mischaracterizations get in the way of being sure of what you are saying. A [relative risk](https://en.wikipedia.org/wiki/Relative_risk) cannot be negative. An exponentiated value cannot be negative. There is no meaning in "compared to the base outcome": a relative risk *already* involves a comparison.
Comment: @whube . Thank you for comment. for a negative value like `-4.6623` we have `exp(-4.6623)=0.0094`. So now we have a positive value. We should consider negative sign here or positive sign? `a one-unit change in the variable corresponding to "X1" will multiply the relative risk of the outcome (negative or positive?) (compared to the base outcome) by 0.0094` ?
Comment: Can you revise this sentence with this new negative coefficient? `a one-unit change in the variable corresponding...` .
Answer: The answer is no, user2991243; you can apply that interpretation in the linear regression model, while the interpretation of $\beta$ coefficient in the logistic regression model is pretty different, because of the logit transformation and the fact that $\beta$ coefficients are odds ratios and, so, not linear in the probability.
Here you can find the explanation of such transformation and the relative interpretation (using stata).
Hope this helps.
Comment: Thank you for answer. So for example for `X4` with value of `6.85763253382301` : we will see 10461% !! increase in the odds of output for a one-unit increase in `x4`. `exp(6.85763253382301) = 951.1127` and `11%*951=10461` ??


Two Sphere inside a cube

Question: Find the radius of the largest possible sphere inside a cube when two sphere of equal radii are made to sit inside a cube of side $x$.
My approach: since it is easier to start with single sphere, do that and find the distance between the sphere and corner of the cube.
Symmetry mandates that the corner distance will not change even if there are two spheres and they will touch each other at center and we are done.
However, if someone can provide a better solution less dependent on symmetry or calculus, that will be more elegant and convincing .
Answer: Your argument is incomplete, there is no prior reason the optimal solution is symmetric.
Instead, consider two spheres of radius $r$. If they can fit inside a cube of side $x$, their centers fit inside a cube of side $x - 2r$. This means the distance between the centers is at most $(x-2r)\sqrt{3}$. Since this need to be at least $2r$, we have
$$(x-2r)\sqrt{3} \ge 2r\quad\implies\quad r \le \frac{3-\sqrt{3}}{4}x$$
It is possible to fit two spheres of radius $\frac{3-\sqrt{3}}{4}x$ inside the cube by placing the centers along a diagonal. As a result, $\frac{3-\sqrt{3}}{4}x \simeq 0.316987 x$ is the largest radius and the configuration you have is indeed the optimal one.
Comment: suppose the spheres are along the diagonals of a cube of side x , then why should the distance between their center be ($x$-2$r$)$\sqrt {3}$?
Comment: @GeorgeCarlin if you pick two points from a cube of side $y$, the largest distance they can have is $\sqrt{3} y$. There are only 4 possible configuration that achieve this distance. Namely, when those two points are the endpoints a diagonal. So $(x-2r)\sqrt{3}$ is an upper bound. Since you want to maximize $r$, one obvious thing to check is whether placing the center at the vertices of the smaller cube works or not. It does work.

Two Sphere inside a cube


How to validate 3 or more fields when creating content in D7

Question: I have a custom content type with 3 fields (integer). How can I check when content is created or edited that the sum of those three fields are equal to a predefined total. For instance, if the three fields represent percentages of a task to be completed, I would like to issue an error message if the sum of the three fields is not 100%
Answer: Create a module to add a validation handler to the form by implementing hook_form_alter():
function MODULE_form_alter(&$form, &$form_state, $form_id) {
  if ($form_id == 'MY_FORM_ID') {
    $form['#validate'][] = '_MODULE_validate';
  }
}

Then define the handler function that checks the field values:
function _MODULE_validate($form, &$form_state) {
  // Check that the sum of the fields equals a predefined total
}

If you're not familiar with creating your own modules, but are comfortable with PHP, you could probably use the Field Validation module to add a custom PHP validator that does the same thing.
Comment: Thanks for your help. I have written modules in the past, so that's not a problem. Just needed that hint.


Pushed to Heroku - Assets(+?) Missing (Rails 5)

Question: I pushed a sample app I was working on to Heroku. Rails 5 - I now notice a lot of issues/differences, a lot I can relate back to an obvious pipeline issue. Majority of the references I see are from R3/R4 but mention putting
 config.assets.compile = true
  config.assets.digest = true

in and modifying all url to font-url or image-url - I did that for majority of the fonts, but I'm still seeing blank images, and broken links/missing icons.
Another weird thing is I used JSON Builder to builds out a site.json page - The page itself is actually fine; No issues. However, when I try to limit it with a parameter it immediately returns a 500 error. I've found 4-5 other posts on SO about this, each with a minor amount of replies, and usually they say to do what I mentioned above.
I even tried to go into all of my CSS files cores, and change it to
font-url('<%= asset_path("mana.eot?#iefix&v=1.3.0") %>') which also didn't do anything.
I read on Heroku's help site that it has issues due to the MD5 fingerprint and I followed through the steps to connect to the bash to see the file's existance AND the name, but I'm unsure how to really.... do anything about that? Don't they have the same fingerprint when locally compiled/C9/IDE?
Answer: In terms of photos missing, you probably have to upload them to AWS or something similar and save the information there. Make an account and follow the instructions, it is pretty straight forward. I had the same problem for my app and that was the solution for me.
In terms of links make sure none of them are localhost:3000 because that will break the links.
Comment: Heroku has the image on it's local instance, but the link is broken. it also can't find jquery etc etc. Basically everything within the assets pipeline.


Test if string exists in file with Python

Question: I'd like to add only servers currently doesn't exist at file
My current code :
 f = open(filename,'a')
 for server in cmo.getServers() :
    print >>f, server.getListenAddress()

Thanks in advance
Answer: Build a list of servers already present in the file:
present = [l.strip() for l in open(filename)]

(assuming the file format is just one server per line, no other symbols).
Then check if an address is in the list:
for server in cmo.getServers():
    address = server.getListenAddress()
    if address not in present:
        print >>f, address

This assumes that the addresses you get from getServers() will not repeat.
If that's also possible, then build a set of them first:
new = set(server.getListenAddress() for server in cmo.getServers())
for address in new:
    if address not in present:
        print >>f, address

Answer: try this:
data = set( [i.strip() for i in open( filename, 'r' ).readlines()] )

for server in cmo.getServers() :
    data.add( server.getListenAddress() )

open( filename, 'w' ).write('\n'.join(data))

Comment: Thank you , nice answer , but I getting error :            TypeError: set() takes at least 2 arguments (1 given)


How to deal with critical value hypothesis test when test statistic is negative?

Question: I am solving the following problem:

An electronic tablet producer claims that the batteries on their
  tablets lasts for 10 hours but from my experience I think it is less
  than that. It is known that the battery life-span follows an
  exponential distribution. A random sample of size 100 (supposed to be
  independent) is tested, for which the mean battery life is found to be
  9 hours. Test whether the producer's claim is true at a 5%
  significance level using the critical value method.

My Work
We can model the $i$th tablet battery life-span as $X_i \sim Exp(\lambda)$ where the lifespan is measured in hours. Each $X_i$ has mean $\dfrac{1}{\lambda}$ and standard deviation $\dfrac{1}{\lambda}$. A sensible estimator of $\mu = \dfrac{1}{\lambda}$ is then just $\bar{X}_n$.
The null hypothesis is $H_0 : \mu = 10 = \dfrac{1}{\lambda}$,
and the alternative hypothesis is $H_1 : \mu = \dfrac{1}{\lambda} < 10$.
The appropriate test statistic is $Z = \dfrac{\bar{X}_n - \mu}{\dfrac{\sigma}{\sqrt{n}}} = \dfrac{\bar{X}_n - 10}{\dfrac{10}{\sqrt{100}}}$
Assuming the sample size is large enough for the CLT to apply, we can get our observed test statistic::
$Z = \dfrac{9 - 10}{\dfrac{10}{10}} = -1$
I now calculate the critical value at a 5% level of significance:
$P(z > z_{0.05}) = 0.05$
$\implies 1 - P(z < z_{0.05}) = 0.05$
$\implies 0.95 = P(Z < Z_{0.05})$
$\implies Z_{0.05} = 1.64$
This is my point of confusion: How do I deal with the negative value for $Z$ (the test statistic)? When doing these types of problems, I have always derived a positive value for the test statistic, so I'm unsure of how to proceed.
I would greatly appreciate it if people could please take the time to explain what one does (and why) when they derive a negative value for the test statistic, as done above.
EDIT:
My understanding is that, when hypothesis testing using the critical value method, we are seeking to reject the null hypothesis if it falls inside an arbitrary interval along the tails of the standard normal distribution, based on the indicated level of significance. When we calculate the test statistic, $Z$, we get a value of $-1$. This is the same $Z$ that is used in calculating quantiles: $P(Z > a_\alpha) = \alpha$. The numerical value of $Z$ for quantiles is irrelevant, but the sign (+ or -) IS relevant. Therefore, we have, in this case, that $P(-Z > a_{0.05}) = 0.05$, which implies that $P(Z < -a_{0.05}) = 0.05$. After doing further simple calculations, we find that $P(Z < a_{0.05}) = 0.95$; therefore, $a_{0.05} = 1.64$.
Now, since we want to reject the null hypothesis if the test statistic is on the tails of the standard normal distribution (too unlikely in either direction), we reject values that are $< -1.64$ or $> 1.64$?
Comment: Right after you state your alternative, try stating whether you should reject the null for very large values of $\hat\mu$, for very small values of $\hat\mu$ (or for both very large and small) ... that is, state the nature of your rejection region. Then when you move to a normal approximation, draw yourself a normal density and shade in the tail corresponding to the region you;should reject for. *THEN* actually do the test, starting with a formal rejection rule.
Comment: You seem to have *completely* ignored the part where I said: "Right after you state your alternative, try stating whether you should reject the null for very large values of μ^, for very small values of μ^ (or for both very large and small)" ... and as a result, you went awry (again). ..  Look at your alternative hypothesis gain and do that very basic thing I suggested.
Comment: consider that this assertion: "We're looking for the value such that the test statistic is in the 95 percentile" does not correctly capture what we want. I believe I now see something of both the specific kind and magnitude of your misconception and may be able to write an answer. I didn't until now realize how far back the misunderstanding was.
Comment: @Glen_b thanks for the response. Please see my comment to the_Ship_of_Fools' answer; is that what you were alluding to?
Comment: @Glen_b Based on the alternative hypothesis ($\mu < 10$), I would say that we should reject the null hypothesis in favour of the alternative hypothesis for values of $\mu < 10$?
Comment: Not quite ... 1. You don't know $\mu$; so your eventual rejection rule won't be in terms of it (reread my first comment, and note that's not a $\mu$ there), and 2. You don't (at the point you're working this out) know how far below 10 you'd need to be to reject (e.g. in small samples you wouldn't reject for a value just below 10), so "$<10$" is no good either.  You're initially just trying to get a general idea whether it's the largest or smallest values (or both) in the sampling distribution you'd reject for so that you can make sure you don't make an error with the formal rejection rule
Comment: @Glen_b I'm not sure what $\hat\mu$ is supposed to be. Anyway, I've done further calculations to try and figure out a solution. See my edit of the OP.
Comment: $\hat\mu$ is your sample estimate of the population mean. If the sample mean was reaally small, you'd want to reject, right?
Comment: @Glen_b Indeed: If $\hat\mu$ was on either tail of the standard normal distribution (too unlikely in either direction), we would want to reject the null hypothesis. I've attempted to use this fact in my edit to the OP.
Comment: @Glen_b I don't see how my response to the_Ship_of_Fools was incorrect. No matter how I think about this, my reasoning leads me to the same conclusion, as stated in my edit to the OP.
Comment: The alternative you stated in the question (correctly) is that the population mean is smaller than the one in the null. You would not reject the null in favor of the alternative if the sample mean was *large* (that is, you would not be saying to the manufacturer "in my extensive tests the batteries lasted on average 14 hours  -- my suspicions that they really don't last nearly as long as you say are confirmed!!"). Well, not unless you wanted to sound crazy, maybe -- because *then* a two-sided rejection rule for a one-sided alternative would work great.
Comment: @Glen_b This still doesn't make sense to me. The calculations give us the value of positive 1.64 for $a_{0.05}$, so where does the -1.64 come from? If -1.64 was the value we were looking for, then why did we not get this value for $P(Z < a_{0.05})$?
Comment: Consider that maybe the "calculations" may not do what's required here.
Comment: @Glen_b I don't know what that means. We're looking for the value such that the test statistic is in the 95 percentile, so how are the calculations inadequate?
Answer: Pointer, I'm a statistical amateur compared to others here, but I think my lack of experience might help me understand what you're having difficulty with.
Negative values are perfectly OK in the test statistic here since your reference distribution can take negative values - as you're using the CLT, your null distribution is the standard normal distribution, which is symmetric around the y-axis (look it up on Wikipedia if you're not sure what I mean).  The other possible source of confusion (if you're already quite familiar with the standard normal) is that you might have only needed two-sided tests up to this point.  When doing a two-sided test with a symmetric distribution, much of the time people just use the absolute value of the test-statistic and halve the significance level (many people code it in R this way).
Last, here, since you have a one-sided alternative hypothesis, and since the Z-transformed statistic essentially puts your hypothesized mean (10) at 0 (the "minus mu" in the equation), if you think about it, you'll see that your alternative hypothesis would have no chance whatsoever of being true, even before looking up Z(0.05), if your test statistic were positive.
To other, more experienced statisticians here, please correct me if I've given any misinformation (and if I have, many apologies!).  I'm trying to learn myself!
Comment: I think you (and @Glen_b) have suggested a solution to the problem: Since the test statistic has a normal distribution, we can use the properties of the normal distribution. Therefore, In this case, we do not reject the null hypothesis if the test statistic is in the interval $[-1.64, 1.64]$?
Comment: not quite; what you just said would be true if you had a two-sided alternative hypothesis (but you'd need to use the .025 and .975 quantiles in that case).  But your Ha is one-sided - you don't care if the mean lifespan is more than 10; you just care if it's less than 10.
Comment: In case it helps with broader understanding, notice that you have "Xbar-10" in the numerator.  B/c your Null assumes that the mean lifespan is 10 hours, you subtract 10 from Xbar to recenter (from 10 to 0) the distribution which your null assumes the data should fit.   If your Xbar is less than 10, you'll have a negative Z-statistic.  Since your alternative hypothesis is 1-sided, you only care about negative values of Z.  As glen_b pointed out, you only care if your battery life is worse than advertised.  No one has a problem with battery life that is better than advertised!
Answer: Let's consider what we're trying to do at a very basic level.
Here's the actual density of the average lifespan of 100 batteries under the null. It's somewhat skew but will be not-too-terribly approximated by a normal:

If the manufacturer's batteries don't last as long as claimed, we should see a lower average than 10. So we want to reject when our estimate of $\mu$ ($\hat \mu = \bar{x}$) -- i.e. the sample mean -- is sufficiently far below 10 that $\mu=10$ isn't a tenable claim.
Consequently, our rejection rule must correspond to something of the form "reject for $\hat \mu \leq C$" for some suitable choice of $C$. If we don't get that we clearly made a mistake.
How do we choose $C$? We want $C$ to be as high as possible (i.e. close to $10$ from below, so that we maximize power) while keeping the probability of falling in the rejection region when $H_0$ is true to be no more than $\alpha$:

When you use a normal approximation for this, you still want it to correspond to rejecting small values of $\bar{x}$. If you don't do that, it won't correspond to your stated alternative hypothesis (potentially leaving you in the silly position of accusing the manufacturer of shorter battery life when maybe it's actually longer).
Can you figure out what the $Z_C$ value would be below which $\alpha$ of the probability will lay?  (under $H_0$, naturally)
(If you use a calculation approach that yields a $Z_C$ that doesn't look qualitatively like the diagram, you cannot be using the right one. Instead just do the basic calculation the diagram indicates. This is the sort of diagram you should have been drawing for us -- such diagrams are crucial to avoiding errors. I really don't know how they can be letting you try to answer questions like this without insisting you draw a diagram every time.)
What rejection region for $\bar{x}$ would it imply?
(If you use a normal approximation, the actual type I error rate for a nominal 5% test turns out to be 4.35%, which is perhaps a bit further out than many people would hope, but that's hardly the big issue here -- much more important is figuring out the right direction for rejecting the null)
Comment: Thank you very much for the elaborate response. I think I understand now. So after we convert $\bar{x}$ to the standard normal distribution (this is our test statistic, right?), the part of the diagram that's highlighted, $\alpha$, becomes $-1.64 <$, the 10 becomes -1? (or is it 0?), and 1.64 would be above -1 (or is it 0?). Therefore, from the diagram, we would see that the 1.64 that we deduced in our calculations is actually implying a LONGER battery life than the manufacturer's claimed mean? Does this all seem correct to you?
Comment: No, $\alpha$ is your significance level and corresponds to the *area* (which is why it's within the region in the diagram, not at the boundary of it). Don't confuse the critical value with the area below it. Rather than pepper guesses at me and hoping I'll confirm one of them, write a probability statement from the diagram, and then transform it via algebraic steps into a statement about a standardized variable.
Answer: Based on your $H_0: \mu = 10$ and $H_1: \mu < 10$, the $H_0$ will be rejected if sample mean is < a specified number. So it should be:
$\Pr(z < z_{0.05}) = 0.05$
$\implies Z_{0.05} = -1.64$
It means if sample z value is < -1.64, then reject $H_0$ at $\alpha = 0.05$ level (one-side test). Your z is -1 > - 1.64, so there is no enough evidence to reject $H_0$ that $\mu = 10$.


How to replace the button when the user is not authenticated?

Question: I copied the twig file of my nodes into my theme.
On my nodes there is a flag displayed, but only authenticated users can see it.
Here is my code :

  {{ content.flag_like_group }}

How to replace the flag for unauthenticated users by :

   J'aime

Comment: Specifying URLs directly is not a good practice. Instead consider using `url()` Twig function which allow to build a URL by route name. This would allow to avoid possible issues with path aliases, domains, languages and so on.
Comment: @ya.teck thanks, can you give me an example with url login ?
Comment: `{{ url('user.login') }}`
Answer: You can use predefined logged_in variable.
{% if logged_in %}

    {{ content.flag_like_group }}

{% else %}

     J'aime

{% endif %}


How do I add a dropdown field in the commerce form?

Question: Does someone know how to add a dropdown field to the commerce form?
I need to add a field where people need to select one of the 3 options. Ex.: "How did you know us?" (options: Google, a friend, I don't know)
Can anyone help me?
Thanks
Comment: Which Commerce form? There are a variety of them...
Answer: I think this is what you are looking for:
/admin/commerce/config/line-items/product/fields
This is where you can add fields to the product line item type.  Make sure you have the Line Item UI module enabled or you might not have access to that UI.


Upgrading CentOS does not update kernel version

Question: I am familiar with Debian based distros and now I have to administer CentOS machines.
I have two CentOS 4.4 machine and I both applied
yum upgrade

on them and they are both (lsb_release -a) 4.9 final now.
However when I looked the kernel version
uname -ar

One of them
 Machine A: 2.6.9-42.0.3.ELsmp
 Machine B: 2.6.9-42.0.8.ELsmp

Then I copied Machine B's CentOS-Base.repo file to Machine A then copied B's yum.conf to A and I retried but yum upgrade  but A's kernel version is still same. What can I do? What do I miss?
Thanks
Answer: you have to reboot for the kernel to actually update, at that point, uname will give the correct results.
its worth pointing out that centos4 is now only on maintenance updates so you may want to consider also planning for a larger update to a later version of centos (currently 6.0 is latest)
Comment: I see B has installonlypkgs=kernel kernel-smp kernel-devel kernel-smp-devel kernel-largesmp kernel-largesmp-devel kernel-hugemem kernel-hugemem-devel But A does not have in yum.conf (although I copied it B to A after upgrade process) so it needs reboot. Thanks
Comment: But A does not have in yum.conf so it needs reboot
Answer: The kernel is probably being excluded in your /etc/yum.conf file.  You can check for excludes= lines in /etc/yum.conf
# grep 'exclude' /etc/yum.conf

You can either comment out the exclude= line or you can tell yum to disable the excludes for a single run with the command:
# yum update --disableexcludes=all

It's probably preferable to use the second form as you don't necessarily want the kernel and other packages updated without some supervision.
HTH


SharePoint Server 2010 change Domain Controller to RODC

Question: i'm struggeling with the SharePoint Server 2010.
I moved my SPS 2010 into a DMZ within the same site as my RODC.
The SPS 2010 should get AD information from the RODC, but however, it dosen't, since it always tries to use the DC for such a matter.
How can i define the RODC/DC to use for the SPS 2010 ?
Thanks !
Comment: When you say "get AD information from the RODC", are you talking about authenticating users against it? Or are you talking about user profile synchronization?
Comment: I'm talking about authenticating user against the AD.
Answer: Be aware that it is not SharePoint, but the client, that authenticates against the Active Directory.
If I have 3 Sites:

Site A - Domain Controller, 1 client (ClientA)
Site B - RODC, SharePoint Server, 1 client (ClientB)
Site C - No DC's, 1 client (ClientC)

ClientA tries to log in to a Sharepoint Site, the client authenticates against a site local Domain Controller, that is, the one in Site A.
ClientB tries to log in to a Sharepoint Site, the client, again, authenticates against a site local Domain Controller, here it is the RODC in Site B
ClientC tries to log in to a Sharepoint Site, the client authenticates against either:

A Domain Controller that serves Automatic Site Coverage for Site C
A Domain Controller within the site for which the Site Link Cost is the lowest from Site C
Comment: I missed out that point totaly. But this now leaded me to another problem existing with the RODC. Thanks !


How much do we know about p-hacking "in the wild"?

Question: The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ways to procure a "more significant" result, including but by no means limited to:

only analysing an "interesting" subset of the data, in which a pattern was found;
failing to adjust properly for multiple testing, particularly post-hoc testing and failing to report tests carried out that were not significant;
trying different tests of the same hypothesis, e.g. both a parametric and a non-parametric test (there's some discussion of that in this thread), but only reporting the most significant;
experimenting with inclusion/exclusion of data points, until the desired result is obtained. One opportunity comes when "data-cleaning outliers", but also when applying an ambiguous definition (e.g. in an econometric study of "developed countries", different definitions yield different sets of countries), or qualitative inclusion criteria (e.g. in a meta-analysis, it may be a finely balanced argument whether a particular study's methodology is sufficient robust to include);
the previous example is related to optional stopping, i.e., analyzing a dataset and deciding on whether to collect more data or not depending on the data collected so far ("this is almost significant, let's measure three more students!") without accounting for this in the analysis;
experimentation during model-fitting, particularly covariates to include, but also  regarding data transformations/functional form.

So we know p-hacking can be done. It is often listed as one of the "dangers of the p-value" and was mentioned in the ASA report on statistical significance, discussed here on Cross Validated, so we also know it's a Bad Thing. Although some dubious motivations and (particularly in the competition for academic publication) counterproductive incentives are obvious, I suspect it's hard to figure out quite why it's done, whether deliberate malpractice or simple ignorance. Someone reporting p-values from a stepwise regression (because they find stepwise procedures "produce good models", but aren't aware the purported p-values are invalidated) is in the latter camp, but the effect is still p-hacking under the last of my bullet points above.
There's certainly evidence that p-hacking is "out there", e.g. Head et al (2015) looks for tell-tale signs of it infecting the scientific literature, but what is the current state of our evidence base about it? I'm aware that the approach taken by Head et al was not without controversy, so the current state of the literature, or general thinking in the academic community, would be interesting. For instance do we have any idea about:

Just how prevalent is it, and to what extent can we differentiate its occurrence from publication bias? (Is this distinction even meaningful?)
Is the effect particularly acute at the $p \approx 0.05$ boundary? Are similar effects seen at $p \approx 0.01$, for instance, or do we see whole ranges of p-values affected?
Do patterns in p-hacking vary between academic fields?
Do we have any idea which of the mechanisms of p-hacking (some of which are listed in the bullet points above) are most common? Have some forms proven harder to detect than others because they are "better disguised"?

References
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biol, 13(3), e1002106.
Comment: Your last question is a nice idea for a research: give some raw data to a group of researchers form different fields, equip them in SPSS (or whatever they use) and then record what they are doing while competing with each other for more significant results.
Comment: @Adrian CV is just a Q&A site, it does not hold any data, or code, does not have any hidden repository - everything you find in answers is yours under CC license :) **This** question seems to be asking about collecting such examples.
Comment: Pseudoreplication, the error of omitting to take into account the hierarchy in data, can also lead to much much smaller p-values, hence unjustified claims, unsupported by data. And that error is quite common in applied sciences IMO.
Comment: One might be able to do it without the subjects knowing it was happening using a history of kaggle submissions.  They aren't publishing, but they are trying every way possible to hit the magic number.
Comment: Does crossvalidated have any collections (e.g. community wikis) of simple simulation examples of p-hacking?  I'm imagining toy examples in which the simulated researcher reacts to "marginally significant" results by collecting more data, experiments with regression specifications, etc.
Comment: @Tim of course, I wasn't imagining any hidden code repos -- just code snippets included in answers.  For example, someone might ask "what is p-hacking?", and someone might include a toy R simulation in their answer.  Would it be appropriate to respond to the current question with code examples?  "How much do we know" is a very broad question.
Comment: @amoeba and anyone else still interested... a lot of the discussion has been about psychology/social science, but there's an interesting (but not uncontroversial) 2020 paper about economics: *Methods Matter: P-Hacking and Publication Bias in
Causal Analysis in Economics* by Brodeur, Cook and Heyes https://web.archive.org/web/20200816043532/https://www.aeaweb.org/content/file?id=12747 I may add an answer to this Q based on its findings, if time allows!
Comment: @Adrian I didn't intend for this Q to cover that ground - I'll try editing to clarify - but I think that might well be the kernel of a good question in its own right. (Also, thanks to Stephan Kolassa for his useful previous edit.)
Comment: I think p-hacking is pretty rare in the field of theoretical mathematics. And I'm sure the correlation of p-hacking and supply and demand of funding of the field is > 0.9.
Comment: @CliffAB, I wonder if you meant theoretical physics or pure mathematics!
Comment: @amoeba: I mostly meant "not applied mathematics". Pure mathematics is probably the right term.
Comment: Hey Silverfish, seeing that you haven't yet accepted any answers here and vaguely remembering that you said some time ago that you were later going to get back to this discussion, I am wondering if there is anything specific in your Q that you feel remains unanswered. Or perhaps there is something in my A that you find unconvincing, then I am happy to discuss it in chat.
Comment: @amoeba I have been toying with putting a bounty up at some point. I'd be really interested to see a literature review that shows any "to and fro" about the Head paper (whether there have been serious attempts to refute it, or studies that have replicated its findings in other fields) but I wonder whether it is still too recent for that to have played out to any great degree. The existing answer is excellent (of course) but as it is largely a personal response I think it would be complemented by a discussion of the mainstream academic response to Head
Answer: EXECUTIVE SUMMARY: if "p-hacking" is to be understood broadly a la Gelman's forking paths, the answer to how prevalent it is, is that it is almost universal.

Andrew Gelman likes to write about this topic and has been posting extensively about it lately on his blog. I don't always agree with him but I like his perspective on $p$-hacking.  Here is an excerpt from the Introduction to his Garden of Forking Paths paper (Gelman & Loken 2013; a version appeared in American Scientist 2014; see also Gelman's brief comment on the ASA's statement), emphasis mine:

This problem is sometimes called "p-hacking" or "researcher degrees of freedom" (Simmons, Nelson,
  and Simonsohn, 2011). In a recent article, we spoke of "fishing expeditions [...]". But we are starting to feel that the term "fishing" was unfortunate, in that it invokes an image
  of a researcher trying out comparison after comparison, throwing the line into the lake repeatedly
  until a fish is snagged. We have no reason to think that researchers regularly do that. We think
  the real story is that researchers can perform a reasonable analysis given their assumptions and
  their data, but had the data turned out differently, they could have done other analyses that were
  just as reasonable in those circumstances.
We regret the spread of the terms "fishing" and "p-hacking" (and even "researcher degrees of
  freedom") for two reasons: first, because when such terms are used to describe a study, there is
  the misleading implication that researchers were consciously trying out many different analyses
  on a single data set; and, second, because it can lead researchers who know they did not try
  out many different analyses to mistakenly think they are not so strongly subject to problems of
  researcher degrees of freedom. [...]
  Our key point here is that it is possible to have multiple potential comparisons, in the sense of
  a data analysis whose details are highly contingent on data, without the researcher performing any
  conscious procedure of fishing or examining multiple p-values.

So: Gelman does not like the term p-hacking because it implies that the researches were actively cheating. Whereas the problems can occur simply because the researchers choose what test to perform/report after looking at the data, i.e. after doing some exploratory analysis.
With some experience of working in biology, I can safely say that everybody does that. Everybody (myself included) collects some data with only vague a priori hypotheses, does extensive exploratory analysis, runs various significance tests, collects some more data, runs and re-runs the tests, and finally reports some $p$-values in the final manuscript. All of this is happening without actively cheating, doing dumb xkcd-jelly-beans-style cherry-picking, or consciously hacking anything.
So if "p-hacking" is to be understood broadly a la Gelman's forking paths, the answer to how prevalent it is, is that it is almost universal.
The only exceptions that come to mind are fully pre-registered replication studies in psychology or fully pre-registered medical trials.
Specific evidence
Amusingly, some people polled researchers to find that many admit doing some sort of hacking (John et al. 2012, Measuring the Prevalence of Questionable
Research Practices With Incentives for
Truth Telling):

Apart from that, everybody heard about the so called "replication crisis" in psychology: more than one half of the recent studies published in the top psychology journals do not replicate (Nosek et al. 2015, Estimating the reproducibility of psychological science). (This study has recently been all over the blogs again, because the March 2016 issue of Science published a Comment attempting to refute Nosek et al. and also a reply by Nosek et al. The discussion continued elsewhere, see post by Andrew Gelman  and the RetractionWatch post that he links to. To put it politely, the critique is unconvincing.)
Update Nov 2018: Kaplan and Irvin, 2017, Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time show that the fraction of clinical trials reporting null results increased from 43% to 92% after pre-registration became required:

$P$-value distributions in the literature
Head et al. 2015
I have not heard about Head et al. study before, but have now spent some time looking through the surrounding literature. I have also taken a brief look at their raw data.
Head et al. downloaded all Open Access papers from PubMed and extracted all p-values reported in the text, getting 2.7 mln p-values. Out of these, 1.1 mln was reported as $p=a$ and not as $p<a$. Out of these, Head et al. randomly took one p-value per paper but this does not seem to change the distribution, so here is how the distribution of all 1.1 mln values looks like (between $0$ and $0.06$):

I used $0.0001$ bin width, and one can clearly see a lot of predictable rounding in the reported $p$-values. Now, Head et al. do the following: they compare the number of $p$-values in the $(0.045, 0.5)$ interval and in the $(0.04, 0.045)$ interval; the former number turns out to be (significantly) larger and they take it as an evidence of $p$-hacking. If one squints, one can see it on my figure.
I find this hugely unconvincing for one simple reason. Who wants to report their findings with $p=0.05$? Actually, many people seem to be doing exactly that, but still it appears natural to try to avoid this unsatisfactory border-line value and  rather to report another significant digit, e.g. $p=0.048$ (unless of course it's $p=0.052$). So some excess of $p$-values close but not equal to $0.05$ can be explained by researcher's rounding preferences.
And apart from that, the effect is tiny.
(The only strong effect that I can see on this figure is a pronounced drop of the $p$-value density right after $0.05$. This is clearly due to the publication bias.)
Unless I missed something, Head et al. do not even discuss this potential alternative explanation. They do not present any histogram of the $p$-values either.
There is a bunch of papers criticizing Head et al. In this unpublished manuscript Hartgerink argues that Head et al. should have included $p=0.04$ and $p=0.05$ in their comparison (and if they had, they would not have found their effect). I am not sure about that; it does not sound very convincing. It would be much better if we could somehow inspect the distribution of the "raw" $p$-values without any rounding.
Distributions of $p$-values without rounding
In this 2016 PeerJ paper (preprint posted in 2015) the same Hartgerink et al. extract p-values from lots of papers in top psychology journals and do exactly that: they recompute exact $p$-values from the reported $t$-, $F$-, $\chi^2$- etc. statistic values; this distribution is free from any rounding artifacts and does not exhibit any increase towards 0.05 whatsoever (Figure 4):
$\hspace{5em}$
A very similar approach is taken by Krawczyk 2015 in PLoS One, who extracts 135k $p$-values from the top experimental psychology journals. Here is how the distribution looks for the reported (left) and recomputed (right) $p$-values:

The difference is striking. The left histogram shows some weird stuff going on around $p=0.05$, but on the right one it is gone. This means that this weird stuff is due to people's preferences of reporting values around $p\approx 0.05$ and not due to $p$-hacking.
Mascicampo and Lalande
It seems that the first to observe the alleged excess of $p$-values just below 0.05 were Masicampo & Lalande 2012, looking at three top journals in psychology:

This does look impressive, but Lakens 2015 (preprint) in a published Comment argues that this only appears impressive thanks to the misleading exponential fit. See also Lakens 2015, On the challenges of drawing conclusions from p-values just below 0.05 and references therein.
Economics
Brodeur et al. 2016 (the link goes to the 2013 preprint) do the same thing for economics literature. The look at the three economics journals, extract 50k test results, convert all of them into $z$-scores (using reported coefficients and standard errors whenever possible and using $p$-values if only they were reported), and get the following:

This is a bit confusing because small $p$-values are on the right and large $p$-values are on the left. As authors write in the abstract, "The distribution of p-values exhibits a camel shape with abundant p-values above .25" and "a valley between .25 and .10". They argue that this valley is a sign of something fishy, but this is only an indirect evidence. Also, it might be simply due to selective reporting, when large p-values above .25 are reported as some evidence of a lack of effect but p-values between .1 and .25 are felt to be neither here nor there and tend to be omitted. (I am not sure if this effect is present in biological literature or not because the plots above focus on $p<0.05$ interval.)

Falsely reassuring?
Based on all of the above, my conclusion is that I don't see any strong evidence of $p$-hacking in $p$-value distributions across biological/psychological literature as a whole. There is plenty of evidence of selective reporting, publication bias, rounding $p$-values down to $0.05$ and other funny rounding effects, but I disagree with conclusions of Head et al.: there is no suspicious bump below $0.05$.
Uri Simonsohn argues that this is "falsely reassuring". Well, actually he cites these papers un-critically but then remarks that "most p-values are way smaller" than 0.05. Then he says: "That's reassuring, but falsely reassuring". And here is why:

If we want to know if researchers p-hack their results, we need to examine the p-values associated with their results, those they may want to p-hack in the first place. Samples, to be unbiased, must only include observations from the population of interest.
Most p-values reported in most papers are irrelevant for the strategic behavior of interest. Covariates, manipulation checks, main effects in studies testing interactions, etc. Including them we underestimate p-hacking and we overestimate the evidential value of data.  Analyzing all p-values asks a different question, a less sensible one. Instead of "Do researchers p-hack what they study?" we ask "Do researchers p-hack everything?"

This makes total sense. Looking at all reported $p$-values is way too noisy. Uri's $p$-curve paper (Simonsohn et al. 2013) nicely demonstrates what one can see if one looks at carefully selected $p$-values. They selected 20 psychology papers based on some suspicious keywords (namely, authors of these papers reported tests controlling for a covariate and did not report what happens without controlling for it) and then took only $p$-values that are testing the main findings. Here is how the distribution looks like (left):

Strong left skew suggests strong $p$-hacking.
Conclusions
I would say that we know that there must be a lot of $p$-hacking going on, mostly of the Forking-Paths type that Gelman describes; probably to the extent that published $p$-values cannot really be taken at face value and should be "discounted" by the reader by some substantial fraction. However, this attitude seems to produce much more subtle effects than simply a bump in the overall $p$-values distribution just below $0.05$ and cannot really be detected by such a blunt analysis.
Comment: `simply because the researches chose what test to perform/report after looking at the data` Yes; and the problem is anavoidable because double-edged. When a better method is being chosen for the data - is it an overfitting of that specific sample or a meeting of technical calls of that population? Or - removimg outliers - is it faking the population or recovering it? Who will say, ultimately?
Comment: You got me curious with this Head et al. paper, @Silverfish, so I must confess that right now, instead of working, I am browsing through some papers criticizing Head et al.'s results and have even already downloaded their raw data... Oh my.
Comment: +1. The latest Gelman blog article (http://andrewgelman.com/2016/03/09/bruised-and-battered-i-couldnt-tell-what-i-felt-i-was-ungeneralizable-to-myself/) covers a lot of ground, and highlights an interesting rejoinder by a group that attempted replications and was then strongly criticized by the original study authors: http://retractionwatch.com/2016/03/07/lets-not-mischaracterize-replication-studies-authors/
Comment: The kind of answer I was most hoping for was perhaps a brief representation of the current literature, some pointers as to whether the Head et al paper is a fair summary of the latest thinking, etc. I wasn't expecting this answer at all. But I think it's great, and Gelman's thoughts and the practical insights are particularly helpful. When I wrote the question I had similar things in mind to @ttnphns actually (perhaps it shows, I even considered including the word "overfitting".)
Comment: Nevertheless, aside from the general and inescapable malaise of "how science works in practice" being an imperfect match for the assumptions of statistical testing, I do wonder if this bogeyman "dark art of the malicious p-hackers" is really out there, and if so, just how far it reaches. There are definitely strong (mis)incentives to encourage it.
Comment: I'm aware that Head's paper generated controversy, which was one of the reasons for posting this question. Otherwise I'd not have bothered posting a question after reading the paper. Perhaps I should clarify that in the question body.
Comment: Yes, @Wayne, I have just added an update linking to this stuff. I read this Gelman's post in the morning and it's probably partially why I felt so compelled to write something after seeing Silverfish's question a couple of hours later.
Comment: The update is a big improvement, thanks! The only thing I'd suggest is to replace "today" with the actual date, since in an answer with substantial edits it isn't clear to a new reader which day is "today"!
Answer: Funnel plots have been a tremendous statistical innovation that turned meta analysis on its head. Basically, a funnel plot shows the clinical and statistical significance on the same plot. Ideally, they would form a funnel shape. However, several meta-analyses have produced funnel plots that show a strong bimodal shape, where investigators (or publishers) selectively withheld results that were null. The result is that the triangle becomes wider, because smaller, less powered studies used more drastic methods to "encourage" results to reach statistical significance. The Cochrane Report team has this to say about them.

If there is bias, for example because smaller studies without statistically significant effects (shown as open circles in Figure 10.4.a, Panel A) remain unpublished, this will lead to an asymmetrical appearance of the funnel plot with a gap in a bottom corner of the graph (Panel B). In this situation the effect calculated in a meta-analysis will tend to overestimate the intervention effect (Egger 1997a, Villar 1997). The more pronounced the asymmetry, the more likely it is that the amount of bias will be substantial.

The first plot shows a symmetrical plot in the absence of bias. The second shows an asymmetrical plot in the presence of reporting bias. The third shows an asymmetrical plot in the presence of bias because some smaller studies (open circles) are of lower methodological quality and therefore produce exaggerated intervention effect estimates.

I suspect most authors are unaware of the methods they use to p-hack. They don't keep track of the overall number of models they fit, applying different exclusion criteria or opting for different adjustment variables each time. However, if I had to mandate a simple process, I would love to see the total number of models fit. That's not to say there might be legitimate reasons to rerun models, for instance we just ran through a Alzheimer's analysis not knowing ApoE had been collected in the sample. Egg on my face, we reran the models.
Comment: I like that you emphasize "investigators **(or publishers)** selectively withheld results that were null". Given that fail to reject the null $\approx$ no publication, the fault is not necessarily squarely on the investigators.
Comment: One aspect of my question was the distinction between "p-hacking" and "publication bias" - this answer in some ways conflates the two. Would I be right to interpret what you're saying in that way, ie "publication bias is in essence a form of p-hacking, but by the publisher"?
Comment: Hmm. First I wanted to protest and claim that publication bias is different from p-hacking (similarly, I think, to how @Silverfish framed his Q too), but then I realized that it is more tricky to draw the boundary than I originally thought. Performing jelly-beans-style multiple comparisons and only reporting significant ones (p-hacking?) is not very different from performing multiple studies and only reporting significant ones (which is publication bias by definition). Still, p-hacking in the sense of massaging the data until they yield p<0.05 does feel sufficiently different to me.
Comment: @amoeba I had the same concern, but after reading OP's question, I realized it concerned the consequences of $p$-hacking on the "sausage end of things". Most $p$-hacking methods are usually omitted from reporting. So having been blinded to what the statistician does, how then do we reconcile the differences? Well, we need independent attempts to replicate and confirm findings.
Comment: @Silverfish Publication bias, per the earlier comment, can be driven by either the authors or the publishers. But yes, it is most definitely $p$-hacking. Funnel plots may have been applied to published research, but they are applicable in any setting where "scientific replication" starts to show discrepancies. Confirmatory trials for drugs, or implementations of business policies across a number of centers or wholesalers, no matter. Whenever you are dealing with replications, a funnel plot can provide some evidence of $p$-hacking by showing gaps where null results should have fallen.
Comment: Just came across this beautiful evidence of publication bias / p-hacking via a funnel plot: http://blogs.discovermagazine.com/neuroskeptic/files/2015/11/funnel_shanks1.png from this paper https://www.ncbi.nlm.nih.gov/pubmed/26501730 (unfortunately paywalled). Picture taken [from here](http://blogs.discovermagazine.com/neuroskeptic/2015/11/10/reproducibility-crisis-the-plot-thickens/#.WIpbsGfavCJ).


Fit a non-linear model in R with restrictions

Question: I'm trying to fit a model to some data in R. To simplify the question, I'll use an example.
If the model that describes my process is y = f(a, b) where a cannot be greater than b. Is there a way I can fit the model with this restriction?
I'm using nlsLM but am open to other methods/packages if necessary.
Answer: The package nlsLM and many other software libraries that perform non-linear regression have the possibility to use upper and lower limits for the parameters to be estimated.
When you rewrite your function as $$f(a,a+c)$$ by substituting $b=a+c$, then the restriction $a\leq b$ becomes $c \geq 0$ and the latter restriction you can regulate with upper/lower boundaries.

Other ways are

(re-)write your own optimisation algorithm.
I imagine this could in some cases improve performance, ie increase convergence rate.

Substitute $f(a,a+e^c)$ such that the non-negative limit is obtained without specifying a boundary, but by using a function that is non-negative.

How to do approach this exactly will depend on your actual problem. There should be several examples here already. (To be honest they might not be easy to find, or at least I tried a few search prompts and they did not reveal them. Yet, I believe that I have seen these type of questions before)
Comment: your second "Other way" seems like the play for anyone not comfortable with optimization, since many nonlinear least squares solvers in R don't allow even box constraints.
Comment: Method 1 worked perfectly. I specified `lower` with `nlsLM` to prevent the difference between `a` and `b` being less than 0.


Which optimization algorithm converges faster?

Question: everyone. I'm having a large scale unconstrained optimization problem. If I treat the unconstrained problem as a constrained problem with infinity constraints, I should be able to use both the fminunc and fmincon function in MATLAB. When using the fminunc function, I should provide the gradient and the sparse pattern of the Hessian. While using the fmincon function, I can choose the l-bfgs method to approximate the Hessian.
So, here is my question. Which one converges faster? Are there any good reasons to choose one over another? Or any principles to guide such choice?
Update:
Actually the problem is a least-squares problem. However the number of terms is big, so the Jacobian would be too large to store. This is the reason why I don't use the lsqnonlin function. The algorithm used in fminunc for large scale problem is a trust-region method (details can be found in fminunc documentation), and the algorithm in fmincon is l-bfgs (see fmincon documentation). What I'm asking is more of a generative comparison because there are many C/C++ implementations of these algorithms. Answers may not be necessarily  constrained by MATLAB. Thanks!
Update 2:
I was unclear, the form of my problem is like this:
$$
\min_W (\sum_i\|Y_i-WX_i\|_2^2).
$$
Update 3:
Thank you all! littleO is right. But I'm still curious about the efficiency of the methods mentioned above under general condition.
Comment: My suggestion is to run a minimal test problem that inherits the structure of a larger problem you wanna solve. Otherwise it is hard to tell, I would put my bet on `fminunc`.
Comment: Since you're using a Frobenius norm, it will be possible to express your problem as a least squares problem, and then solve it using a least squares solver from numerical linear algebra.  This should be much more efficient than using a general purpose solver.
Comment: Since your problem is unconstrained, I'd guess fminunc would be faster. But can you tell us more about the function you're minimizing?
Comment: Are $Y_i$ and $X_i$ matrices?  If so, are you using $\| \cdot \|_2$ to denote what is usually called the [Frobenius norm](http://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm) of a matrix?  Or is it actually an [induced norm](http://en.wikipedia.org/wiki/Matrix_norm#Induced_norm) ?
Comment: I think this minimization problem has a closed-form solution!
Comment: They are matrices, and the norm is Frobenius norm. Nevertheless, they can be rewritten to vector forms.
Comment: Really? What is the closed-form?
Comment: Yes, you are right. But I'm still curious about the efficiency of both methods.
Answer: If your problem is truly a least squares problem:
\begin{equation}
\text{minimize}_x \quad \|Ax - b\|_2^2
\end{equation}
then you should probably use a technique from numerical linear algebra, rather than some general purpose optimization algorithm.  Least squares problems have been studied extensively in numerical linear algebra and very good algorithms / libraries exist for least squares problems.
In Matlab, you can solve least squares problems just using the backslash operator:

x = A\b ;

$A$ can be stored as a sparse matrix in Matlab if it's huge.
Edit: Your specific problem can be expressed as a least squares problem.  First note that
\begin{align}
\|Y_i - W X_i\|_F &= \| Y_i^T - X_i^T W^T \|_F \\
&= \| B^i - A^i Z \|_F
\end{align}
where I have changed notation in the last line. (Note that $W^T = Z$.)
Furthermore,
\begin{align}
\| B^i - A^i Z \|_F &=
\left \|
\begin{bmatrix}
B^i_1 - A^i Z_1 \\
\vdots \\
B^i_N - A^i Z_N
\end{bmatrix} \right \|_2
\end{align}
where $B^i_1,\ldots,B^i_N$ are the columns of $B^i$ and $Z_1,\ldots,Z_N$ are the columns of $Z$.
Let
\begin{equation}
\tilde{B}^i =
\begin{bmatrix}
B^i_1 \\ \vdots \\ B^i_N
\end{bmatrix}
\end{equation}
and
\begin{equation}
\tilde{A}^i =
\begin{bmatrix} A^i & & \\
& \ddots & \\
& & A^i
\end{bmatrix}
\end{equation}
and
\begin{equation}
\tilde{Z} = \begin{bmatrix} Z_1 \\ \vdots \\ Z_N \end{bmatrix}.
\end{equation}
Then
\begin{equation}
\|B^i - A^i Z \|_F = \|\tilde{A}^i \tilde{Z} - \tilde{B}^i \|_2.
\end{equation}
Note that $\tilde{Z}$ and $\tilde{B}^i$ are column vectors while $\tilde{A}^i$ is a matrix, so we are close to expressing our problem as a standard least squares problem.
Next note that
\begin{align}
\sum_{i=1}^K \|Y_i - W X_i \|_F^2 &= \sum_{i=1}^K \|\tilde{A}^i \tilde{Z} - \tilde{B}^i \|_2^2.
\end{align}
Let
\begin{equation}
A = \begin{bmatrix} \tilde{A}^1 & & \\
& \ddots & \\
& & \tilde{A}^K \end{bmatrix}
\end{equation}
and
\begin{equation}
b = \begin{bmatrix} \tilde{B}^1 \\ \vdots \\ \tilde{B}^K \end{bmatrix}.
\end{equation}
We finally see that your problem is equivalent to
\begin{equation}
\text{minimize}_{\tilde{Z}} \quad \|A \tilde{Z} - b \|_2^2.
\end{equation}
An analytical solution is given by
\begin{equation}
\tilde{Z} = (A^T A)^{-1} A^T b.
\end{equation}
($\tilde{Z}$ satisfies the normal equations.)
However, in practice you should probably compute $\tilde{Z}$ using a least squares solver.  In Matlab, it's just

Ztilde = A\b ;
Comment: I was unclear. I've added details about my problem.
Answer: If you can approximate the action of the Jacobian, I'd try fminunc with the HessMult option. This amounts to trust-region Gauss-Newton. In terms of data storage, you don't need the actual Jacobian of the map (or Hessian of the objective function), only a function that can apply it to vectors. L-BFGS is basically a clever way to approximate the action of the inverse Hessian, so why not actually use the Hessian itself?
In detail, suppose you want to solve the following least-squares optimization problem,
$$\min_x \frac{1}{2}||F(x)-y||^2,$$
where $F$ has a Taylor expansion at $p$,
$$F(x) = F_0 + J (x-p) + \dots .$$
Then the objective function can be expanded to,
$$\frac{1}{2}||F(x)-y||^2 = \frac{1}{2}[F_0 + J (x-p) + \dots -y]^T[F_0 + J (x-p) + \dots -y] \\
=\frac{1}{2}||F_0-y||^2 + (F_0-y)^T J (x-p) + \frac{1}{2}(x-p)^T J^T J(x-p) + \dots  \\
= A + G (x-p) + (x-p)^T H (x-p) + \dots .$$
This is not a true second order Taylor series of the objective function because of possible interactions of $A,G,H$ with terms in the $\dots$'s. However, it is the Gauss-Newton approximation which is generally well accepted. The motivation for this is to note that at the true solution, we expect the misfit will be small. Ie, $``(F_0-y)^T = O(x-p)"$.
You need functions, say myGradient(p) and myHessian(p,w), that compute $G = (F_0-y)^T J$ and the action $Hw = J^TJw$ on an input vector $w$. Then, simply pass the function handles @myGradient & @myHessian as arguments to fminunc.
fminunc will then apply Newton's method to the optimization problem,
$$x_{k+1}=x_k - H^{-1} G x_k.$$
The inverse $H^{-1} G x_k$ is applied through conjugate gradient, each step of which involves evaluating products like $Hw$. Whenever the action of the Hessian, $Hw$, is needed, fminunc will call your myHessian(p,w) function.
Thus you never need to store the full Jacobian of the forward map (Hessian of the objective function).
Edit: This method is exact after 1 Newton step for the linear least squares problem in the updated version of the question, as it amounts to using conjugate gradient on the normal equations.


VBA Macro to open/save/close workbooks in folder and subfolders

Question: I have the following code that will open/save/close any/all workbooks in a folder. It works great, however, I also need it to include sub folders. The code needs to work without restrictions on the number of folders, sub folders and files, if possible.
I'm working with Excel 2010 and I'm new to VBA  - would really appreciate any help!
Sub File_Loop_Example()
    'Excel VBA code to loop through files in a folder with Excel VBA

    Dim MyFolder As String, MyFile As String

    'Opens a file dialog box for user to select a folder

    With Application.FileDialog(msoFileDialogFolderPicker)
       .AllowMultiSelect = False
       .Show
       MyFolder = .SelectedItems(1)
       Err.Clear
    End With

    'stops screen updating, calculations, events, and statsu bar updates to help code run faster
    'you'll be opening and closing many files so this will prevent your screen from displaying that

    Application.ScreenUpdating = False
    Application.DisplayStatusBar = False
    Application.EnableEvents = False
    Application.Calculation = xlCalculationManual

    'This section will loop through and open each file in the folder you selected
    'and then close that file before opening the next file

    MyFile = Dir(MyFolder & "\", vbReadOnly)

    Do While MyFile <> ""
        DoEvents
        On Error GoTo 0
        Workbooks.Open Filename:=MyFolder & "\" & MyFile, UpdateLinks:=False
        ActiveWorkbook.Save
        Workbooks(MyFile).Close SaveChanges:=True
        MyFile = Dir
    Loop

    'turns settings back on that you turned off before looping folders

    Application.ScreenUpdating = True
    Application.DisplayStatusBar = True
    Application.EnableEvents = True
    Application.Calculation = xlCalculationManual

    MsgBox "Done!"

    End Sub

Comment: You can use: https://stackoverflow.com/questions/22645347/loop-through-all-subfolders-using-vba
Comment: @Error1004 reply should give you your answer but please be aware of **recursive** UDF's. search for something like: **`recursive functions and its pitfalls`**
Comment: How would I incorporate the code in the @Error 1004 reply above? Or is it a case of incorporating my existing code into that? Apologies if this sounds dumb - I'm VERY new to this! Also, i would like to keep the folder picker that I have - would this be possible?
Answer: For anyone interested, I found an alternative which I managed to adapt and does exactly what I want:
Sub Loop_Example()

Dim MyFolder As String
Dim file As Variant, wb As Excel.Workbook

With Application.FileDialog(msoFileDialogFolderPicker)
       .AllowMultiSelect = False
       .Show
       MyFolder = .SelectedItems(1)
       Err.Clear
    End With

Application.ScreenUpdating = False

For Each file In Filter(Split(CreateObject("WScript.Shell").Exec("CMD /C DIR """ & startFolder & "*.xl*"" /S /B /A:-D").StdOut.ReadAll, vbCrLf), ".")
    Set wb = Workbooks.Open(file)
    ActiveWorkbook.Save
    wb.Close SaveChanges:=True
    Set wb = Nothing
Next

Application.ScreenUpdating = True

    MsgBox "Done!"

End Sub


Union query wrong execution

Question: I'm trying to execute the query below but the problem is that it shows execute a bunch of nonsense what I want it to show those who are female dependent and there associated employee first name  ....... where is my failing point + I have to use union operator
 SELECT Dep_FName AS 'First name' ,Dep_LName AS 'Last name'  ,Em_FName AS 'Employee"s First name'
 FROM evnt_db.dependent,evnt_db.employee
 where Dep_gender='F'
Union SELECT Dep_FName AS 'First name'  ,Dep_LName  AS 'Last name'  ,Em_FName AS 'Employee"s First name'
 FROM evnt_db.dependent as d,evnt_db.employee as e
 where d.Emp_Id=e.Em_Id ;

Comment: When you execute the second part, do you get results? Do you double checked?
Comment: Maybe try removing the parenthesis around the Select statements. I'm not sure this is the problem but the Union(Select part does not look right to me.
Comment: Your column name aliases do not match up and they have to for a UNION statement. 'Last name ' has a space at the end of the first one, but not the other.
Answer: Union only selects distinct values, so maybe it is filtering overlap out. Also I don't know why you need to use Union for this statement since you are selecting from the same table. You could just use an OR in the where of the first half of the query.
SELECT Dep_FName AS 'First name ',Dep_LName AS 'Last name '
 FROM evnt_db.dependent
 where Dep_gender='F' OR Relationship='Wife';

If you must use UNION, this should work. I think you query had something weird with the parenthesis. Also the column names have to be identical in a UNION statement. There was an extra space on 'Last name ' and not on the other 'Last name'
SELECT Dep_FName AS 'First name ',Dep_LName AS 'Last name '
 FROM evnt_db.dependent
 where Dep_gender='F'
UNION
SELECT Dep_FName AS 'First name ',Dep_LName  AS 'Last name '
 FROM evnt_db.dependent
 where Relationship='Wife';

If you want to get the duplicate records (prevent union from filtering duplicates), use UNION ALL instead of just UNION
Comment: I think you are right, but to select duplicate values OP could use UNION ALL.
Comment: Yeah, I should add that as an alternate answer
Comment: yah thank you I did try that and it worked but in the requirement for the project is implementing the union operator ..... so the union operator is only for two different tables ??
Comment: You can use the union operator for the same table, but an OR is better I think.
Comment: UNION ALL still is UNION. That's the way you want to go.
Comment: Kevin, you can edit your answer whenever you want :)
Comment: still the same :(((
Comment: I just noticed another error where the column aliases did not match. That may have prevented results from the second part of the query from showing. I edited my answer.
Comment: Does the second query run on its own and get results?
Comment: Looks like you have the same problem where there is a space after the first 'Last name ' but not the other. Column aliases must be identical in a UNION statement. Just add a space to 'Last name'
Comment: i edited the union to something else please help me with that
Comment: What's wrong with it now? Do both selects execute properly on their own? Get both of them to find results when they execute properly on their own before trying to put them together with a UNION.
Answer: UNION by default removes duplicate result rows, if you want those duplcates you can use UNION ALL.
If you don't want those duplicates, your current query already takes care of that; though you should also consider Kevin's answer in that case.
In any case, you should only expect to see results from the latter query if you have dependent rows that are non-'F' wives.

Union query wrong execution


Extracting lines from file using grep in a for loop, exporting to new file with variable in file name

Question: I am trying to extract all lines from a file that contain a string using a for loop with a file that contains a list of possible strings. I also want to export the results of grep to a new file with the variable in the file name.
Here is what I have:
file="variables.txt"
listofvariables=$(cat ${file})

for variable in ${listofvariables}
do
    samtools view sample.bam | \
    grep "'${variable}'" \
    > sample.${variable}.bam
done

What this code does is simply make a blank file for every variable. Why isn't grep extracting lines that contain that variable and putting it into those files?
For reference, here is what the variables.txt file looks like:
mmu-let-7g-5p
mmu-let-7g-3p
mmu-let-7i-5p
mmu-let-7i-3p
mmu-miR-1a-1-5p
mmu-miR-1a-3p
mmu-miR-15b-5p
mmu-miR-15b-3p
mmu-miR-23b-5p
mmu-miR-23b-3p

And here is what the samtools view output looks like:
7238520-1_CATAAT.mmu-miR-125b-5p    0   chr1    11301523    60  75M *   0   0CAGGTGTTTTCTCAGGCATTTGGATTTCTATAGAATCATAGTATTAAAATTTCAAAGTAATAACATTGCTTTTTA    IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:75 YT:Z:UU NH:i:1
1422982-2_CCCCGC.mmu-miR-132-3p 0   chr1    11301726    60  97M *   0   0   AAGTCTGTTTTTATGTGAGTGTTCCTGTGAAACTGAGGTCTGATGACTCTTCCTTAAGCAATTACAACTTCATTAGCATACATAAGGTTCAATTAAA   IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:97 YT:Z:UU NH:i:1
5675450-1_CCCCGC.mmu-miR-132-3p 0   chr1    11301726    60  97M *   0   0   AAGTCTGTTTTTATGTGAGTGTTCGTGTGAAACTGAGGTCTGATGACTCTTCCTTAAGCAATTACAACTTC^C

For those who may be unfamiliar samtools view simply reads out the .bam file. You can think of it like cat.
Thanks in advance!
Comment: `grep "'${variable}'"`  is looking for `${variable}` to be wrapped in single quotes **in** the `samtools` output, so first idea would be to try `grep "${variable}"` (sans the single qutoes); also, I don't see any of the sample strings (from `variables.txt`) in the sample `samtools` output so I would expect your script to create a bunch of empty files
Answer: Since ...

What this code does is simply make a blank file for every variable.

... you know that your variables file is being read correctly, and your for loop is correctly iterating over the results.  That the resulting files are empty indicates that grep is not finding any matches to your pattern.
Why not? Because the pattern in your grep command ...

    grep "'${variable}'" \

... doesn't mean what you appear to think it means.  You have taken some pains to get literal apostrophes (') into the pattern, but these have no special meaning in that context.  Your pattern does not match any lines because in the data, there are no apostrophes around the appearances of the target strings.
This would be better:
    grep -F -e "${variable}" \

The -F option tells grep to treat the pattern as a fixed string to match, so that nothing within is interpreted as a regex metacharacter.  The -e ensures that the pattern is interpreted as such, even if, for example, it begins with a - character.  The double quotes remain, as they are required to ensure that the shell does not perform word splitting on the expanded result, and of course the inner apostrophes are gone, since they were causing the main problem.


Parallel task execution breaks the order when parallel ,normal tasks mixed together

Question: apply plugin: 'java'

task taskA <<{
    println 'task A'
}

task taskB (type:P){
    namee='B'
}

task taskC (type:P){
    namee='C'
}

@ParallelizableTask
class P extends DefaultTask {
    String namee

    @TaskAction
    def broccoliBuild() {
        println "Task "+namee
    }
}

task taskBC (dependsOn:['taskB','taskC'])<<{
    println 'BC'
}
taskBC.mustRunAfter taskA

task taskD <<{
    println'task D'
}
taskD.mustRunAfter taskBC

task taskE (type:Qtask){
    nname='E'
}

task taskF (type:Qtask){
    nname='F'
}

@ParallelizableTask
class Qtask extends DefaultTask {
    String nname

    @TaskAction
    def broccoliBuild() {
        println "Task "+nname
    }
}

task taskEF (dependsOn:['taskE','taskF'])<<{
    println 'task EF'
}
taskEF.mustRunAfter taskD

task main(dependsOn: ['taskA','taskBC','taskD','taskEF'])<<{
    println 'Build Completed'
}

When i execute the main task in parallel mode:

gradle main --parallel -Dorg.gradle.parallel.intra=true

taskEF executes along with taskBC (Both of them is parallel tasks) which violates the flow i have mentioned. taskD must be executed before taskEF (I have used mustRunAfter ) . Please can any one help me out of this. (Used Gradle 2.12)
Comment: Can you please provide your Gradle output? I run the script locally many times and `taskD` was always executed before `taskEF`.
Answer: After the contents of your build.gradle file and adding the wrapper task:
task wrapper(type: Wrapper) {
    gradleVersion = '2.12'
}

I ran the following command: gradlew wrapper and then checked the version, which shows I am properly running the expected Gradle 2.12 version
E:\Data\Test>gradlew --version

------------------------------------------------------------
Gradle 2.12
------------------------------------------------------------

Build time:   2016-03-14 08:32:03 UTC
Build number: none
Revision:     b29fbb64ad6b068cb3f05f7e40dc670472129bc0

Groovy:       2.4.4
Ant:          Apache Ant(TM) version 1.9.3 compiled on December 23 2013
JVM:          1.8.0_111 (Oracle Corporation 25.111-b14)
OS:           Windows 10 10.0 amd64

Then, running the command you provided, here is the results:
E:\Data\Test>gradlew main --parallel -Dorg.gradle.parallel.intra=true

Parallel execution is an incubating feature.
:taskA
task A
:taskB
:taskC
:taskE
:taskF
Task C
Task B
Task F
Task E
:taskBC
BC
:taskD
task D
:taskEF
task EF
:main
Build Completed

BUILD SUCCESSFUL

Total time: 2.609 secs

So, with Gradle 2.12 the output does run as you expect and the taskEF correctly runs after taskBC.
It appears that you are either not running the proper version of Gradle you intend or the version of the build.gradle you are actually running against is different than the one you provided here.
As a side note: I would suggest always run Gradle using the wrapper task as I have shown, which makes updating or choosing a specific Gradle version a snap.


How to Extend IOS SimpleURLConnections HTTP POST to Send Description Decodable by CGI Perl

Question: I would like to extend the sample ios code called SimpleURLConnections. This sample code can upload a file using HTTP POST. I have enabled my apache web server with a cgi script written in Perl to receive the HTTP POST that addresses the cgi script and successfully uploads the file. The extension I would like to make is to add a title or description of the uploaded file. I have tried to update the multipart form data in the sample code and a corresponding change to my cgi script to transfer this description, but it is failing. The cgi script cannot read in the description using the cgi.pm method called param(). I'm hoping someone can help. Here are the details:
The multipart form data BEFORE my modification looks like the following (NOTE: the SimpleURLConnections routine inserts the image file between the Content-Type and the 2nd boundary on the fly):
--Boundary-D4AFFBA9-AD92-4697-9184-7BDA128C3B97
Content-Disposition: form-data; name="fileContents"; filename="1234567/TestImage1.png"
Content-Type: image/png

--Boundary-D4AFFBA9-AD92-4697-9184-7BDA128C3B97
Content-Disposition: form-data; name="uploadButton"

Upload File
--Boundary-D4AFFBA9-AD92-4697-9184-7BDA128C3B97--

The multipart form data AFTER my modification looks like the following:
--Boundary-D4AFFBA9-AD92-4697-9184-7BDA128C3B97
Content-Disposition: form-data; name="Title"; title="Hello World"
Content-Type: text/plain
--Boundary-D4AFFBA9-AD92-4697-9184-7BDA128C3B97
Content-Disposition: form-data; name="fileContents"; filename="1234567/TestImage1.png"
Content-Type: image/png

--Boundary-D4AFFBA9-AD92-4697-9184-7BDA128C3B97
Content-Disposition: form-data; name="titleButton"
my Title

--Boundary-D4AFFBA9-AD92-4697-9184-7BDA128C3B97
Content-Disposition: form-data; name="uploadButton"

Upload File
--Boundary-D4AFFBA9-AD92-4697-9184-7BDA128C3B97--

My cgi script looks like the following:
#!/usr/bin/perl -wT
use strict;use CGI;
use CGI::Carp qw ( fatalsToBrowser );
use File::Basename;

my $upload_dir = "/Library/WebServer/Documents/upload";

my $query = new CGI;

my $title = $query->param("Title");

#if (!$title) {
#    die "the title is $title. $!";
#}

if ( !$filename ) {
        print $query->header ( );
        die "There was a problem uploading your photo (try a smaller file). : $!";
        exit;
}
...

open ( UPLOADFILE, ">$upload_dir/$filename" ) or die "$!";
while ( <$upload_filehandle> ) {
    print UPLOADFILE;
}

close UPLOADFILE;

This code works fine and the uploaded file gets written to the upload_dir.
But, if I uncomment that little block of code:
if (!$title) {
    die "the title is $title. $!";
}

the script dies with an error that indicates that $title has no value:
[Tue Sep 27 17:09:17 2011] [error] [client 10.0.1.2] [Tue Sep 27 17:09:17 2011] upload.cgi: Use of uninitialized value $title in print at /Library/WebServer/CGI-Executables/upload.cgi line 79.

So, is the problem with my multipart form data, the cgi script, or is it not possible to pass this data in the multipart form data? Is there an alternate way?
Thanks
Ok, after getting help from @AFresh1, I found the problem. The problem was in the details of how the multipart form data was created. First I will show the wrong way.
NSString *title = @"Hello World";
bodySuffixStr = [NSString stringWithFormat:     // create the footer for the POST message
            @
            "\r\n"
            "--%@\r\n"
            "Content-Disposition: form-data; name=\"titleButton\"\r\n"
            "%@\r\n\r\n"
            "--%@\r\n"
            "Content-Disposition: form-data; name=\"uploadButton\"\r\n"
            "\r\n"
            "Upload File\r\n"
            "--%@--\r\n"
            "\r\n",
            boundaryStr,
            title,
            boundaryStr,
            boundaryStr
        ];

Now the correct way (a very subtle change in the 7th line - moved the CR/LF):
NSString *title = @"Hello World";
bodySuffixStr = [NSString stringWithFormat:     // create the footer for the POST message
            @
            "\r\n"
            "--%@\r\n"
            "Content-Disposition: form-data; name=\"titleButton\"\r\n"
            "\r\n%@\r\n"
            "--%@\r\n"
            "Content-Disposition: form-data; name=\"uploadButton\"\r\n"
            "\r\n"
            "Upload File\r\n"
            "--%@--\r\n"
            "\r\n",
            boundaryStr,
            title,
            boundaryStr,
            boundaryStr
        ];

Answer: I believe the content goes in the space after the header
and before the next boundary not in a title attribute.  The part header being everything after the boundary and before the blank line.
If you are want to get "Hello World" into $title using $query->param('Title') I think you will need something more like this (note that Content-type defaults to text/plain):
--Boundary-D4AFFBA9-AD92-4697-9184-7BDA128C3B97
Content-Disposition: form-data; name="Title"

Hello World
--Boundary-D4AFFBA9-AD92-4697-9184-7BDA128C3B97

Comment: Not sure how I missed that, but does it really say "the title is $title.$!"? That would explain more why it outputs 0, because there wasn't likely an error that would set $!. That doesn't really explain why it won't get the form data.  Perhaps try `use Data::Dumper` and then `die Dumper $query` right after `my $query = new CGI;` to see what is in the object?
Comment: no, sorry. If I understood the change you suggested correctly, that does not seem to work.
Comment: I believe it should work, in my testing it does. Can you try with the original working code (without your additions) and see if you can get the value of `$query->('titleButton')`?
Comment: Ok, so I misunderstood where you were suggesting the change should be. I have made the change, but it is still not working (although the error is slightly different. First, just to be clear, here is the changed multipart form data:
Comment: --Boundary-CD342BC3-E13C-4CEA-A896-3CC9F04283FE
    Content-Disposition: form-data; name="fileContents"; filename="1234567/TestImage1.png"
    Content-Type: image/png

    --Boundary-CD342BC3-E13C-4CEA-A896-3CC9F04283FE
    Content-Disposition: form-data; name="titleButton"
    Hello World

    --Boundary-CD342BC3-E13C-4CEA-A896-3CC9F04283FE
    Content-Disposition: form-data; name="uploadButton"

    Upload File
    --Boundary-CD342BC3-E13C-4CEA-A896-3CC9F04283FE--
Comment: The error used to be a 2 line error. The first line at the web server error log said "Use of uninitialized value $title in concatenation (.) or string". The second line used to say "the title is . ". That second line is my "die" statement resulting from $title not having a value. The new error only contains my "die" error. It says "the title is . 0".  That "0" is the error message passed from the system (which is not very helpful). But the system doesn't think the $title is uninitialized, but my script doesn't think there is a value there. Any further thoughts?
Comment: The new form-data is hard to read in the comment but I don't see a part with the name "Title".  However, `0` is a value (and not uninitialized), it just evaluates to false. I agree it is not  what you want. Does this new error come when you are trying to get the value of titleButton?
Comment: Based on your earlier comment, it looked like you were suggesting that I should be putting my title "Hello World" after the Content-Disposition that contained the name="titleButton". I came to that conclusion because your comment said query->('titleButton'). Is that correct? To answer your question, the error comes as a result of my cgi script trying to set the value of $title. Specifically, the script says: my $title = $query->param("titleButton"); if (! $title) { die "the title is $title. $!"; }
Comment: It should not matter where you put it as long as the multipart form-data is formatted properly.  However, since you moved "Hello World" into the part with the name attribute of titleButton you need to `$title = $query->param('titleButton')` but that was just to make sure you could get any useful data out of the submitted form.
Comment: ok, but as you can see from the previous comment, that how my script is written. Notice that the text "Hello World" is on a line of its own. Is that correct? Here is the form data again (wish I could make this look pretty): 'Boundary...' 'Content-Disposition...filename=...' 'Content-Type...' ''  '--Boundary...' 'Content-Disposition...name="titleButton' 'Hello World'  '--Boundary...' 'Content-Disposition...name="uploadButton" 'Upload File'  '--Boundary...'
Comment: good tip on dumper. Not sure what the dump means though. The title is in the dump. Here is what I think is relevant:                         '.parameters' => [                                                                           'fileContents',                                                                          'titleButton'                                                        ],
Comment: followed by 'titleButton' => ['Hello World'], 'fileContents' => [$VAR1->{'.tmpfiles'}{'*Fh::fh000011234567/TestImage1.png'}{'hndl'}].
Comment: Actually, what I just showed you from the dump is actually working. Prior to running that dump, I changed the way I created the multipart form data. I moved the CR/LF that preceded the 'Hello World'. I will create an answer so I can show the change. Thanks for the tips


Postfix/Dovecot ++ mail server. Can send mail but not receive mail

Question: Some back story
I have the 3 last days struggled hard to try to get a mail server up and run. After first trying to get it up on a server with blocked ports I decided yesterday after several reinstalls to transfer my server over to linode.com so I dont have the struggels with blocked ports.
I finally have a mail server up and running with Postfix, Dovecot, SpamAssassin, ClamAV, Sieve and Roundcube. And even tho my server dont have blocked ports I'm running all mail through dynu.com with a SMTP Outbound Relay and Email Store/Forward for the extra security layer and storing/backup if my server for some reason should be unavailable.
The Problem
Now to the problem, I can't recive any mail for some reason. But I can send mail just fine from the terminal, roundcube and from my iphone and it gets to the reciver without any problems.
When I check the logs dynu acctualy try's to deliver but something are failing along the way as this is what the log contains after a delivery request
Dec 22 22:44:17 rajohan postfix/smtpd[22192]: connect from mx1.dynu.com[ip address]
Dec 22 22:44:18 rajohan postfix/smtpd[22192]: lost connection after UNKNOWN from mx1.dynu.com[ip address]
Dec 22 22:44:18 rajohan postfix/smtpd[22192]: disconnect from mx1.dynu.com[ip address] unknown=0/1 commands=0/1

I would be really grateful if someone could take a look at my master, main and dovecot config's and see if they can see why. I'm by no means a linux expert, in fact I just started toying with linux about 1 month ago and are now feeling quite lost on what to do.
My master.cfg: https://pastebin.com/pH12rCq7
My main.cfg: https://pastebin.com/uZCFHNzS
My dovecot.cfg  https://pastebin.com/9SCKz1dj

And here are some of my DNS settings

MX record 1 and 2 is applied to mail.rajohan.no
mail.rajohan.no has a A record to the server ip
rajohan.no has a A record to the server ip
Reverse DNS is applied to rajohan.no (not sure if this should be
    applied to mail.rajohan.no and not rajohan.no)
DKIM is applied to mail._domainkey.relay.rajohan.no
SPF is applied to rajohan.no

And another question when i set hostname in terminal to rajohan would what overwrite localhost? Or is thoose 2 separate things?
Comment: This isn't a site for hiring people for help.
Answer: Your mail server is not configured to allow unauthenticated senders.  This would be normal on port 587 (Submission) but is incorrect for an MX which must allow unauthenticated user to send mail (but not relay to other domains).
$: telnet MX1.DYNU.COM 25
Trying 184.108.40.206...
Connected to MX1.DYNU.com.
Escape character is '^]'.
220 mx1.dynu.com ESMTP
EHLO mail.systemajik.com
250-mx1.dynu.com
250-SIZE
250-STARTTLS
250-AUTH LOGIN
250 HELP
Comment: I dont really understand what you are saying. Would't reciving mail from the relay server be the same as reciving mail from whoever  else? And i can send mail just fine through the relay thats not a problem. The issue is reciving mail other people send to me
Comment: @Rajohan They are entirely different and have different naming in the documentation,   Sending servers usually have the MTA (Mail Transfer Agent) rle, while a server for general incoming mail has the MX (Mail eXchange) role.  Your server was definitely not configured as an MX when I checked it.
Answer: I managed to fix this by just chainging connection security from SSL/TLS to STARTTLS on dynu. Everything is working perfectly now!