Is it correct to model percent change as a continuous variable?

Question: I have data from a longitudinal study where there are 46 subjects, each with 13 observations over time on a given outcome measure.  A prior analysis conducted on this data performed a linear mixed model on the percent change (treatment, baseline value, time, and treatment*time were independent variables in the model).  I am wondering if using raw change as the outcome variable is more correct, especially since baseline value is controlled in the model anyway.
Answer: Whether this is the best approach depends on what your data looks like, and why the percent change was used.  For example, % change is fairly easy to understand by laypeople, and so it's sometimes preferred when that's the target audience. Including the baseline as a covariate somewhat complicates how that percent would be stated, though, since the expected change would depend on the initial value.
Certainly, to use percent change, the data must be on a scale where 0 means 0.  That is, a 0 must mean an absence of disease or similar.
I think this article does a pretty good job of summarizing some problems associated with modeling percent change: http://allenfleishmanbiostatistics.com/Articles/2012/06/18-percentage-change-from-baseline-great-or-poor/.

In all cases, Analysis of Covariance (ANCOVA) with baseline as the
  covariate was the most efficient statistical methodology.  Analyzing
  the change from baseline "has acceptable power when correlations
  between baseline and post-treatment scores are high; when correlations
  are low, POST [i.e., analyzing only the post-score and ignoring
  baseline – AIF] has reasonable power.  FRACTION [i.e., percentage
  change from baseline – AIF] has the poorest statistical efficiency at
  all correlations."
[Note: In ANCOVA, one can analyze either the change from baseline or
  the post treatment scores as the d.v.  'Change' or 'Post' will give
  IDENTICAL p-values when baseline is a covariate in ANCOVA.]
As an example of his results, when the correlation between baseline
  and post was low (i.e., 0.20) the percentage change was able to be
  statistically significant only 45% of the time.  Next worse, was
  change from baseline with 51% significant results.  Near the top was
  analyzing only the post score at 70% significant results.  The best
  was ANCOVA with 72% significant results.
Furthermore, percentage change from baseline "is sensitive in the
  characteristics of the baseline distribution."  When the baseline has
  relatively large variability, he observed that "power falls."

Also a potential concern: the ratio will blow up as the denominator approaches 0, which can create obvious problems when modeling.
Answer: The best way to do this is to model the final score as a function your listed independent variables PLUS the baseline score, like this:
final ~ baseline + treatment + time + treatment*time
There are a few important considerations:

You should not model % change (final-baseline/baseline) and include baseline as a predictor, because these variables are structurally correlated. Even worse might be to model raw change (final - baseline) with baseline as a predictor. As baseline changes, you expect the dependent variables to change in both cases, all else equal. Conclusion: If you use % or raw change, do not include baseline as a predictor.
Modelling percentages as continuous variables is a dying approach, because these variables are typically not normally distributed, by virtue of being bounded at 0 and 1, and also see last comment by TiffTiff. The arcsine square-root (aka angular) transformation has traditionally been used to combat this problem, but is also falling out of favor due to highly conditional efficacy (i.e., it often doesn't work well).
Modelling raw change, even when baseline is not included as a predictor, is not ideal because you in essence constrain the relationship between baseline and final to be 1:1. Basic algebra to see why:

Here is the raw change model:
final - baseline ~ treatment

which is more precisely,
1*final - 1*baseline ~ b0 + b1*treatment + error

where b0 and b1 are parameters to be estimated. if you rearrange, you get:
1*final ~ b0 + 1*baseline + b1*treatment + error

hence, the parameter describing the relationship between baseline and final is set to 1.0.
If instead you model like this:
final ~ baseline + treatment

that is more precisely
1*final ~ b0 + b1*baseline + b2*treatment + error

If indeed there is not a 1:1 relationship between baseline and final, the parameter estimate b1 will be greater than or less than 1. If it turns out that b1 ≠ 1, you will have more power to test the effect of treatment than if you used raw change, and also get information about the relationship between baseline and final. On the other hand, if b1 = 1, then you should have less power, since you're using up one more degree of freedom.
As far as reporting these stats, I understand why you want to say "The difference between scores increased when the treatment was applied": You want to control for the baseline. But that's exactly what the final ~ baseline + treatment model does, and it does it better. You can say "Scores were higher with the treatment, controlling for the baseline score".
Comment: Very nice answer!
Comment: What if there is a relationship between your baseline and treatment (this violates an assumption of ANCOVA). Would it be OK to do a %change ~ treatment model?
In other words would a %change ~ treatment model increase the chance of a Type I or II error?
Comment: @tim.farkas Just a note on the 1st point. Opinion varies across domains. In clinical research, change from baseline adjusted for baseline is very often kind of analysis, demanded by both sponsors and regulators. Its role is to account for the fact, that patients start from different levels and account for that variance and the change X from level A may differ from change X from level B. Whether you analyse change or post value depends on your research question. Very often researchers and statistical reviewers want exactly the change, not the post value.


Representing the snapping of the damper pedal to create an ambient tone in sheet music

Question: The other day I was sitting down at my piano while composing when I encountered something that I had noticed a few times in the past. Here is what happened:

I pressed the down the damper pedal
Immediately afterwards, I realized I didn't want to press that yet so I lifted my foot.
Immediately afterwards, I realized that I actually did want to press the pedal so I pressed it down again.

This quick "snap" of the damper pedal left an ambient - almost percussive - sound echoing out of the piano body. I also noticed that if I kept "pumping" the damper pedal, the sound would increase in volume and would become more "surrounding".
I realized that for my piece that I was writing, that would create a perfect occasional background sound for the actual piano sound to sit on top of.
My main question is this: how would I represent this in sheet music? I realized I could try something like this (I apologize for the formatting):

This looks a little weird though and the performer may not understand it. Perhaps a note should be included in the piece description?
Some side questions for clarification:

Does this actually work on all pianos? I own an upright piano that's about ~100 years old and I don't get that many opportunities to play other types of pianos, so I don't know if this is something that is common across other pianos.
Is this "okay practice" for piano pieces?
Comment: I think a written description should be included. Maybe even define your own symbol for it. It has worked on all pianos I've ever tried it but I haven't tried what it sounds like for the audience. In a bigger hall with not-so-excellent acoustics or a tiny bit of extra background noise it might not even be audible at all.
Comment: I expect someone somewhere has notated this. I'm really surprised this document I just read didn't have it: http://www.mtna.org/media/79410/2014_e-Journal_AOY.pdf I included the link here anyway because there is every other type of unusual piano technique I can think of with notation in that document.
Comment: @nonpop These are good points. Would you mind putting it in an answer? I'd like to give you a vote.
Comment: I agree with @nonpop here.  You can also use the more modern bar notation if you want to be very specific about where to hit the pedal.
Comment: @nonpop I've been able to get a piano very loud indeed with successive "damper bangs" if you really pop the pedal good. I'm sure that "fortissimo" damper banging on a 5'+ grand would be audible in most cases.
Comment: @BobRodes What is modern bar notation? Is that what is shown in the image I put in the post?
Comment: @SirPython No it isn't.  Look near the bottom of [this](http://www.dolmetsch.com/musicalsymbols.htm).  There are several different pedal notations discussed.  The bar notation has a short vertical bar for when you depress the pedal (analogous to Ped.), a horizontal bar for as long as you hold it, and another short vertical bar when you let up the pedal (analogous to *).  If you are to "clear" the pedal by raising it and immediately depressing it, the bar will have an upside-down V.  (I do not agree that this is a half-pedal notation, as it says there.  It means to fully clear the pedal.)
Comment: This effect will be very instrument-specific. And it will certainly require explanation! There's a whole raft of "extended" techniques for piano, including many that make use of the general resonance available when the dampers are lifted. This Wikipedia article scratches the surface.
[Extended technique](https://en.wikipedia.org/wiki/Extended_technique#Piano_2)
Comment: Huh. I thought this was a widely known modern technique, but my web searches seem to support your position that it is esoteric and in need of explanation. It probably does sound a lot different on an upright versus a grand also. Good points.
Answer: There's no standard notation for this. In general, prefer the bracket notation to the old school Ped. * type notation: bracket is much more precise and is also cleaner... it makes more sense in virtually every way.
A few thoughts on this: yes, it's an interesting sound. Be aware that it will differ between pianos and rooms... you'll have a very different sound on a grand compared to an upright and the sound may be surprisingly difficult to produce at all on some instruments. This is something that pianists train pretty hard to avoid (pedaling should be silent), but you as the composer can certainly ask for something else. There may be a mechanical issue here and I would check with a piano technician if that's a tool you intend to use a lot... excessive heavy "snapping" pedaling could perhaps damage some of the mechanism. I don't think that is likely, but it's worth asking someone who knows.
write a brief note at the beginning of piece explaining what you're asking for. Then I would notate with an "inverted V" bracket below the lower staff, but other notations are possible. Just explain clearly what you want and a good performer will make it happen.
Answer: As user5747140 answered, some prose is required to explain whichever notation you choose.
Messing around with the damper pedal won't damage anything.  The dampers individually move extremely quickly.  The pedal mechanism normally goes for years without needing professional adjustment.  You could rig up a robot to "snap" the damper pedal every second for weeks nonstop.
A louder version of this effect is to hold the damper pedal, then hold the sostenuto pedal (middle pedal on a grand; your upright might not have this) to keep the dampers raised, release the damper pedal, and then -- at the right moment -- release the sostenuto pedal and an instant later repress the damper pedal.  This bounces the dampers off the strings more forcefully and more time-aligned than with the damper pedal alone, because you're releasing a trigger to drop them, instead of lowering them no faster than the damper's return spring permits.
But messing around with the sostenuto pedal can cause damage.  If you depress the sostenuto at the same moment as you depress keys (or the damper pedal), then the sostenuto "doesn't know" which halfway-up dampers to grab, and you can feel the grinding in the mechanism through your foot.


Overriding Field Collection output

Question: I have a content type named: lectures
In this content type I have a field collection field named: field_course_video_slide_sync
This field collection consists of 3 fields: field_start_slide_sec, field_slide_title and field_slide_number
What I'm trying to achieve is to rewrite them into this:

field_slide_title

I've tried tpl.php, preprocess_node, field_get_items and views but nothing seems to work. I can't figure out what I'm missing.
Answer: Please override field-collection-item.tpl.php file present in the Field Collection module directory. You can put your code in that. Just simply copy it to your theme folder, and place the code inside it.
Comment: Thanks for the advise. I've created a file in my theme folder **field-collection-item--field-course-video-slide-sync.tpl.php** and wrapped the items in an '' but how can I format the 3 fields in this field collection from that tpl file?
Comment: Ok, So I hope your tpl is get called. Now we have a variable in it as **$content**. So you can check the content in that and print that according to you want.
Comment: Thanks for pointing me in the right direction. If anyone else wanted to know, here is what I did: I created a **field-collection-item--field-course-video-slide-sync.tpl.php** with only **** in it. And did the field theming in **field--field-start-slide-sec--field-course-video-slide-sync.tpl.php**, **field--field-slide-title--field-course-video-slide-sync.tpl.php** and **field--field-slide-number--field-course-video-slide-sync.tpl.php**.

Overriding Field Collection output


See only the commits created on a branch, with merged commits hidden

Question: I'd like to be able to see the commits that have been created on a branch, but without any of the commits that have been merged on to it from other branches. Something like git log, but with activity on other branches either completely hidden, or abbreviated to a single entry noting that a merge was made.
Comment: I get what you want, just don't know if it's possible. A branch is just a pointer to a commit, so short of tracking parentage in the graph, I'm not sure git even keeps track of where a thing was first committed, in the case of cherry-picks and such.
Comment: I didn't word my comment very well. I deleted it, if I can think of a better way to say this I will post another one or else an answer...
Comment: Heh, thanks @DanLowe, I was going to reply. I understand the point that you made, by the way – I appreciate that those commits are now on this branch having been merged there. I will want to see able to ignore them, though. I want to see just the commits that were created on this branch rather than commits that were merged to this branch. I've edited a little for clarity.
Answer: If I am understanding you right, I think you may be looking for
git log --no-merges

However, this only filters out commits with more than one parent. If something was merged to this branch via cherry-pick or fast-forward merge, they will probably still show up unless they had multiple parents for another reason.
Comment: Ah. This helps (thanks), but I think because of the fast-forward thing it's of limited use. I'm certainly still seeing lots of commits that weren't made on this branch.
Comment: mMmm – `--first-parent` as a further additional flag seems to possibly show me what I want to see…


HTTP/1.1 New Session Failed

Question: I'm getting reports from my users that they are getting the following error message when going to my website:

HTTP/1.1 New Session Failed

It doesn't happen all the time. (I checked my site today randomly about 20 times and I only saw it once.)
My site is built with ASP.  It is actually simple HTML, but I use ASP because I have my HTML broken up into 5-6 files and am using #include (the only ASP command I'm using) to dynamically combine these files.  I'm doing this because several areas of my pages are static (such as a "News" section that is the same on every page of my site, and I keep it in a separate file to easily update it once for the entire site).
My site is hosted in a Windows IIS environment.  As I'm using a hosting company, I don't have access to IIS, I configure everything through web.config.
I added the following to web.config:
<code><system.web>
    <sessionState mode="Off" />
</system.web>
</code>
...thinking that this would prevent the use of sessions.  Like I said before, my site is all simple HTML and therefore sessions aren't necessary. HOWEVER, this didn't help and I did see the error after adding this bit.
I'm assuming that I'm getting the error because of memory issues. If I'm only using ASP's #include command, how can I simplify my site so that I no longer get the error?  Is there some web.config section I should be including so that IIS manages memory better?
EDIT: I just asked the server company about adding memory to my instance and they explained that because I'm on a shared server, that's not possible. They did offer me a more expensive plan, but that just seems ridiculous for my simple site that hardly gets any traffic.
My web.config is quite long, so if you need to see a specific part of it to help debug the problem, just let me know and I'll post it.
Now, the techs at the hosting company I'm using are useless.  They're suggesting stuff like, "I don't see an error when bringing up your site. Try using Firefox browser instead of Chrome... see if that helps." Ugh. I'm not sure how changing an end user's browser would fix a server issue... Anyway...
If I can't resolve this issue, I'm thinking of doing the following:

Switch all of my pages to PHP instead of ASP to see if PHP works better. I originally chose ASP because I figured it is ancient technology that would be stable at this point.
Last resort: Switch all of my pages to HTML and use an offline program to generate them when one of the static areas changes. (The news section typically gets updated about once per month so this wouldn't be such a bad solution really--but, seriously, isn't this sort of thing what ASP is for?!)

EDITEDIT: I've switched over the main page on my site to PHP and will keep an eye on the site over the next few days to see if I get any more errors. Let's see if PHP is more stable than ASP. NOTE: I did run up against this error when converting my main page to PHP, so I converted my included files to plain UTF-8 encoding (without BOM, whatever that is...) and it seems to be working.
EDITEDITEDIT: I checked my site two dozen times throughout the day and didn't see the error even once.  I'm going to continue monitoring this for a couple of days, and if all goes well I'll convert the rest of my site from ASP to PHP.
Comment: I recommend trying PHP, it's far more powerful, faster, stable, secure techology than ASP.
Comment: @suspectus, thanks for the comment. The translation of the pages would be trivial, so I'm going to give this a few more days to see if anyone can come up with an ASP solution before doing the conversion.
Answer: I never did find a solution to the ASP problem. So, I converted all of my pages from ASP to PHP and I haven't seen any errors at all. I'm not thrilled, but the problem is solved.
I hope you find this useful.

HTTP/1.1 New Session Failed


interpreting ancova and extracting estimated means

Question: Imagine the following data:
ds <- data.frame(x=1:10,y=1:10,z=rep(c("A","B"),each=5))

the means for groups in z are:
library(plyr)
ddply(ds, "z", function(x) mean(x$y))
#  z V1
#1 A  3
#2 B  8

do a couple of models:
m1 <- glm(y ~ x , data = ds)
m2 <- glm(y ~ z , data = ds)

In m1 the intercept is zero and the estimate for x is 1 as we would expect form the created data. Intercept meaning the value of y for x=0.
In m2 the intercept is 3 and the estimate for B is 5. This is because of the contrasts between the intercept with the estimate i.e. the estimate for B is 8. These results make sense in terms of the means of my data as calculated separately above.
so for m1 I interpret intercept as y where x=0 and slope and for m2 as this is an anova, intercept as mean of first level and the rest are differences between that level and the first.
However in the following models m3 and m4
m3 <- glm(y ~ x + z , data = ds)
m4 <- glm(y ~ x * z , data = ds)

# m3
#Coefficients:
#              Estimate Std. Error    t value Pr(>|t|)
#(Intercept)  1.123e-15  6.201e-16  1.812e+00    0.113
#x            1.000e+00  1.720e-16  5.814e+15   <2e-16 ***
#zB          -7.628e-17  9.880e-16 -7.700e-02    0.941

# m4
#Coefficients:
#              Estimate Std. Error    t value Pr(>|t|)
#(Intercept)  1.123e-15  8.714e-16  1.289e+00    0.245
#x            1.000e+00  2.627e-16  3.806e+15   <2e-16 ***
#zB          -9.349e-17  2.305e-15 -4.100e-02    0.969
#x:zB         3.130e-18  3.716e-16  8.000e-03    0.994

The coefficients for intercept and x are the same as m1 but how do I interpret the estimate for B and A as the estimate for B is -7.628e-17.
My question is how do I interpret the categorical estimates in m3 and from the output extract the estimated means, which I would report? and for m4 is the slope for x:zB 1.000e+00 + 3.130e-18 and for A 1.000e+00 - 3.130e-18?
Answer: Your models have a problem, you have added no error to your y so you get a perfect fit with your x variable. So let's fix this first:
set.seed(123)
ds <- data.frame(x=1:10,y=1:10+rnorm(10, 0, 0.5),z=rep(c("A","B"),each=5))
m1 <- glm(y ~ x , data = ds)
m2 <- glm(y ~ z , data = ds)
m3 <- glm(y ~ x + z , data = ds)
m4 <- glm(y ~ x * z , data = ds)
summary(m1)
summary(m2)
summary(m3)
summary(m4)

Now you get:
#m3
Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   0.3810     0.4124   0.924    0.386
x             0.9053     0.1144   7.914 9.77e-05 ***
zB            0.3547     0.6571   0.540    0.606

And:
#m4
Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.1552     0.4677  -0.332 0.751219
x             1.0840     0.1410   7.687 0.000254 ***
zB            2.3208     1.2374   1.876 0.109830
x:zB         -0.3575     0.1994  -1.793 0.123219

Now why do you get those odd estimates for B and for the interaction? If you look at R-squared in models m1 and m2 you'll notice that R^2 for m1 is 0.98 and for m2 is 0.77. That is, you are already explaining almost all the variability with just one variable so when you add another there's not much to add to the fit of the model. Look how the B coefficient is not significant in m3 but it was in m2.
You can interpret the coefficient of B in m3 as the independent effect of B in y taking into account the effect of x in y. Since y is almost fully explained by x the estimated independent effect of B is much lower than what you'd expect.
Comment: Thank you @Aghila +1 if I had more rep, I think that makes a lot of sense. However, if you could tell If I were to report the 2 means for `z` and  2 slopes for`z` in your `m4` what they would be? So I know I am subrtactingadiing the effect sizes in the correct way.


Finding the number of surfaces in a solid (which is formed by performing some operations on a 2d shape)

Question: Need help understanding how x1 in the image has just 11 surfaces and not 12, i cannot visualize it :(
I cannot understand what's going on in the image, can someone please explain in easy terms (dummy alert!), i would greatly appreciate it.
Or maybe any useful links/resources could also be of great help! :)
see image here
Comment: Let $A$ be the leftmost vertex of $X$ and $\ell$ be the union of the two leftmost edges of $X_1$. It seems you can generate $X_1$ from $X$ by sliding a copy of $X$ along the $z$-direction with $A$ always lies on top of $\ell$. Based on the picture, the edge of $X$ you cannot see is parallel to the direction of horizontal part of sliding, that particular edge will trace out a single concave hexagon as face instead of two faces.
Comment: Welcome to MSE. Please read this text about [how to ask a good question](https://math.meta.stackexchange.com/questions/9959/how-to-ask-a-good-question).
Comment: Well I'm a dummy too. To me it seems you just drag that shape along the trajectory of a letter $V$. As the original shape has 5 edges, you get 10 surfaces vertically + 2 horizontal surfaces. I count 12 as well. I do wonder what the definition of "surface" is in this context, especially if you look at the circle for the second part of the question.
Comment: Welcome to MSE. Please write full details of the question in the body (instead of referring to another source). You can even add images here. Also, the title should be more general than what you wrote. I suggest you take a look at other questions.
Comment: Probably the one that cannot 'see' the sunlight will not be counted. :-)
Comment: All surfaces have to be counted,curved or flat, visible from the front or not... I think that the transformations being performed (especially the second step of the transformation) on x are such that it can only generate 4 sides. Thus, 6(from the upper part) + 5(lower part) = 11 (total, including top and bottom). But i cannot completely understand this.

Could that even be possible?
Comment: If the so-called "3D operations" are not specified, the number of surfaces could be anything. It could well be that the two faces on the back lie on the same plane, and thus count as a single surface.
Answer: It seems the unknown 3D operation is the "extrude" operation appears in various
3D CAD/modelling softwares (AutoCAD,SketchUp, ...)
or 3D graphics libraries (Three.js, X3DOM).
Different softwares/libaries may use different name but the basic idea is the same. It is a way to create
a solid from a cross-section (a geometric shape lies on a plane).
Geometrically, one starts with a 2D planar shape $S$ and a continuous curve
$\gamma : [0,1] \to \mathbb{R}^3$ with starting point $\gamma(0)$ belongs to $S$. One then parallel translate
$S$ so that the image of $\gamma(0)$ always lies on top of $\gamma(t)$. The resulting solid is the union of the image of $S$
at different $t$. More precisely, it is the 3D solid:
$$V = \bigg\{ p + \gamma(t) - \gamma(0) : p \in S, t \in [0,1] \bigg\}$$
If $\gamma(0)$ is the origin, the solid $V$ is known among geometers as the Minkowski sum
of the point set $\gamma([0,1])$ and $S$.
For practical application, one typcially impose some extra constraint to limit what $\gamma$ could be.
One choice is require $\gamma$ to be at least piecewise $C^1$, i.e continuous differentiable up to $1^{st}$ order. Aside from finitely many $t$, the tangent vector $\gamma'(t)$ is defined, non-zero and never parallel to the plane
holding $S$.
Consider the special case where the cross-section is a planar $n$-gon and $\gamma$ is a polyline consists of $m$ line segments.
When one increases $t$ from $0$ to $1$, each edge of $S$ will trace out $m$ quadrilaterals, one for each segment of $\gamma$.
In the absense of any relations among directions of segments on $\gamma$ and edges on $S$, solid $V$ will have $mn + 2$ faces.
The actual number of faces of $V$ can be less than this.
At those $t_0$ where $\gamma'(t_0)$ is discontinuous.
If the two limiting tangent vector $\gamma'(t_0\pm) \stackrel{def}{=}\lim\limits_{\epsilon \to 0\pm}\gamma'(t_0+\epsilon)$
is coplanar with the tangent vector of an edge $e$ of $S$. The two quadrilaterals traced out by $e$ at $t \ge t_0$ and $t \le t_0$
will share the same normal vector. This mean they will merge into a single face.
Let $N_\parallel$ be the number of this sort of events. The number of faces of $V$ is $mn + 2 - N_\parallel$.
Let's take the solid $X_1$ as an example. One can generate $X_1$ by taking the pentagon $X$ as $S$ and the union of two leftmost edges of $X_1$ as $\gamma$. Notice at the back of $X$, the tangent vector of the edge there
seems to be coplanar with the two tangent vectors on $\gamma$. This means $N_\parallel = 1$ the number of faces of $X_1$ becomes $2(2)(5) + 2 - 1 = 11$.
One can generalize this to other planar shape. In particular, those $S$ whose boundary consists of $n_c$ circular arcs and $n_\ell$ line segments. If $N_\parallel$
is the number of instances where the two limiting tangent vectors $\gamma'(t_0\pm)$ is coplanar with the tangent vector of a straight edge of $S$,
the number of faces of $V$ with be $m(n_c + n_\ell) + 2 - N_\parallel$.
Let's take the $Y$ as another example of $S$. Its boundary $\partial Y$ consists of $n_c = 2$ circular arcs and $n_\ell = 10$ line segments. $4$ of the line segment seems to be parallel to the reference direction indicated by the orange dashed line. Since $m = 2$ and there is only one discontinuity in $\gamma'(t)$,  $N_\parallel = 4$. The number of faces of corresponding 3D solid $Y_1$ should be $2(2 + 10) + 2 - 4 = 22$.
Notes
I first learned this operation in SketchUp but never know what it is. I finally know its name when looking up documents on x3dom.js many years later. Neither of them explain the concept well. A web search returns this article on the extrude command in AutoCAD. It is quite readable and explain the operation clearly. Take a look if you are interested.


GlassFish not launching with NetBeans

Question: I have NetBeans 7.0.1 and GlassFish 3.1. After installing (or reinstalling) GlassFish it deploys correctly and launches my java ee project.
If i close NetBeans and launch it again - GlassFish doesn't deploy.
Java DB says:
2011-09-08 20:33:28.876 GMT : DRDA_SecurityInstalled.I
2011-09-08 20:33:29.246 GMT : DRDA_ListenPort.S, <1527>, <localhost>, <java.net.BindException: Address already in use: JVM_Bind>

The 1527 port is used by java.exe.
But it keeps saying Launching GlassFish Server for 20 minutes and throws a timeout error
This is the error log for GlassFish
08.09.2011 17:40:14 null
INFO: Running GlassFish Version: GlassFish Server Open Source Edition 3.1.1 (build 12)
WARNING: Record begin marker is not a proper value so using default.
WARNING: Record end marker is not a proper value so using default.
WARNING: Log Format field separator is not a character so using default.
INFO: Registered org.glassfish.ha.store.adapter.cache.ShoalBackingStoreProxy for persistence-type = replicated in BackingStoreFactoryRegistry
INFO: Grizzly Framework 1.9.36 started in: 124ms - bound to [0.0.0.0:44235]
INFO: Grizzly Framework 1.9.36 started in: 202ms - bound to [0.0.0.0:44240]
INFO: Grizzly Framework 1.9.36 started in: 296ms - bound to [0.0.0.0:44239]
INFO: Grizzly Framework 1.9.36 started in: 156ms - bound to [0.0.0.0:44196]
INFO: Grizzly Framework 1.9.36 started in: 187ms - bound to [0.0.0.0:44207]
INFO: GlassFish Server Open Source Edition 3.1.1 (12) startup time : Felix (20 290ms), startup services(1 567ms), total(21 857ms)
INFO: JMXStartupService: Started JMXConnector, JMXService URL = service:jmx:rmi://G62-120ey:44245/jndi/rmi://G62-120ey:44245/jmxrmi
INFO: Hibernate Validator 4.1.0.Final
INFO: Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.
INFO: Grizzly Framework 1.9.36 started in: 5ms - bound to [0.0.0.0:44239]
INFO: unable to read instance state file I:\glassfish\glassfish\domains\localhost\config\.instancestate, recreating
INFO: Grizzly Framework 1.9.36 started in: 0ms - bound to [0.0.0.0:44240]
INFO: SEC1002: Security Manager is OFF.
INFO: SEC1010: Entering Security Startup Service
INFO: SEC1143: Loading policy provider com.sun.enterprise.security.provider.PolicyWrapper.
INFO: SEC1115: Realm [admin-realm] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.
INFO: SEC1115: Realm [file] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.
INFO: SEC1115: Realm [certificate] of classtype [com.sun.enterprise.security.auth.realm.certificate.CertificateRealm] successfully created.
INFO: SEC1011: Security Service(s) Started Successfully
INFO: WEB0169: Created HTTP listener [http-listener-1] on host/port [0.0.0.0:44239]
INFO: WEB0169: Created HTTP listener [http-listener-2] on host/port [0.0.0.0:44240]
INFO: WEB0169: Created HTTP listener [admin-listener] on host/port [0.0.0.0:44207]
INFO: WEB0171: Created virtual server [server]
INFO: WEB0171: Created virtual server [__asadmin]
INFO: WEB0172: Virtual server [server] loaded default web module []
INFO: WEB0671: Loading application [Course_Project] at [/Course_Project]
INFO: Course_Project was successfully deployed in 4 068 milliseconds.
SEVERE: Error compiling file: I:\glassfish\glassfish\domains\localhost\generated\jsp\Course_Project\org\apache\jsp\index_jsp.java
WARNING: StandardWrapperValve[jsp]: PWC1406: Servlet.service() for servlet jsp threw exception
org.apache.jasper.JasperException: PWC6033: Error in Javac compilation for JSP
PWC6197: An error occurred at line: 27 in the jsp file: /index.jsp
PWC6199: Generated servlet error:
string:///index_jsp.java:77: getRealPath(java.lang.String) in javax.servlet.ServletRequest cannot be applied to ()
    at org.apache.jasper.compiler.DefaultErrorHandler.javacError(DefaultErrorHandler.java:129)
    at org.apache.jasper.compiler.ErrorDispatcher.javacError(ErrorDispatcher.java:299)
    at org.apache.jasper.compiler.Compiler.generateClass(Compiler.java:392)
    at org.apache.jasper.compiler.Compiler.compile(Compiler.java:453)
    at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:625)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1539)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
    at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:330)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:174)
    at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:828)
    at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:725)
    at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
    at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
    at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
    at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
    at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
    at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
    at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
    at java.lang.Thread.run(Thread.java:662)

SEVERE: Error compiling file: I:\glassfish\glassfish\domains\localhost\generated\jsp\Course_Project\org\apache\jsp\index_jsp.java
WARNING: StandardWrapperValve[jsp]: PWC1406: Servlet.service() for servlet jsp threw exception
org.apache.jasper.JasperException: PWC6033: Error in Javac compilation for JSP
PWC6197: An error occurred at line: 27 in the jsp file: /index.jsp
PWC6199: Generated servlet error:
string:///index_jsp.java:77: getRealPath(java.lang.String) in javax.servlet.ServletRequest cannot be applied to ()
    at org.apache.jasper.compiler.DefaultErrorHandler.javacError(DefaultErrorHandler.java:129)
    at org.apache.jasper.compiler.ErrorDispatcher.javacError(ErrorDispatcher.java:299)
    at org.apache.jasper.compiler.Compiler.generateClass(Compiler.java:392)
    at org.apache.jasper.compiler.Compiler.compile(Compiler.java:453)
    at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:625)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1539)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
    at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:330)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:174)
    at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:828)
    at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:725)
    at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
    at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
    at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
    at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
    at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
    at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
    at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
    at java.lang.Thread.run(Thread.java:662)

SEVERE: Error compiling file: I:\glassfish\glassfish\domains\localhost\generated\jsp\Course_Project\org\apache\jsp\index_jsp.java
WARNING: StandardWrapperValve[jsp]: PWC1406: Servlet.service() for servlet jsp threw exception
org.apache.jasper.JasperException: PWC6033: Error in Javac compilation for JSP
PWC6197: An error occurred at line: 27 in the jsp file: /index.jsp
PWC6199: Generated servlet error:
string:///index_jsp.java:77: getRealPath(java.lang.String) in javax.servlet.ServletRequest cannot be applied to ()
    at org.apache.jasper.compiler.DefaultErrorHandler.javacError(DefaultErrorHandler.java:129)
    at org.apache.jasper.compiler.ErrorDispatcher.javacError(ErrorDispatcher.java:299)
    at org.apache.jasper.compiler.Compiler.generateClass(Compiler.java:392)
    at org.apache.jasper.compiler.Compiler.compile(Compiler.java:453)
    at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:625)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1539)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
    at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:330)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:174)
    at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:828)
    at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:725)
    at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
    at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
    at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
    at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
    at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
    at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
    at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
    at java.lang.Thread.run(Thread.java:662)

WARNING: StandardWrapperValve[jsp]: PWC1406: Servlet.service() for servlet jsp threw exception
java.lang.NullPointerException
    at org.apache.jasper.runtime.JspRuntimeLibrary.internalIntrospecthelper(JspRuntimeLibrary.java:368)
    at org.apache.jasper.runtime.JspRuntimeLibrary.introspecthelper(JspRuntimeLibrary.java:354)
    at org.apache.jasper.runtime.JspRuntimeLibrary.introspect(JspRuntimeLibrary.java:332)
    at org.apache.jsp.index_jsp._jspService(index_jsp.java from :56)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:403)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1539)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
    at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:330)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:174)
    at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:828)
    at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:725)
    at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
    at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
    at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
    at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
    at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
    at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
    at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
    at java.lang.Thread.run(Thread.java:662)

INFO: Server shutdown initiated
INFO: JMXStartupService: Stopped JMXConnectorServer: null
INFO: JMXStartupService and JMXConnectors have been shut down.
INFO: Shutdown procedure finished

What could be the problem? Why is GlassFish not launching for the second time?
Comment: i found the root of the problem (some file in domains, i think. don't remember actualy). So i gave up on glassfish, using tomcat now :c
Comment: thanks for the info, glassfish runs fine but i think its just netbean is not really closing 1527 port. Thats what i think. I tried modify domain.xml but didn't really work. Tomcat didn't give me any of such errors back in time i was using it...
Answer: According to this JIRA issue, this has been fixed in glassfish version 3.1.2.  Please try updating your glassfish and check if this happens again.

GlassFish not launching with NetBeans


Deploy Rails 5 API with Capistrano and Puma

Question: I'm trying to deploy a Rails 5 API with Capistrano to an AWS server and I'm running into some problems...
I've followed this tutorial and so I'm trying to deploy with the following command:
$ cap production deploy

Which produces a log ending in:
00:20 deploy:log_revision
      01 echo "Branch master (at <uuid>) deployed as release 20180315215050 by <user>" >> /<path_to_app>/revisions.log
    ✔ 01 <user>@<Public DNS (IPv4)> 0.206s

It all seems ok, I guess. But when I query the server, I receive a 502 Bad Gateway from nginx.
And when I $ ps aux | grep puma or $ ps aux | grep rails I don't find any process running...
My config files are as follows:
Capfile
# Load DSL and set up stages
require 'capistrano/setup'

# Include default deployment tasks
require 'capistrano/deploy'

# Load the SCM plugin appropriate to your project:
require 'capistrano/scm/git'
install_plugin Capistrano::SCM::Git

require 'capistrano/rvm'
require 'capistrano/bundler'
require 'capistrano/rails/assets'
require 'capistrano/rails/migrations'
require 'capistrano/puma'

# Load custom tasks from `lib/capistrano/tasks` if you have any defined
Dir.glob('lib/capistrano/tasks/*.rake').each { |r| import r }

config/deploy.rb
# config valid for current version and patch releases of Capistrano
lock '~> 3.10.1'

set :application, '<app_name>'
set :repo_url, '<git_repo>'
set :branch, 'master'
set :deploy_to, '/<path_to_app>'
set :pty, true
set :ssh_options, {
    forward_agent: true,
    auth_methods: ['publickey'],
    keys: ['/<path_to_key>.pem']
}
set :linked_files, %w{config/database.yml config/application.yml}
set :linked_dirs, %w{bin log tmp/pids tmp/cache tmp/sockets vendor/bundle public/system public/uploads}
set :keep_releases, 5
set :rvm_type, :user
set :rvm_ruby_version, '2.5.0'

set :puma_rackup, -> { File.join(current_path, 'config.ru') }
set :puma_state, "#{shared_path}/tmp/pids/puma.state"
set :puma_pid, "#{shared_path}/tmp/pids/puma.pid"
set :puma_bind, "unix://#{shared_path}/tmp/sockets/puma.sock"
set :puma_conf, "#{shared_path}/puma.rb"
set :puma_access_log, "#{shared_path}/log/puma_error.log"
set :puma_error_log, "#{shared_path}/log/puma_access.log"
set :puma_role, :app
set :puma_env, fetch(:rack_env, fetch(:rails_env, 'production'))
set :puma_threads, [1, 8]
set :puma_workers, 0
set :puma_worker_timeout, nil
set :puma_init_active_record, true
set :puma_preload_app, false

@server: /etc/nginx/sites-available/default
upstream app {
  # Path to Puma SOCK file, as defined previously
  server unix:/<path_to_app>/shared/tmp/sockets/puma.sock fail_timeout=0;
}

server {
  listen 80;
  server_name localhost;

  root /<path_to_app>/public;

  try_files $uri/index.html $uri @app;

  location / {
    proxy_set_header X-Forwared-Proto $scheme;
    proxy_set_header X-Forwared-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $host;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Connection '';
    proxy_pass http://app;
  }

  location ~ ^/(assets|fonts|system)/|favicon.ico|robots.txt {
    gzip_static on;
    expires max;
    add_header Cache-control public;
  }

  error_page 500 502 503 504 /500.html;
  client_max_body_size 4G;
  keepalive_timeout 10;
}

Can anyone figure out what I have done wrong please?
Thank you very very much!
Answer: You should set parameters with <...> manually.
For config/deploy.rb
set :application, '<app_name>'
set :repo_url, '<git_repo>'
set :deploy_to, '/<path_to_app>'

and
set :ssh_options, {
    forward_agent: true,
    auth_methods: ['publickey'],
    keys: ['/<path_to_key>.pem']
}

For @server: /etc/nginx/sites-available/default
server unix:/<path_to_app>/shared/tmp/sockets/puma.sock fail_timeout=0;

and
root /<path_to_app>/public;

Comment: I was the one to write those `<...>` to anonymize my code.
I don't actually have them in my code.


Webapplication server attack

Question: I checked the log of my  web application (Tomcat )  and i am getting some strange requests. It looks like someone is trying some random pages to access on my application .
My question is how to stop such activities as i am having my log full of these attempts.
188.8.131.52 - - [08/Nov/2015:00:24:45 +0000] "POST /register.do HTTP/1.1" 404 973
22.214.171.124 - - [08/Nov/2015:00:27:56 +0000] "GET /testproxy.php HTTP/1.1" 404 404
126.96.36.199 - - [08/Nov/2015:01:03:48 +0000] "POST /Main.action HTTP/1.1" 404 973
22.214.171.124 - - [08/Nov/2015:02:26:41 +0000] "GET /web-console/ServerInfo.jsp HTTP/1.1" 404 1003
126.96.36.199 - - [08/Nov/2015:02:29:48 +0000] "GET /jmx-console/HtmlAdaptor?action=inspectMBean&name=jboss.system:type=ServerInfo HTTP/1.1" 404 997
18.104.22.168 - - [08/Nov/2015:02:32:44 +0000] "GET /invoker/JMXInvokerServlet HTTP/1.1" 404 1001
220.127.116.11 - - [08/Nov/2015:10:07:51 +0000] "GET /testproxy.php HTTP/1.1" 404 404
188.8.131.52 - - [08/Nov/2015:14:40:30 +0000] "GET /httptest.php HTTP/1.1" 404 402
Comment: Is it causing a problem or just annoying you?  If it's not causing a problem don't worry about it.
Comment: add a rule to refuse trafic without user agent
Comment: The whole story is that . First i got login attempts on the DB port 1433   , i fixed it. I got login attempts on RDP and i fixed it  . Now i am getting on http an i can not restrict the traffic . If it is a bot which loops each 1 min on http trying random pages my app server will have to respond by 404 and that will kill the bandwidth  in my opinion .
Comment: sorry i don't have tomcat so i can't describe but tomcat doc can, just search on google `block empty user agent trafic tomcat`
Answer: Serving a 404 is a log entry and a neglible amount of bandwidth/resources on your web server.
You can start banning IPs but it will not stop.  If you do, you may want to configure a short expiry for it, as you may end up blocking legitimate users when those IP addresses get reassigned.
The source IPs will constantly change as there are thousands of bots/trojans which will continue to scan your web server to find a way in.
Comment: is that a real solution ? redirecting trafic before 404 wouldn't save hdd access to see if page exist ?

Webapplication server attack


Latent growth curve model: should you include predictors if the variance for slope and/or intercept factor is not significant?

Question: Hi I am running a series of latent growth curve models.
I am wondering: can predictors of the slope and intercept factor(s) be included when the slope and/or intercept variance are not significant, or should this be avoided?
Further, I am wondering whether the answer to this is impacted by 1) missingness in the data (and whether it is related to predictors/covariates), and 2) if the slope variance has been fixed to a specific number (i.e., it is modelled as a fixed effect); specifically in one model I have fixed the slope variance to be greater than 0 (i.e., 0.00001) after receiving a lavaan warning that the variance was negative.
Answer: In principle, you can add predictors even when the variances are small. However, you may not find any effects. In your case, it looks like the slope variance may be very close or even equal to zero in the population. If individuals don't differ in their slopes, then obviously, there are no differences to explain. In other words, predictors cannot account for non-existent inter-individual differences.


i have installed jdk9 on my linux machine but jre file is missing, did i downloaded wrong tar file?

Question: I have installed jdk9 on my linux machine but jre file is missing, did i downloaded wrong tar file?
this is the tar file i downloaded jdk-9-ea+174_linux-x64_bin.tar.gz
Comment: In jdk, jre is in $JAVA_HOME/jre
Comment: jre file is missing in extracted jdk9 file thn how can i add jre path?
Comment: Where did you look for it?
Comment: inside extracted file is usually it exist there only
Comment: Why didn't you use your package manager? It's unusual to download on Linux manually. Only few programs should be downloaded manually (like Eclipse)
Comment: how it is useful?how to install it?
Answer: There isn't any problem with the tar file you have downloaded. There is a change in the directory layout with JDK-9. Until JDK 8, if you downloaded JDK, it contained the same jre/ directory (which is separately available for download) plus some more tools and libraries. This resulted in duplicate binary files in bin directories of the JDK and the jre within JDK .
With JDK 9, there are no more jdk/ versus jre/ wrapper directories, and binaries (such as the java tool) are not duplicated. The JDK and JRE are two separate types of modular runtime images and both have the same directory structure containing the bin, conf and lib directories.Just that,JDK contains the compilation tool like javac as earlier.
Reference : JDK 9 Migration doc
Comment: @ZahidShah the documentation doesn't take Java 9 into account as it's not released. It wouldn't surprise me if at the moment no production version of Tomcat is Java 9 compatible. MAYBE the Tomcat 9 milestones will work, you'd have to try
Comment: so shall i still set the jre path in bash?
Comment: What is your purpose? If it is development, then you can just set the path to JDK bin directory. If you need to just run applications , you can download the JRE and set the path accordingly.
Comment: actually i want to run tomcat but its nt running, may be due to the absence of jre file.
Comment: Is the version of Tomcat you are trying to run compatible with JDK 9 ? The documentation of Tomcat tells the versions it can run on.
Comment: @Pallavisonalyeah it says that tomcat9 requires java8 or later
Comment: As @jwenting said in his comment, even though documentation says JDK8 or later, it may not have taken JDK9 into account and may not be compatible with JDK 9.
Comment: able to install the tomcat now but could not change the tomcat port to 80, previously i used sudo with ./startup.sh to change the port 8080 to 80 it woorked but now whn i did run sudo ./startup.sh on tomcat9 it doesnt work.


exposed filter default values from user fields

Question: How can i set the default value of exposed filter for a view with the current logged in user profile checkboxes?
There can be multiple choices.
Thank you
Comment: oh i get it. I implement that checkboxes based on a field term reference from a vocabulary.
Comment: Hello and welcome. Please provide more details about your current implementation. There are at least 3 ways to get some checkboxes on user profile page, for example.
Comment: Hi, you missunderstood. I have that checkboxes but i want to get the value from there(from checkboxes) and make this value the default for an exposed filter. Ex. If current logged in user has checked the management well the exposed filter default should be the managment articles in articles view. DO you get now?
Comment: No I didn't. But the way you implemented checkboxes determines where their data is stored. And this determines how to access data to use in another part of CMS, like filters.
Answer: Try like this.
<code>function custommodule_form_views_exposed_form_alter(&$form, &$form_state){
  if($form['#theme'][0] == 'views_exposed_form__article__page_1'){
      $form['ct']['#options']['All'] = 'All Article';
  }
}
</code>
Comment: thanks but i dont think is what i need. Please see above the more detailed case.
Comment: here I just trying to show you this hook only. Use this hook in your module and do the functionality what ever you want (Set default values for filters according to the user).
Answer: <code> function mymod_form_views_exposed_form_alter(&$form, &$form_state, $form_id) {
 if (! _mymod_isFormSubmit()) {
 //exposed filters form hasn't submitted so I am using values from user entity
 global $user;
 $wrapper = entity_metadata_wrapper('user', $user);
 foreach(_mymod_getFiltersMapping() as $filterName => $fieldName ) {
  $tids = array();
  foreach($wrapper->{$fieldName}->value() as $term) {
    $tids[] = $term->tid;
 }
 $form_state['input'][$filterName] = empty($tids) ? 'All' : reset($tids);
}
</code>
}
}
i have this code but did not works..


jQuery $.ajax execution slow

Question: I'm having difficulty on some websites I work on with slow executing jQuery $.ajax.
I am not talking about the server taking a long time to respond. I am talking about the actual execution of the $.ajax function taking a long time.
Normal (fast) example
Going to jQuery.org and running the following code takes 3ms:
var start = (new Date()).getTime();
$.get("https://jquery.org/projects/")
var end = (new Date()).getTime();
console.log("executed in " + (end - start) + "ms")

Bad (slow) example
However, running very similar code on a demo website I'm working on takes 324ms:
var start = (new Date()).getTime();
$.get("http://sauce-demo.myshopify.com/collections/frontpage/products/bronze-sandals")
var end = (new Date()).getTime();
console.log("executed in " + (end - start) + "ms")

Now obviously the response from $.ajax (and therefore $.get) is going to be async. But what could be causing such a huge increase in execution time?
Thanks
Comment: Are you sure you aren't setting the secong one as sync, using maybe ajax global setup???
Comment: Definitely looks like it's not async. Add an alert after the get and you'll see there's a delay. Try the scripts in [here]( http://jsfiddle.net/2L7g7be3/2/) and you'll see there's a difference. The network tab also has a timing tab where you can see that they all take mostly 300ms but ones is async and one is not
Comment: @A.Wolff make an excellent suggestion, indeed I could say that you must take a bps (size/time), not ms difference because you are not getting the same data.
Comment: Takes only 2ms here: http://jsfiddle.net/2L7g7be3/
Comment: I tried in jsfiddle and both of those ran in 1ms for me.
Answer: Thanks for all the awesome comments guys. You're absolutely right - it looks like there was a global async: false set up nested in one of the JavaScript files:
jQuery.ajaxSetup({
  async: false
});

I wasn't aware jQuery had a global async setup.
Comment: yup - this did it for me... thank you!

jQuery $.ajax execution slow


Memory over-release problem when I am animating UIView

Question: I have enabled NSZombie's and I am getting the following message in my console when I am running my application:
<code> *** -[UIViewAnimationState release]: message sent to deallocated instance 0xf96d7e0
</code>
Here is the method that is performing the animation
<code>-(void)loadAvatar:(STObject*)st
{
    NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];

    avatar.alpha = 0;
    avatar.frame = avatarRectSmall;

    avatar.image = [ImageCache getMemoryCachedImageAtUrl:st.avatar_url];

    [UIView beginAnimations:nil context:nil];
    [UIView setAnimationDuration:.50];

    avatar.frame = avatarRectNormal;
    [avatar setAlpha:1];
    [UIView commitAnimations];

    [pool release];
    pool = nil;
}
</code>
I don't always get a crash, only sometimes. I'm wondering what is getting released?
Comment: The backtrace only lets me go back to main. Anything after that is all assembly.
Comment: Try setting some breakpoints.
Answer: You have an autorelease pool there which prompts me to ask, is this a separate thread? If the answer is yes then you can't do stuff to UIView there. UIKit is not thread safe. You can do other things like calculating positions or updating images which you later put on the screen but any user interface stuff has to happen in the main thread.
Graphics and Drawing section of iPhone Application Programming Guide
Comment: You are correct, this is a separate thread. I just want to load an image from a URL in my UIImageView. If I remove the animation code, it will only load the images sometimes.
Comment: Do you specifically need a separate thread for loading an image from a URL? Why not just use an async NSURLConnection or some other class with a delegate pattern?
Answer: You can use very simple security check fof all functions doing something with UI:
<code>-(void)functionModifyingUIelements:(id)object
{
 // fire itself in main thread if it is not in it already
 if (![[NSThread currentThread] isMainThread]) {

        [self performSelectorOnMainThread:@selector(functionModifyingUIelements:) withObject:object waitUntilDone:NO];
        return;
    }

}
</code>


Django Rest Framework. What is queryset and serializer_class?

Question: Let's say I have the following viewset:
<code>class CategoryViewSet(viewsets.ModelViewSet):
    queryset = Category.objects.all()
    serializer_class = CategorySerializer
</code>
I am wondering what does the queryset and serializer_class do here? I can assume queryset is about how the models' order will be displayed (such as ordering, filtering and so on), but I may be wrong. Please correct me, and, if possible give broad explanation.
Answer: queryset this to select data from database, you can select all data or apply filter or what you want, in more basic way.. where to write your database query using Django ORM
serializer where the queryset will serialize like convert it to json object to use it in response for example, because some ORM query will return QuerySet not a json object
this a basic explained for both, you can check for more details in Django RestFramework Serializer


How to create a sleep timer app for android

Question: I want to create an app that can shutdown the phone after a time set by the user , I dont know really how to start .
Any help would be much appreciated .Thanks.
Answer: For obvious reasons, you cannot power off a device programmatically.
Answer: So far It's only possible with rooted device or app should be signed with the system firmware signing key.
<code>try {
    Process proc = Runtime.getRuntime()
                    .exec(new String[]{ "su", "-c", "reboot -p" });
    proc.waitFor();
} catch (Exception ex) {
    ex.printStackTrace();
}
</code>
refs:

How to shutdown an android mobile programatically?
Programmatically switching off Android phone


Issue while building eclipse che project using "mvn clean install" command

Question: I am using online IDE of eclipse che.
Below is the issue that i am facing while trying to build the project:-
<code>[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.eclipse.che:che-bookmark-plugin-parent:0.1.0-SNAPSHOT:
Could not transfer artifact org.eclipse.che.depmgt:maven-depmgt-pom:pom:5.0.0-M5-SNAPSHOT from/to codenvy-public-repo (https://maven.codenvycorp.com/content/groups/public/):
Transfer failed for https://maven.codenvycorp.com/content/groups/public/org/eclipse/che/depmgt/maven-depmgt-pom/5.0.0-M5-SNAPSHOT/maven-depmgt-pom-5.0.0-M5-SNAPSHOT.pom and 'parent.relativePath' points at wrong local POM @ line 16, column 13

</code>
You can get the Code used at :- https://github.com/vzhukovskii/che-bookmark-plugin
Comment: Looks like a network issue: `Transfer failed for...`
Answer: That project is very old and it seems that example has been made for Che 6 (or even Che 5).
The error you are facing is probably because it's trying to download the (again very old) maven artifact from maven.codenvycorp.com, which no longer exists.
If you write down what you are trying to achieve (I believe building some obsolete plugin for obsolete version of Che is not your goal) we might be able to help you ;-)
If you want some "bookmark" plugin in some recent version of Che (7.x), I suggest you to take a look for some vscode extension which would solve your needs, as Che 7.x (Theia in particular, which is the default editor for Che) should be able to use that vscode extension.
Comment: Thank you. What I need is to create a custom view in eclipse Che. Can anyone please suggest some examples or useful links?


Nearest Neighbor Matching in R using matchit

Question: I am using the matchit package to do propensity score matching on a data set.  However, when doing nearest neighbor matching, if I use the caliper option, I get a different set of matched pairs every time - i.e. Treatment #18 matches to Control #2276 the first time, but if I rerun the code, Treatment #18 matches to Control #2079 (and so on).  If I remove the caliper option, I get the same match results every time, but the additional matches that are produced with the removal of the caliper produce matches that are a little far apart for my liking.
For example, if I run the following code, notice the differences in the control means:
match.out <- matchit(Category ~ FactorA + FactorB, Data,
                     method = 'nearest', distance = 'logit', caliper = .10)
round(summary(match.out)$sum.matched, digits = 3)

           Means Treated   Means Control   SD Control    Mean Diff
distance       0.506           0.496         0.151        0.010
FactorA        24.243          24.450        3.344       -0.207
FactorB        3.542           3.551         0.392       -0.008

match.out <- matchit(Category ~ FactorA + FactorB, Data,
                     method = 'nearest', distance = 'logit', caliper = .10)
round(summary(match.out)$sum.matched, digits = 3)

           Means Treated   Means Control   SD Control    Mean Diff
distance       0.506           0.496         0.151        0.010
FactorA        24.243          24.427        3.351       -0.184
FactorB        3.542           3.541         0.392       -0.002

This is a problem for me, as I prefer to be able to exactly reproduce my results if the need would ever arise.  Yet I can run matchit without the caliper argument:
match.out <- matchit(Category ~ FactorA + FactorB, Data,
                     method = 'nearest', distance = 'logit')

and get the exact same Treatment-Control matches all day long.  (I actually checked the matrix of matches to verify this - it's not just the same control mean by chance).
Is there a way to still do the nearest neighbor matching that I was doing in the first code chunk with the caliper to narrow my matches a little bit, but still get the same results if I re-run the code?
Thanks for any help (not just on this question, but all - while this is the first question I've felt the need to post here, I've found many answers here)
Answer: I am not an expert on either R nor propensity matching, but I ran into the same problem while working on a project. I think what matchit does is randomly pick one of the control subjects that falls within the caliper interval around the treated subject. If you set your seed to the same number every time you run your match.out line, you will get the same result:
set.seed(100)
match.out <- matchit(Category ~ FactorA + FactorB, Data,
                     method = 'nearest', distance = 'logit', caliper = .10)

Try running these two lines together.
Comment: FYI, this is no longer the case, and `matchit()` always selects the nearest available unit within the caliper.
Answer: I came across the same problem. Setting the seed to a fixed number with the set.seed() function, however, by altering the number given in this function, the outcomes will change. It is true that matchit() will randomly select control subjects when they fall into the caliper.
By making use of the argument mahvars you can define on basis of which variables a subject from the pool of control subjects within a caliper should be picked.
From the MatchIt manual (p. 19):

mahvars: variables on which to perform Mahalanobis-metric matching
  within each caliper (default $=$ NULL). Variables should be entered as a
  vector of variable names. (e.g., mahvars = c("X1", "X2")). If mahvars
  is specified without caliper, the caliper is set to 0.25.


Emails go to spam ( Reverse DNS is set , DKIM pass , SPF pass , DMARC pass )

Question: It's been now 3 days that I am struggling with configuring my emails to go to the inbox.
I have a VPS server at OVH and a domain name. I have configured everything in virtualmin with OpenDKIM and Postfix and used various check tools and everything seems okay (as shown in this screenshot from the Gmail email) but the emails still go to spam ( Yahoo, Hotmail , Gmail ).

Here are the details of a test message sent to Yahoo. As shown, the email is signed and everything seems okay.
X-Apparently-To: firstname.lastname@example.com; Sat, 24 Nov 2018 16:33:22 +0000
Return-Path: <firstname.lastname@example.com>
X-YahooFilteredBulk: 126.96.36.199
Received-SPF: pass (domain of onlineportfolio.pro designates 18.104.22.168 as permitted sender)
X-YMailISG: UMSYaSMWLDtCAbV4UOzfJPY8yYEmqasl9uS0CNCWQntZxG6M
 Fw6igajRZYFVNN.FU7eUuLVuUbBmv8lFUM0RbkU0s.73TLyFu2IZfAN0ZEov
 yQ.tdV13b.3qc_on.gyqnfa4_19bGlEUb_UUTMULgkfYb0yEoSUqdzy_8CLw
 GwFgHO9neZeCKQ8AvDSXa3g8v_Qa40x7QvK8EAzLLiUJ_Kn8rQotnMwmp7hN
 Zm858LGbPSDCGjYTa361wcT7agiIWwtzynbYu97ZseIEKW2RJbKWbVi59JI1
 C0fl1i_XqyTam52bsd8ql2B_84h9jEGr644BIpltsOA6E4IxP1NAXxdkGGoR
 Y1OxWS94_pFHHR0rZDZXjF2YYJoCApDNqlHrP7ZKxlFYFGyzNuFVjFixd5Ki
 kNZytf0GiOaCGrKDkJBzR3HT1bUK9pwiM2jPAXv0h.PNBWWTOc13p_jSqhEx
 46zsxGpf2l.oIL24OxrXtDwAwth1dAvgHLz_mt_hHXzLuNm1pAJbc4bwS.UL
 wnFkJn9Ldc4tZ7j5YgxznNxSDvvWm7xjPkWNNL.OhfI1N0_GrRxkM0ADPkl2
 YgK6THvVic4s3eLw3E91M8tqpb0zQh3M23yAvi4eaI.nVjweS.Q1LZvybbwI
 W3sNsRGY_Ydhoxpeu0Nzro8jF4OPy6EFYeGcH2T7tDsCCgc0N0ncrRVPkQ27
 9rGqFtRh5yNluX0K9wIGQU2vn63c41D5hLs5WFAlvBficYVE2py0FL5L94.n
 WrFrMqHDgAuEW2Iy6_qPNAWUJDpIJykrsGI2ph9g02KhLzTL4B4S59TH4X8W
 xVgYCu13zRMB7kfcyM0QXZsV1uxSIBi8Ewku3zWuu3lwqfNFDsHU6w6k7P1h
 fVgg_nRNjSXKNT24qqSQUh2jJWFQlanGwKLkhDCpiBxEoQgr84OUo13hdjap
 YnY7ncw1iOBlKudLxJc029WYtH.ukMbMEsa03soD7ihS97rb3tLz6GxrDqo3
 Sq.GP.krEjUdJAAB0VYG63LAmeFAoy6yrDRB60A9Zdlu_IORPautWynT2MNz
X-Originating-IP: [184.108.40.206]
Authentication-Results: mta4434.mail.ne1.yahoo.com  from=onlineportfolio.pro; domainkeys=neutral (no sig);  from=onlineportfolio.pro; dkim=pass (ok)
Received: from 127.0.0.1  (EHLO vps590830.ovh.net) (184.108.40.206)
  by mta4434.mail.ne1.yahoo.com with SMTPS; Sat, 24 Nov 2018 16:33:21 +0000
Received: from onlineportfolio.pro (localhost [127.0.0.1])
    by vps590830.ovh.net (Postfix) with ESMTPA id 1AA6E7F98C
    for <email@example.com>; Sat, 24 Nov 2018 17:33:19 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=onlineportfolio.pro;
    s=mail; t=1543077199;
    bh=lJB/nYL/lnJN4VoNZZF46fk1OIxhohTSP6uovXwnXY8=;
    h=Date:From:To:Subject;
    b=yocLTdOVKGS8kMbxBeHao07IHZF6tc021mMBro7TTLhfZnDlgIgZesZ1uXHT3cmuQ
     1AJBh0MBHnFt35j1VrlAOT+16qwAjq9QGO3vyHfW3rOSmMAOifU9pv+zsizYHpKno0
     OlQXSplwOwMPxmjaQ7CxqX4BrpPjcJo1+xzaz4dkDXrQ6EU83UEElLseOlrux2k3r5
     OGybMbxkW4uttUYjEhao06GiW5EGsplDfqG67trjOOvg5n7cc54FwxLvKRAa+EC+k8
     085lCvxR8jjeeFQTDpCDInARfBPhYNkk15jgsqMo8HmfC1bRGPcZifzm9WaoZcQCkq
     NzcI1PknzIEmw==
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Sat, 24 Nov 2018 17:33:19 +0100
From: onlineportfolio <firstname.lastname@example.com>
To: email@example.com
Subject: Meeting
Message-ID: <email@example.com>
X-Sender: email@example.com
User-Agent: Roundcube Webmail/1.3.8
Content-Length: 93

Dear sir,

Thank you very much for you kindness , we appreciate it very much.

Best regards.

Answer: X-YahooFilteredBulk: 22.214.171.124

Yahoo seems to be indicating that your IP address is responsible for sending bulk mail.
I checked your IP address against several blacklists and it's ok.
Quite simply, you don't have enough reputation to be a trusted sender yet. You will need to exhibit a positive sending history to reduce your chances of being sent to junk mail. This is not uncommon for brand new IPs sending mail.


Para que serve  em XML?

Question: Em alguns exemplos de plataformas, tem entre as tags o <![CDATA []]>, mas afinal para que serve isso no XML?
Modelo de um XML a ser seguido de uma plataforma.
<?xml version="1.0" encoding="UTF-8"?>
<loja>
   <produto>
      <link><![CDATA[http://www.loja.com.br/blusas/blusa-feminina?idparceiro=P1]]></link>
      <imagem><![CDATA[http://www.loja.com.br/imagens/blusa-feminina-1.jpg]]></imagem>
      <nome><![CDATA[Blusa feminina vermelha]]></nome>
      <categoria><![CDATA[Blusas Femininas]]></categoria>
      <descricao><![CDATA[Blusa feminina com gola em V e detalhes em strass.]]></descricao>
      <valor><![CDATA[119,00]]></valor>
      <valor_promo><![CDATA[95,90]]></valor_promo>
      <parcelamento><![CDATA[2x R$47,95]]></parcelamento>
      <cor><![CDATA[Vermelho, Azul, Branco]]></cor>
      <marca><![CDATA[Tommy Hilfiger]]></marca>
      <modelo><![CDATA[Único]]></modelo>
      <material><![CDATA[100% algodão]]></material>
      <tamanho><![CDATA[P, M, G]]></tamanho>
      <sexo><![CDATA[Feminino]]></sexo>
   </produto>
</loja>

Answer: O CDATA serve para indicar que o texto dentro da área dele é um texto comum e não pode ser interpretado como parte da marcação do XML.
Isto é útil quando uma parte do texto de um determinado elemento pode ser confundido com partes da marcação do XML, o que não é o caso do exemplo mostrado. É uma forma de escapar os caracteres.
Deve ter um motivo específico da plataforma para usar indiscriminadamente. Talvez por falha da especificação deles. Talvez porque eles não têm controle sobre o conteúdo e pode haver caracteres conflitantes (pode conter HTML). Talvez por algum motivo que só eles podem explicar (provavelmente via documentação). O que é certo é que deve ser seguido se eles mandam.
Um exemplo de boa utilização seria:
<secao><![CDATA[<sexo>Feminino</sexo>]]><secao>

Coloquei no GitHub para referência futura.
O que parece ser uma tag do XML é apenas um texto comum que por acaso se parece com, e provavelmente representa um outro XML.
Wikipedia.
Comment: Entendo que, uma vez que o trecho CDATA pode aparecer em qualquer parte do XML em que possa ir texto, como por exemplo entre duas tags, acredito que a orientação geral seja buscar na biblioeca de _XML parsing_ que estiver usando se existe o recurso pronto de escapar esse conteúdo CDATA no texto trazido para um determinado componente da árvore XML..

Para que serve  em XML?


sql_mode not updating, even though I've changed it in my.cnf

Question: I'm trying to tweak a database that we moved from a mySQL 5.5 server, to a 5.7 server. It's complaining about the date being 0000-00-00 format. I'm trying to fix this, but I can't get past the errors.
In my.cnf, I have:
sql_mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,TRADITIONAL,ALLOW_INVALID_DATES

I restart mySQL, and then do this in phpmyadmin:
SHOW VARIABLES LIKE 'sql_mode'

But it shows a totally wrong/different list!
sql_mode
STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ALLOW_INVALID_DATES,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

What am I doing wrong? This is driving me nuts!
Comment: Maybe wrong my.cnf? Executed upgrade script? I prefer MySQL Workbench over phpMyAdmin. https://stackoverflow.com/questions/35565128/mysql-incorrect-datetime-value-0000-00-00-000000
Comment: @BitAccesser mmm I'm pretty sure. I used VestaCP for the control system, and in the online tool it shows as `/etc/mysql/my.cnf`  (I'm editing it manually myself, and the online tool is showing my changes as well. Very odd :/
Comment: You talk about phpMyAdmin! What is shown on the console ?
Comment: @BitAccesser all I was saying, is that I use VestaCP to *administer* the server (bit like CPanel), and from that panel I can see the my.cnf location is /etc/mysql/my.cnf , which is the one I'm editing.
Comment: OK, thought it was a MySQL-Tool. A my.cnf file in the file-system has no meaning. Important is the used my.cnf file. You should find the path in phpMyAdmin.
Comment: @BitAccesser nah its just a WHM GUI. Looks like Solarflare got to the bottom of it. What I was seeing with the `SHOW VARIABLES LIKE 'sql_mode' ` is actually the computized value. Because I had `TRADITIONAL` as part of my variables, that was causing a few others to get added (STRICT_TRANS_TABLES, STRICT_ALL_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER), which is where the problem was coming from. All good now :) Thanks for your offers though!
Answer: According to the manual, TRADITIONAL includes several other modes:

TRADITIONAL
Before MySQL 5.7.4, and in MySQL 5.7.8 and later, TRADITIONAL is equivalent to STRICT_TRANS_TABLES, STRICT_ALL_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER, and NO_ENGINE_SUBSTITUTION.
From MySQL 5.7.4 though 5.7.7, TRADITIONAL is equivalent to  STRICT_TRANS_TABLES, STRICT_ALL_TABLES, NO_AUTO_CREATE_USER, and NO_ENGINE_SUBSTITUTION. The NO_ZERO_IN_DATE, NO_ZERO_DATE, and ERROR_FOR_DIVISION_BY_ZERO modes are not named because in those versions their effects are included in the effects of strict SQL mode (STRICT_ALL_TABLES or STRICT_TRANS_TABLES). Thus, the effects of TRADITIONAL are the same in all MySQL 5.7 versions (and the same as in MySQL 5.6). For additional discussion, see SQL Mode Changes in MySQL 5.7.

So the expansion of the mode is expected and your two modes are equivalent for MySQL 5.7.8+. The sql mode setting is actually evaluated: you can see e.g. that your final mode includes the non-default ALLOW_INVALID_DATES, as defined in your configuration file.
Comment: And two people on earth smarter (someday I have to upgrade too!) :-)
Comment: Ah now that is interesting! Sure enough, removing `TRADITIONAL`, rebooting mysql, and then looking at sql_mode in phpmyadmin shows: `

ALLOW_INVALID_DATES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION` , and now it lets me update the DB as expected! :) Thanks!


Choice of technology for loading large CSV files to Oracle tables

Question: I have come across a problem and am not sure which would be the best suitable technology to implement it. Would be obliged if you guys can suggest me some based on your experience.
I want to load data from 10-15 CSV files each of them being fairly large 5-10 GBs. By load data I mean convert the CSV file to XML and then populate around 6-7 stagings tables in Oracle using this XML.
The data needs to be populated such that the elements of the XML and eventually the rows of the table come from multiple CSV files. So for e.g. an element A would have sub-elements coming data from CSV file 1, file 2 and file 3 etc.
I have a framework built on Top of Apache Camel, Jboss on Linux. Oracle 10G is the database server.
Options I am considering,

Smooks - However the problem is that Smooks serializes one CSV at a time and I cant afford to hold on to the half baked java beans til the other CSV files are read since I run the risk of running out of memory given the sheer number of beans I would need to create and hold on to before they are fully populated written to disk as XML.
SQLLoader - I could skip the XML creation all together and load the CSV directly to the staging tables using SQLLoader. But I am not sure if I can a. load multiple CSV files in SQL Loader to the same tables updating the records after the first file. b. Apply some translation rules while loading the staging tables.
Python script to convert the CSV to XML.
SQLLoader to load a different set of staging tables corresponding to the CSV data and then writing stored procedure to load the actual staging tables from this new set of staging tables (a path which I want to avoid given the amount of changes to my existing framework it would need).

Thanks in advance. If someone can point me in the right direction or give me some insights from his/her personal experience it will help me make an informed decision.
regards,
-v-
PS: The CSV files are fairly simple with around 40 columns each. The depth of objects or relationship between the files would be around 2 to 3.
Answer: Unless you can use some full-blown ETL tool (e.g. Informatica PowerCenter, Pentaho Data Integration), I suggest the 4th solution - it is straightforward and the performance should be good, since Oracle will handle the most complicated part of the task.
Answer: In Informatica PowerCenter you can import/export XML's +5GB.. as Marek response, try it because is work pretty fast.. here is a brief introduction if you are unfamiliar with this tool.
Answer: Create a process / script that will call a procedure to load csv files to external Oracle table and another script to load it to the destination table.
You can also add cron jobs to call these scripts that will keep track of incoming csv files into the directory, process it and move the csv file to an output/processed folder.
Exceptions also can be handled accordingly by logging it or sending out an email. Good Luck.


Ao usar fnReloadAjax não permanece na página atual

Question: Estou usando a fnReloadAjax para quando o usuário fazer alguma ação na tabela que é montado com uso do datatables já recarregue os dados novamente, só que não permanece na página atual.
Esta mantendo o filtro, a ordenação, a quantidade de registros por página, mas sempre volta para a primeira página.
Alguém sabe de alguma coisa para solucionar isso?
Answer: Para essa questão cheguei a abrir uma issue no github sobre esse problema na página de plugins do Datatables também e lá me sugeriram usar o método ajax.reload() e foi o que resolveu meu problema, mas lembrando que essa função só esta disponível a partir da versão 1.10 do datatables e para que funcionasse realmente tive que fazer assim:
<code>var oTable = $('#example').DataTable( {
    ajax: "data.json"
});

// na chamada de uma ação, qdo precisar recarregar o conteúdo da tabela fiz assim
oTable.api().ajax.reload(null, false);
</code>


Nearby Messages Api successfully subscribed but onFound never called for Registered Google Beacon

Question: I have been trying out Nearby API for detecting beacons in my Android app. Have followed the instructions on https://developers.google.com/nearby/messages/android/get-started and https://developers.google.com/nearby/messages/android/get-beacon-messages
The connection to GoogleAPI Client is initiated on click of a button on app (for explicit intent) and subscription to Nearby Messages is called in the onConnected callback.
//Inside onCreate
FloatingActionButton fab = (FloatingActionButton) findViewById(R.id.fab);
    fab.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View view) {
            startNearby();
        }
    });

private void startNearby() {
    mGoogleApiClient = new GoogleApiClient.Builder(this)
            .addApi(Nearby.MESSAGES_API)
            .addConnectionCallbacks(this)
            .enableAutoManage(this, this)
            .build();
}
@Override
public void onConnected(Bundle bundle) {
    if(mGoogleApiClient.isConnected()){
        System.out.println("Google_Api_Client: It was connected on (onConnected) function, working as it should.");
    }
    else{
        System.out.println("Google_Api_Client: It was NOT connected on (onConnected) function, It is definetly bugged.");
    }
    subscribe();
}
private void subscribe() {
    mMessageListener = new MessageListener() {
        @Override
        public void onFound(Message message) {
            String messageAsString = new String(message.getContent());
            System.out.println("On Found: "+messageAsString);
            adapter.add("FOUND: "+messageAsString);
        }

        @Override
        public void onLost(Message message) {
            String messageAsString = new String(message.getContent());
            System.out.println("On Lost: "+messageAsString);
            adapter.add("LOST: "+messageAsString);
        }
    };
    SubscribeOptions options = new SubscribeOptions.Builder()
            .setStrategy(Strategy.BLE_ONLY)
            .build();
    Nearby.Messages.subscribe(mGoogleApiClient, mMessageListener, options).setResultCallback(new ResultCallback<Status>() {
        @Override
        public void onResult(@NonNull Status status) {
            if (status.isSuccess()) {
           //     Log.i(TAG, "Subscribed successfully.");
                System.out.println("DONE. Subscribed successfully." +status);
                adapter.add("Subscribing done. "+status);
            } else {
                System.out.println("ERROR. Could not subscribe, status = " + status);
        //        logAndShowSnackbar("Could not subscribe, status = " + status);
         //       mSubscribeSwitch.setChecked(false);
            }
        }
    });
}

In the ResultCallback of Nearby.messages.subscribe call, I am getting success in the response status object-
I/System.out: Google_Api_Client: It was connected on (onConnected) function, working as it should.
I/System.out: DONE. Subscribed successfully.Status{statusCode=SUCCESS, resolution=null}

For the beacons, I am using Beacon Simulator app on Android Play store to simulate Eddystone UID beacon. I have registered two beacons using Beacon Tools app and can  see the beacons active on Google Beacon Dashboard.
The issue is that the onFound / onLost callbacks are never called in my app. Please note that the beacons are detectable by a Beacon Simulator app (on Android) as well the Google Beacon Tools app (under the Registered beacons nearby section).
Will really appreciate any help on what I might be doing wrong here. Thanks!!
Answer: Try this.
btnScan = (Button)findViewById(R.id.btnScan);

btnScan.setOnClickListener(new View.OnClickListener(){
    public void onClick(View v){
        startNearby();
    }
});

mMessageListener = new MessageListener() {
    @Override
    public void onFound(Message message) {
        String messageAsString = new String(message.getContent());
        Log.d(null, "Found message: " + messageAsString);
        Toast.makeText(getApplication(), "Found message: " + messageAsString, Toast.LENGTH_SHORT).show();
    }

    @Override
    public void onLost(Message message){
        String messageAsString = new String(message.getContent());
        Log.d(null, "Lost sight of message: " + messageAsString);
        Toast.makeText(getApplication(), "Lost sight of message: " + messageAsString, Toast.LENGTH_LONG).show();
    }
};

void startNearby() {
    mGoogleApiClient = new GoogleApiClient.Builder(this)
            .addApi(Nearby.MESSAGES_API)
            .addConnectionCallbacks(this)
            .enableAutoManage(this, this)
            .build();
}

private void subscribe() {
    Log.i("", "Subscribing...");
    Toast.makeText(getApplication(), "Subscribing...", Toast.LENGTH_LONG).show();

    try {
        SubscribeOptions options = new SubscribeOptions.Builder()
                .setStrategy(Strategy.BLE_ONLY)
                .build();
        // Nearby.Messages.subscribe(mGoogleApiClient, mMessageListener, options);
        Nearby.Messages.subscribe(mGoogleApiClient, mMessageListener, options);/* */
    }
    catch(Exception ex) {
        Toast.makeText(getApplication(), ex.getMessage(), Toast.LENGTH_LONG).show();
    }
}

@Override
public void onConnected(Bundle connectionHint) {
    subscribe();
}


Quickjob MOVZON X'FF' to OFA1

Question: what does MOVZON X'FF' do in quickjob. I believe it just moves input to output. Please let me know, if I am wrong.
Answer: This reference is used.

Given the instruction:

MOVZON X'FF' to OFA1

The receiving field OFA1 refers to the first record position (the 1) of the output file ( the OF) designated as A. The instruction will set the high-order bits (0-3 or "zone bits") of the first position to ones, matching bits 0-3 of the X'FF'.
However, it appears, as a matter of style, the instruction should have been written as MOVZON X'F0' TO OAF1 since the low-order bits (4-7) are not used.
Comment: Whether or not the target field is signed or unsigned, or whether is really is a numeric field cannot be said from that MOVZON alone. MOVZON moves left halves (nibbles) of bytes from source to target. That's it. Right halves remain untouched.
Comment: Thanks Rick. I too reffered that manual. Thats the only answer I got from google. However, the manual doesnt provide much value for my quesiton.
Comment: @Rick: Just a minor comment. I would talk about "bytes" not "characters" in this context.
Comment: @phunsoft -  Page 2-16 of the referenced document uses "positions". I was more comfortable with _character_ than with _byte_, but had not considered _position_ until after posting the change. I will use the same term as the document.
Answer: The smallest unit of information is the bit. Processors usually don't work on single bits when accessing memory; they work on bytes. A byte consists of 8 consecutive bits (for most architectures).
To describe how different processor instructions work with bytes, bytes are sometimes subdivided into two 4-bit groups, called nibbles. Counting left to right, bits 0-3 are called "left nibble", "high order nibble", or "zone nibble". Bits 4-7, the right half, are called "right nibble", "low order nibble", or "number nibble".
There are instructions that work on the whole byte, e.g. MOVE. And there are instructions that work on nibbles. MOVEZONE (MOVZON) works on zone nibbles and leaves the number nibbles alone; MOVENUM (MOVNUM) works on number nibbles, and leaves the zone nibbles alone.
This kind of instructions are usually used with bytes that contain numeric values, coded as either zoned decimal, or packed decimal. They are rather exotic when working on text data.

Quickjob MOVZON X'FF' to OFA1


Write the sum of the values in to another file using Nodejs asynchronously

Question: I have a file readData.txt has the values "10,20,30,40,50,........"like this I have the numbers.
Now I want to write the sum of those values in to the other file called sumfile.txt. I'm using fs.readFile and fs.writeFile functions which are asynchronous.
I have tried using Promises, It worked. But I'm curious that can we do it without using Promises. I'm trying to achieve that without Promises.
If anybody know any other ways I'll be thankful.
Comment: Using promises is... Promising. It helps you avoid the callback hell.
Comment: I have given you examples using callbacks, which are not promises. However, they are asynchronous too. You need to implement the logic you need after the async operation into the fulfillment of the promise or into the callback. This is the way to go, doing this synchronously is anti pattern and should be avoided.
Comment: You are welcome. In the meantime I edited my answer to give you information about how you can do this synchronously.
Comment: Thank you @Lajos Arpad, The thing is I just want to dig the NodeJs deeper. The main requirement is I have numbers in that file which I'm reading, I want to add those values and I want to store that sum in to another file, Being Asynchronous read and write I'm not sure how much data I'll get at each iteration, So is there any other ways to do this other than Promises?
Comment: Yeah I got an idea ,Thanks a lot, Your help much appreciated:)
Answer: You can use the callback parameter of fs.readFile:
fs.readFile('/etc/passwd', (err, data) => {
  if (err) throw err;
  console.log(data);
});

You can use the callback parameter of fs.writeFile:
const data = new Uint8Array(Buffer.from('Hello Node.js'));
fs.writeFile('message.txt', data, (err) => {
  if (err) throw err;
  console.log('The file has been saved!');
});

EDIT
You can do this synchronously as well, using fs.readFileSync
fs.readFileSync('<directory>');

and fs.writeFileSync
But it is better to keep things async. It is difficult at first, but all your struggles will be rewarded.


Find third and fourth coordinate of a rectangle knowing two coordinates and width

Question: Supposed we have a rectangle ABCD. We have two coordinates of A=(x1,y1) and B=(X2,y2) and its width=W. I want to find the other two coordinates C and D.

I know I might be able to calculate the third coordinate using Thales's theorem as follows.
AO and OC are a circle's radius which in that circle B and D are valid point. So I need to solve the following equation.
(x1-ox)^2 + (y1-oy)^2 = AO^2 and we have AO^2 = w^2 + L^2 and L is distance between A and B.
So I was thinking maybe there is any already solved equation out there to easily find the D and C coordinates.
Thanks
Answer: Sketch of the solution:

Find the perpendicular direction to the line passing through $A$ and $B$.
Take the point $A$ and add $W$ in the direction of the perpendicular line found previously.
To find out $C$, use the equation:
\begin{align*}
\frac{A + C}{2} = \frac{B + D}{2}
\end{align*}
Notice there are two solutions to the proposed problem.
Answer:
Define the vector $v = (v_x, v_y) = (x_2 - x_1, y_2 - y_1) $

Normalize $v$, i.e define $v_N= (v_{Nx}, v_{Ny}) = \dfrac{v}{\|v\|} $

Find the vector $u$ that points $90^\circ$ counter clockwise from the direction of $v_N$, this vector is given by $u_N = (u_{Nx}, u_{Ny}) = (- v_{Ny}, v_{Nx} ) $

Now points $C $ and $D$ are given by (two solutions)
$ C = B \pm w \cdot u_N $ and $D = A \pm w \cdot u_N $
Comment: Thanks, can you please explain part 2 and 3? I want to implement it in code and got confused there a bit
Comment: Vector $v_N = \dfrac{ (x_2 - x_1, y_2 - y_1) }{\sqrt{ (x_2 - x_1)^2 + (y_2 - y_1)^2 } }$
Comment: For vector $u_N$ , swap the coordinates of $v_N$ and change the sign of the first coordinates, so $(v_{Nx}, v_{Ny})$ becomes $(-v_{Ny}, v_{Nx})$


Encoding sem configuração: Eclipse (JSP) e VSCode

Question: Estou com alguns problemas relacionados a encoding. Tanto no eclipse quanto no Visual Studio Code estão apontados para o UTF-8.
Quando eu conserto e salvo usando o eclipse, ele vai de boa. Mas quando eu salvo no VSCode, o encoding desconfigura.
Segue um exemplo:

Eu uso o eclipse só para compilar a minha aplicação, e me sinto muito mais confortável ao desenvolver no VSCode.
Answer: Se estiver em inglês seu VSCode, confirme se a configuração está correta:

Vá em File -> Preferences -> User Settings
Adicione a entrada
<code>"files.encoding": "utf8"</code> na janela da direita e salve.

Você também pode optar por adicionar apenas a entrada:
<code>"files.autoGuessEncoding": true
</code>
E deixar com que o Visual Studio Code adivinhe o encoding do arquivo aberto, como explicado na documentação:

When enabled, will attempt to guess the character set encoding when opening files. This setting can be configured per language too.
Comment: @ThiagoCunha tentou usar o `files.autoGuessEncoding`? O problema pode estar também no Eclipse (usando outro encoding diferente de UTF-8)
Comment: @ThiagoCunha, está usando Windows? Se sim, abra o arquivo no Notepad, vá em Arquivo > Salvar Como e veja o que aparece selecionado no campo "Codificação"
Comment: Qual aparece selecionada quando você abre o arquivo? Este é o encoding atual do arquivo. Pedi isto para você para confirmar o encoding do arquivo.
Comment: Vamos [continuar esta discussão no chat](https://chat.stackexchange.com/rooms/79484/discussion-between-dherik-and-thiago-cunha).
Comment: No meu VSCode já se encontra nessa configuração: "files.encoding": "utf8"
Comment: Tentei sim! Fui lá no eclipse, e ainda esta sendo usado o UTF-8, inclusive na configuração do projeto.
Comment: ANSI, UTF-8, UNICODE, UNICODE BIG Endian
Comment: No meu eclipse esta:Text file encoding: UTF-8/ New text file line demimiter: Unix
Answer: Eu consegui isolar o problema e acabei descobrindo que esse problema é herdado do eclipse. Para tal teste, fiz os seguintes procedimentos:

Criei arquivos no VSCode e no Bloco de Notas com extensão .txt, e ambos abriram normalmente
Criei arquivos em ambas as aplicações com a extensão .jsp. Quando eu criei e passei do eclipse para outras aplicações, gerou problema de caracter. Quando criei também no bloco de notas e no VSCode, os dois se comunicaram bem, mas quando abri no Eclipse, deu erro nos caracteres.

Então, erro no eclipse!
Answer: Boa tarde pessoal,
Primeiramente desculpa pela pobreza de argumentações.
Fiz um teste no VS alterando o encoding (File > Preferences > Settings > Files: Encoding) para utf8bom (Byte order mark). Parece que ele não traz grandes implicações para o projeto. (Fonte: Qual a diferença entre arquivos encodados com BOM e sem BOM? )
Deste modo, o Eclipse passou a identificar as acentuações corretamente.
Espero ter ajudado!