Density of a sum

Question: Let $X, Y$ be random variables having the joint density $f(x,y) = (a+1)(a+2)(y-x)^a, 0 \leq x < y \leq 1$, and $f(x,y) = 0$ elsewhere. Compute the density of $Z = X+Y$.
My solution doesn't match the textbook's and I'm not sure why.
If $0 < z \leq 1$,
$$
\begin{align}
F_Z(z) & = \int_0^z \int_x^{z-x} (a+1)(a+2)(y-x)^a \mathrm{d}y\, \mathrm{d}x \\[8pt] & = \left. \int_0^z  (a+2)(y-x)^{a+1} \right|_x^{z-x}\mathrm{d}y\, \mathrm{d}x \\[8pt]
&= \int_0^z (a+2)(z-2x)^{a+1} \mathrm{d}x \\[8pt]
&= \left. \frac{(z-2x)^{a+2}}{-2} \right|_0^z \\[8pt]
&= \frac{z^{a+2}}{2} - \frac{(-z)^{a+2}}{2}
\end{align}
$$
Therefore,
$$
\begin{align}
f_Z(z) = \frac{\mathrm{d}}{\mathrm{d}z}F_Z(z) = \frac{a+2}{2}\left(z^{a+1} + (-z)^{a+1} \right)
\end{align}
$$
My setup for the case $1 < z < 2$,
$$
\begin{align}
F_Z(z) &= \int_0^{z-1} \int_0^1  (a+1)(a+2)(y-x)^a \mathrm{d}y\, \mathrm{d}x  + \int_{z-1}^1 \int_0^{z-x}  (a+1)(a+2)(y-x)^a \mathrm{d}y\, \mathrm{d}x
\end{align}
$$
The solution for the first case is $\frac{a+2}{2} z^{a+1}$, so there is something with the term $(-z)^{a+1}$ but I can't figure out.
Answer: To find the bounds, it is useful to draw a picture. The joint density "lives" in the part of the square that is above the line $y=x$.
The interval of integration in the $0\lt z\le 1$ part is not right. The minimum value of the variable $y$ should be $x$. You have $y$ going instead from $0$ to $z-x$.
For the range of $x$, the largest possible value is where $x+y=z$ meets the line $x=y$. Thus $x$ should go from $0$ to $z/2$.
Comment: I agree, but that still doesn't remove eliminate the term $(-z)^{a+2}$.
Comment: I added to the post, giving the modification needed for the outer integral.

Density of a sum


How to cope with serial correlation and time effects in a panel data model in R

Question: I am building a panel-data model for macroeconomic analysis and I am currently stalling on how to deal with some problems. Diagnostic tests indicate that there is cross-sectional dependence (which is logical), serial correlation and that i should use time-fixed effects. (And probably there is much more...)
Here is a link to my data: Panel Data
Here is the code (abbreviated for better usage):
#--------------load libraries
options(java.parameters = "-Xmx1024m")
library(XLConnect)
library(plm)

#--------------read data
wb <- loadWorkbook("Panel.xlsx")
df <- readWorksheet(wb, sheet="Sheet1")

#--------------Fixed effects paneldata model
fixed <- plm(Price ~ Income + Housing_units + Population_age + Population_density + Unemployment + Real_mortgage_rate + Expected_GDP_growth,
             data=df, drop.unused.levels = TRUE, index=c("Id", "Year"), model="within")
summary(fixed)

#--------------Random effects paneldata model
random <- plm(Price ~ Income + Housing_units + Population_age + Population_density + Unemployment + Real_mortgage_rate + Expected_GDP_growth,
              data=df, drop.unused.levels = TRUE, index=c("Id", "Year"), model="random")
summary(random)

#--------------Diagnostics
phtest(fixed, random) # Fixed-effects model better than random-effects model
plmtest(fixed, c("time"), type=("bp")) # Need of time-fixed effects
pcdtest(fixed, test = c("lm")) # Test indicates cross-sectional dependance
pcdtest(fixed, test = c("cd")) # Test indicates cross-sectional dependance
pbgtest(fixed) # Test indicates serial correlation

Do you have any ideas for improvement or next steps? Or to be more precise: How can I implement time-fixed effects? And how can I deal with serial correlation?

Edit:

Here is a sample of the data:
                                      Id Year     Price   Income Housing_units Population_age Population_density Unemployment Real_mortgage_rate Expected_GDP_growth
1                Arrondissement d'Anvers 2001 106447.82 18826.26     0.2844412      0.3608039       0.0010737967   0.05793009         0.04277500               0.021
2                Arrondissement d'Anvers 2002 118716.84 18679.44     0.2838416      0.3593445       0.0010694183   0.07099468         0.03880833               0.021
3                Arrondissement d'Anvers 2003 123591.79 18503.93     0.2829195      0.3583308       0.0010633930   0.07683871         0.03296667               0.015
4                Arrondissement d'Anvers 2004 127741.49 18489.21     0.2820850      0.3568954       0.0010586214   0.07371481         0.03050833               0.017
5                Arrondissement d'Anvers 2005 175851.72 18446.58     0.2814552      0.3548318       0.0010542290   0.07328001         0.01900000               0.014
6                Arrondissement d'Anvers 2006 194872.80 18863.60     0.2803233      0.3524605       0.0010477999   0.06778650         0.01948333               0.018
7                Arrondissement d'Anvers 2007 208739.77 19226.49     0.2786186      0.3504475       0.0010407672   0.06076701         0.02731667               0.018
8                Arrondissement d'Anvers 2008 210862.56 19225.50     0.2766267      0.3484786       0.0010317159   0.05498332         0.01218333               0.010
9                Arrondissement d'Anvers 2009 212451.01 19482.80     0.2748636      0.3466580       0.0010232215   0.07004942         0.03553333               0.013
10               Arrondissement d'Anvers 2010 221431.63 18989.30     0.2727264      0.3451426       0.0010152046   0.07529093         0.02287500               0.018
11               Arrondissement d'Anvers 2011 219939.30 18277.79     0.2693328      0.3440205       0.0010021064   0.06460002         0.01052500               0.015
12               Arrondissement d'Anvers 2012 218487.50 17916.28     0.2664027      0.3424343       0.0009904085   0.06914538         0.01032500               0.010
13             Arrondissement de Malines 2001 100110.58 19524.44     0.3509642      0.3715508       0.0016651232   0.03690743         0.04277500               0.021
14             Arrondissement de Malines 2002 108968.27 19334.37     0.3514912      0.3711313       0.0016595987   0.04664770         0.03880833               0.021
15             Arrondissement de Malines 2003 115356.87 19336.42     0.3516660      0.3701327       0.0016529746   0.05534163         0.03296667               0.015
16             Arrondissement de Malines 2004 119105.32 19200.12     0.3515692      0.3689736       0.0016465732   0.05250104         0.03050833               0.017
17             Arrondissement de Malines 2005 154714.71 19360.47     0.3510717      0.3672112       0.0016380779   0.05320718         0.01900000               0.014
18             Arrondissement de Malines 2006 172113.51 20007.19     0.3496116      0.3647687       0.0016236333   0.04852456         0.01948333               0.018
19             Arrondissement de Malines 2007 186916.68 20447.05     0.3496034      0.3628630       0.0016134620   0.04139262         0.02731667               0.018
20             Arrondissement de Malines 2008 190700.73 20521.17     0.3483314      0.3602553       0.0015988850   0.03687283         0.01218333               0.010

Here are some results of the diagnostics:
    Lagrange Multiplier Test - time effects (Breusch-Pagan)

data:  Price ~ Income + Housing_units + Population_age + Population_density +  ...
chisq = 1522.558, df = 1, p-value < 2.2e-16
alternative hypothesis: significant effects

    Breusch-Godfrey/Wooldridge test for serial correlation in panel models

data:  Price ~ Income + Housing_units + Population_age + Population_density +     Unemployment + Real_mortgage_rate + Expected_GDP_growth
chisq = 233.2558, df = 12, p-value < 2.2e-16
alternative hypothesis: serial correlation in idiosyncratic errors

Comment: I think you have to do more to explain what the results show and what your doubts are. Although you add comments to the code, you seem to be expecting that we will run the code, look at the results and then tell you what to do next. That might work, but it's expecting more effort than people will typically give here for free. It's best to ask very specific questions and show the evidence directly (e.g. tables and graphs). Advice on an entire project is a large question! Also, beware that software-specific questions without an evident statistical question at their heart are off-topic here.
Comment: I edited my question, hoping it helps. Because there is apparently no programming problem/error, I thought it would be best to post my question here instead of in stackoverflow. Thanks for your suggestion.
Comment: I agree that this belongs here rather than SO. You have made a good answer more likely. Strongly econometric questions seem to get few answers here, perhaps because the econometricians have been to too many lectures on competition, but there are some very competent people lurking who may reply.
Answer: Your question is not very clear, and the link to the data is no longer working...
For the time fixed effects, your call should look like this:
fixed <- plm(Price ~ Income + Housing_units + Population_age +
   Population_density + Unemployment + Real_mortgage_rate + Expected_GDP_growth,
   data=df, index=c("Id", "Year"), model="within", effect="time")

If you want both individual and time FEs you can also use effect="twoways".
To deal with serial correlation you can use vcovHC.plm(), which by default computes SEs clustered by group, i.e. robust wrt heteroscedasticity and arbitrary correlations within the clusters. See Chapter 14.4 of Using R for Introductory Econometrics (Heiss, 2016). (You can also read it online.) To obtain robust SEs is easy:
require(lmtest)
coeftest(fixed, vcov. = vcovHC)

All of this is discussed in the plm vignette:
http://cran.at.r-project.org/web/packages/plm/vignettes/plm.pdf


Function that returns negative number for negative x and y

Question: I need a function, $$f(x,y):\mathbb{R} \times \mathbb{R} \rightarrow \mathbb{R} $$, that returns negative a number only if both x and y are negative. It can use only the four basic operations, so for example computing the absolute value or square root are not allowed.
I only found solution using the absolute value:
$$ -(x-|x|)*(y-|y|)$$
Is it possible with only the four basic operation (+, -, *, and/)? And if it isn't, why?
Comment: Should the return of a negative be 'if and only if' both $x$ and $y$ or negative; or, as stated in the post, 'only if both $x$ and $y$ are negative'?
Comment: if and only if both x and y are negative.
Comment: @mvw finite many operations
Comment: Out of curiosity,  is this restriction to those four operations something that's an arbitrary part of an assignment from an instructor,  or does it have some other context?
Comment: @Alan I don't really know why is this restriction. I was just asked for help to figure out this small part of a bigger problem.
Comment: Why don't you construct a two-branch function which will be positive if x,y are positive and negative if x,y are negative instead?
Comment: Finite many or infinite many operations?
Comment: It is not for some clever programming trick using overflow effects of the finite representation?
Comment: Nit: since you're defining an algebra, what you really have is *two* operations.  Subtraction is actually addition with a different element (-6 , rather than "minus" +6).
Comment: @CarlWitthoft Clear, but how do You write reciprocals without division?
Comment: The reciprocal of $x$ is the element $y$ (in the same set of elements underlying your algebra) such that $x * y = 1$ -- or whatever your defined "unity" for the defined $ * $ operator is.
Answer: If you are restricted to the four basic operations, all you can get is a rational function $f(x,y)=\frac{p(x,y)}{q(x,y)}$ where $p,q$ are polynomials in $x$ and $y$.
Since we want $f$ to be defined everywhere, $q$ must not have zeroes, so we may assume wlog. that $q(x,y)>0$ for all $x,y$. This allows us to use simply $f(x,y)=p(x,y)$, i.e. a polynomial.
If your assignment formulation really reads "only if", you can simply take $f(x,y)=0$, which never returns negative values; or try something more interesting such as $(x+1)^2+(y+1)^2-1$, which at least becomes negative for some points in the thord quadrant, but never in the other quadrants.
If what you want is "if and only if", however, there is no such polynomial $p$: If one plugs in an arbitrary constant $c\in\mathbb R$ for $y$, one obtains an univariate polynomial $f_c(x)=p(x,c)$ and $\deg f_c$ is the same for all $c$ with at most finitely many exceptions: If we write $p(x,y)=\sum_{i,j}a_{i,j}x^iy^j$ (where only finitely many $a_{i,j}$ are nonzero), let $d=\max\{\,i\mid a_{i,j}\ne 0\,\}$. Then $\deg f_c\le d$ for all $c$ and the inequality is strict only if $c$ is among the fintely many solutions of $\sum_{i=d}a_{i,j}c^j=0$.
But for the desired condition, we need that $\deg f_c$ is even for all $c\ge 0$ and odd for all $c<0$.
Comment: @Nickolas That's a matter of notational convention. If $S$ is a set I usually write $\max S$, $\sup S$, $\min S$, $\inf S$ for the maximum, supremum etc. For finite cases (i.e. if $S=\{x,y,z\}$, say) I usually write $\max(x,y,z)$ instead of $\max\{x,y,z\}$, but I personally won't write $\max(\{x,y,z\})$, for instance.
Comment: Pardon me, but should'nt $d=\max\{\,i\mid a_{i,j}\ne 0\,\}$ be $d=\max \left( \{\,i\mid a_{i,j}\ne 0\,\} \right)$ ?
Comment: You mean *third* quadrant, not fourth quadrant.
Comment: @JeppeStigNielsen Late, but edited accordingly. While math.SE is a very active site, I should possibly subscribe to counting.SE  ;)
Comment: Couldn't $q$ be a sum of squares? On $\mathbb R^2$, such a polynomial is never negative. I don't think this affects your answer in any essential way…
Comment: @Lubin Yes $q$ could be a sum of squares, hence be always positive, hence can be ignored and we can simply use the numerator instead. That's what I meant.


Recalibration by regressing on intercept only with offset

Question: Consider a risk prediction model developed on one population. When transporting this model to another population, the calibration can be off. There are several strategies for recalibration. One is to fit a new logistic regression model with an intercept only and an offset term for the linear predictor (coefficients from developed model times new data). The resulting linear predictor is considered the recalibrated risk (Steyerberg, Clinical Predictions Models, Chapter 20). What is the exact logic of this approach? I can imagine if there is only a difference in prevalence, then the intercept will be something like the average difference across scores.
Specifically, what is the exact interpretation of the linear predictor resulting from the logistic regression of an intercept only with offset and why?
Answer: If you read:
Validation and updating of predictive logistic regression
models: a study on sample size and shrinkage (pdf)
The same author explains in a little bit of detail. The recalibration approach is to define i linear predictor, $Z$ as $Z = \tilde{\alpha}_{model} + \tilde{\beta}_1 X_1 + ... + \tilde{\beta}_p X_p$ and to use this linear predictor in a regression model: $\hat{Y}_{cal} = \hat{\alpha}_{overall} + \hat{\beta}_{overall}$. If no recalibration is required, $\hat{\alpha}_{overall} = 0 $ and $\hat{\beta}_{overall}$ = 1. If they significantly differ from this case then there is a need to recalibrate the model.
Recalibrated regression coefficients can be calculated as $\hat{\beta}_{cal} = \hat{\beta}_{overall} \tilde{\beta}_i$. One could say that the recalibrated model borrows the relative effects (ratios) of the regression coefficients from the model in the training set.
The case which you describe in your question arises when $\hat{\beta}_{overall} = 1$, i.e. $\hat{\alpha}_{overall} | \hat{\beta}_{overall}=1$ where the slope is set to unity by entering $Z$ as an offset term to the model. Predicted risks are on average understimated if $\hat{\alpha}_{overall} | \hat{\beta}_{overall}=1>0$ and overestimated if $\hat{\alpha}_{overall} | \hat{\beta}_{overall}=1 < 0$.
Updating of the intercept and fixing $\hat{\beta}_{overall}$ to one is a simple recalibration method that intends to correct calibration at the macro scale, i.e. to make the average predicted probability equal to the observed overall event rate.


Destructor of a static object constructed within the destructor of another static object

Question: I have some problems with destructor, in next code:
#include <stdlib.h>
#include <cstdio>

class Foo2
{
    public:
        Foo2() { printf("foo2 const\n"); }

        ~Foo2()
        {
            printf("foo2 dest\n"); //  <--- wasn't called for bionic libc
        }
};

static Foo2& GetFoo2()
{
    static Foo2 foo2;
    printf ("return foo2\n");
    return foo2;
}

class Foo1
{
    public:
        Foo1() { printf("foo1 const\n"); }

        ~Foo1()
        {
            printf("foo1 dest\n");
            GetFoo2();
        }
};

int main( int argc, const char* argv[] )
{
        printf("main 1 \n");
        static Foo1 anotherFoo;
        printf("main 2 \n");
}

Why destructor for foo2 wasn't called for bionic and was for glibc?
EDIT
Output for bionic:
main 1
foo1 const
main 2
foo1 dest
foo2 const
return foo2

Debug info:
(gdb) break 22
Breakpoint 1 at 0x8048858: file test.C, line 22.
(gdb) info breakpoints
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   0x08048858 in Foo2::~Foo2() at test.C:22
(gdb) cont
[    exited with code 0]

Comment: @LuchianGrigore Two popular standard C and C++ library implementations: glibc is the GNU one, Bionic is for Android.
Comment: Seems like a bug in bionic...
Comment: i would think this is a bug in Bionic
Comment: what are bionic and glibc?
Comment: http://stackoverflow.com/questions/246564/what-is-the-lifetime-of-a-static-variable-in-a-c-function
Comment: Seems like undefined behavior to me (see my answer), so both compilers are correct.
Comment: See also: http://stackoverflow.com/questions/3736212/destruction-of-objects-with-static-storage-duration
Answer: I think your code has undefined behavior, although the standard
isn't really clear about it (or I can't find it in the
standard).  Your code constructs a new static object in the
destructor of a static object.  The standard doesn't address
this case, but:

It does say that destructors must be called in the reverse
order of construction.  In your case, this would imply that the
static object in GetFoo2 must be destructed before it was
constructed, which is self-contradictory.
The text in §3.6/3 describes the sequencing of destructors
and functions registered with atexit.  The requirements are
such that the same registration mechanism must be used for each.
And calling atexit once you've called exit (or returned from
main) is undefined behavior.
There's also §3.6/2, which says that "If a function contains
a block-scope object of static or thread storage duration
that has been destroyed and the function is called during the
destruction of an object with static or thread storage
duration, the program has undefined behavior if the flow of
control passes through the definition of the previously
destroyed blockscope object."  This sentence talks about alread
destroyed objects, but it doesn't take much imagination to think
that the absense of "not yet constructed" objects is just an
oversight.

In the end, I'd say that my first point above is conclusive with
regards to intent.  In §1.3.24, there is a note (non normative,
but indicative of intent) "Undefined behavior may be expected
when this International Standard omits any explicit definition of
behavior or when a program uses an erroneous construct or
erroneous data."  In this case, the only description of the
required behavior is impossible (since you cannot destruct an
object before it was constructed), and the standard says nothing
about how this should be resolved.
Comment: I'm less convinced. (1) I think the intent is as if all statics were on a stack, which woukdnt pose a problem for this code. (2) I dont think static destructors use atexit.  I think this is well defined behavior.
Comment: I may agree on the undefined based on §1.3.24 but that's about it. I see nothing (in n3337) in §3.6 [termination] that addresses the specific case at hand. Verbatim all that is addressed is that if an object is constructed after another, then its destructor shall be called first, however we routinely construct object during the execution of destructors...
Comment: I don't see any undefined behavior or contradiction. The lifetime of the Foo1 has ended (since its destructor call started), so the destructor of the Foo2 should run, and it should run after the destructor of the Foo1.
Answer: All the instances I see in this code are static.
By consequence their destructor is called at the end of the executable, after main is finished.
if the destructor wasn't called then it was a bug.
Comment: Looks to me like `foo2` will not be constructed until program termination, at the time when `anotherFoo` is destructed.
Comment: None of the instances I see should be constructed before `main()`.  They're all local statics, and should be constructed the first time the come into scope.
Comment: You bring me some doubts, I'm gonna check about construction of static instances... Let's be sure.
Comment: @JasonD and JamesKanze **You are right**. Cf. http://stackoverflow.com/questions/3063027/when-exactly-is-constructor-of-static-local-object-called . I'm gonna edit my answer accordingly. Thx for pointing me this out.
Answer: static object will be destroyed when program exists. put a break point at ~Foo2(), you will see it or write log to a file should help you diagnose it. If really not called then it's a compiler bug.

And it's fun to upload a pic to answer a question.
Comment: It is fun, but when compilers disagree, answers should refer to a higher authority to back up their claims: the Standard.
Answer:
C++11 3.6.3/1: Destructors for initialized objects [...] with static storage duration are called as a result of returning from main

At the point at which the program returns from main, anotherFoo has been initialised; but foo2 hasn't, since it is not initialised until the first call to GetFoo2 during the destruction of anotherFoo. Therefore, a strict interpretation of the rules would imply that its destructor should not be called.
Comment: @JamesKanze: "called as a result of **returning from `main`**", (or calling `std::exit`, but that's not relevant here), not as a result of leaving the program. As I said, it's not initialised when the program returns from `main`.
Comment: Or that its destructor should be called before the constructor.  Or whatever.  I think it's a case of undefined behavior.
Comment: @JamesKanze: It certainly doesn't imply that the destructor should be called before the constructor: "Destructors **for initialized objects** [...] are called".
Comment: And the object is initialized before you leave the the program.  (But I agree that the standard isn't at all clear about the required behavior.  Which is why I would argue that it is undefined behavior.)


Is the set of probability measures on a compact metric space (weak*-)closed?

Question: Let $(S,\mathcal B)$ be a compact metric space with the Borel-$\sigma$-Algebra. Let $\mathcal M$ be the space of signed Borel measures and $\mathcal P \subset \mathcal M$ the set of probability measures on $(S,\mathcal B)$.
Now we say a sequence of measures $(\mu_n)_{n\in\mathbb N}$ with $\mu_n\in \mathcal M$ for each $n$ converges in the weak*-topology to a measure $\mu \in \mathcal M$ if
(I) $\int fd\mu_n\overset{n\rightarrow\infty}{\rightarrow}\int f d\mu$ for each countinuous and bounded function $f$ from $S$ to $\mathbb R$ that vanishes at infinity
(since $S$ is compact there aren't unbounded continuous $f$ anyways, and also "vanishing at infinity" has no meaning, hence this convergence coincides with the weak convergence of measures).
I know that for non-locally-compact sets $S$ the $\mathcal P$ is not closed in this topology. But several times I stumbled over the restriction non-locally-compact which makes me wonder how to show closedness of $\mathcal P$ in the compact case, i.e.
Question: Why can't there be a $\mu \in \mathcal M\setminus\mathcal P$ and a sequence of probability measures $(\mu_n)_{n\in\mathbb N}$ such that (I) holds?
What I have tried:
If $\mu\not\in\mathcal P$ then either $\mu(S)\neq1$ or there is a set $A\in\mathcal B$ such that $\mu(A)<0$. In the former case choose $f\equiv1$ and see that (I) cannot be true.
In the latter case I think there has to be some open set $U$ such that $\mu(U)<0$ and then I can take a function $f$ that is strictly positive on $U$ and zero elsewhere, then (I) wouldn't hold either (left side would be zero or greater and right side negative). But I don't know if there really has to be such a set $U$.
I already asked a different question regarding the existence of such a set (Obtaining the measure of a set from a limit of measures of open sets), but my approach there doesn't work.
Comment: There will be an open $U$ with $\mu(U) < 0$ (this follows from outer regularity of the measure) but it does not imply that a function $f$ strictly positive on $U$ and zero elsewhere must have negative integral, since $\mu$ could assign positive measure to some subset of $U$, and $f$ could be large on that subset.
Answer: Let $A$ be any Borel set in $S$.  By Lusin's theorem, we can choose a sequence of continuous functions $f_m$ with $f_m \to 1_A$ $\mu$-almost everywhere.  Moreover, if we set $g_m = \max(\min(f_m, 1), 0)$ then $g_m$ is also continuous, $0 \le g_m \le 1$ and $g_m \to 1_A$ $\mu$-almost everywhere as well.
Since $\int g_m\,d\mu = \lim_{n \to \infty} \int g_m\,d\mu_n$, we have $\int g_m\,d\mu \ge 0$.  And by dominated convergence, $\lim_{m \to \infty} \int g_m \,d\mu =  \int \lim_{m \to \infty} g_m \,d\mu = \int \mathbf 1_A \,d\mu = \mu(A)$.  So $\mu(A) \ge 0$, and we conclude that $\mu$ is a positive measure.
Comment: @Paul: Yes, the same argument works in the locally compact case.
Comment: The compactness of $S$ is only needed for Lusin's theorem, which also applies to spaces that are only **locally** compact, as a quick web search told me. Why doesn't your idea work for those spaces (or does it)?
Comment: @NateEldredge Lusin's theorem and the dominated convergence theorem also work for signed measures? Implicitly, this is what you are doing.
Comment: @Satana: Lusin is valid for signed measures - you can apply the standard version of Lusin using the total variation measure $|\mu|$, and saying that the sequence converges $\mu$-a.e. is the same as saying $|\mu|$-a.e.  DCT is valid for signed measures when you suppose that the dominating function is $|\mu|$-integrable, which in this case it is.
Comment: @NateEldredge In general, are you saying that if $\mu$ is a signed measure and $\lim_{m\rightarrow\infty} f_m = f$ $\vert\mu\vert$-a.e. and $\vert f_m\vert \le g$ $\vert \mu\vert$-a.e. with $\int_{X}gd\vert\mu\vert <\infty$ then $\int_X f_md\mu\rightarrow \int_X fd\mu$? Obviously, in our case $g=1$, but how do we know that $\vert\mu\vert(X)<\infty$? Some consider the set of positive measures as a subset of the set of all signed measures (where you define a signed measure to take at most one of $\infty$ or $-\infty$). So it doesn't seem like $\vert \mu\vert(X)<\infty$ in general.
Comment: @NateEldredge Or are we implicitly taking $\mathcal{M}$ to be the space of Borel signed measures of **finite total variation**, and not the space of **all** signed measures?
Comment: @NateEldredge Sorry for all the comments. I see your first point. We have $\vert \mu\vert = \mu^+ + \mu^-$ where $\mu = \mu^+ - \mu^-$ is a Hahn decomposition of $\mu$. Then since $\vert \mu\vert(E) = 0$ implies $\mu^+(E)= \mu^-(E)=0$, we can get a DCT for each $\int f_md\mu^{\pm}$ and piece everything together. (Correct?) But it still seems like my second point could be an issue if $\mu$ does not have finite total variation (equiv., if $\vert \mu\vert(X)=\infty$). So I still have the question as to whether $\mathcal{M}$ is the space of all Borel signed measure of finite total variation?
Comment: Yes, I'm assuming here that "signed measure" means "finite signed measure", which implies that it has finite total variation.  That is the more common definition, in my experience, and it's the only way for the weak-* convergence defined in the question to make sense.  (Your DCT proof is correct, by the way.)


Limit of sin function

Question: I need some help solving the following limit:
Here is how I started:
$$\lim_{x \to 0^+}\frac{\sin(6x)}{\sqrt{\sin(2x)}} = \lim_{x \to 0^+}\frac{\sin(6x)\sqrt{\sin(2x)}}{{\sin(2x)}}=\lim_{x \to 0^+}\frac{\frac{6 \sin(6x)}{6x}\frac{\sqrt{\sin(2x)}}{x}}{{\frac{2 \sin(2x)}{2x}}}$$
I don't know how to deal with this sqrt...
Thanks! :)
Comment: Unfortunately $\lim_{x \to 0^+} \frac{\sqrt{\sin 2x}}{x}$ does not exist.
Comment: Write $\sin x$ as \sin x for better formatting. You can see more at [how to use MathJax](https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference).
Answer: The limit equals:
$$\lim_{x \to 0^+} \frac{\sin(6x)}{6x} \frac{1}{\sqrt{\frac{\sin(2x)}{2x}}} \cdot 6x \cdot \frac{1}{\sqrt{2x}} = \lim_{x \to 0^+} 1 \cdot 1 \cdot 6x \cdot \frac{1}{\sqrt{2x}} = \lim_{x \to 0^+} \frac{6}{\sqrt{2}} \sqrt{x} = 0.$$
Comment: I see, so I need to multiply the whole thing with $\sqrt (2x)$
Comment: I didn't understand what you did here, how the sqrt contains 2x?
Comment: We have $\frac{1}{\sqrt{\frac{\sin(2x)}{2x}}} \cdot \frac{1}{\sqrt{2x}} = \frac{1}{\sqrt{\sin(2x)}}$.
Answer: Substitute $y=2x\to 0^+$:
$$\lim_{y\to 0^+} \frac{\sin 3y}{\sqrt{\sin y}}=\lim_{y\to 0^+} \frac{3\sin y-4\sin^3 y}{\sqrt{\sin y}} =\lim_{y\to 0^+} 3\sqrt{\sin y}-4(\sin y)^{5/2} =0 $$
Comment: Thanks, why is the limit of $(\sin(y))^\frac{5}{2}$ is 0?
Comment: @Math4me $\sin y$ goes to zero, so $0^{\frac 52}$ will also go to zero!
Answer: Use the asymptotic equivalence $\sin(x) \sim x$ for $ c\rightarrow 0^{+}$.
Hence you get $$\lim_{x \rightarrow 0^{+}} \frac{6x}{\sqrt{2x}} = \lim_{x \rightarrow 0^{+}} \frac{6\sqrt{x}}{\sqrt{2}} =0 $$
Comment: Thank you, unfortunately I do not know what asymptotic is yet.
Comment: Thank you for that! :)
Comment: If $g(x) \sim h(x)$, $\lim_{x \to \infty} \frac{g(x)}{h(x)} = 0$. For the limit to $0^+$, you can get this by substituting $u = 1/x$ which gives you a limit in terms of $u$, and $\sin(1/u) \sim 1/u$ is really the same as $\sin x \sim x$ by another change of variables. Note that for the limit to exist, you need $g(x)$ and $h(x)$ to be continuous around $0^+$ (or usually $+\infty$), which in this case is true.
Comment: I refer you to this threat to have a glimpse of it https://math.stackexchange.com/questions/3959757/visual-intuition-for-the-definition-of-asymptotically-equivalent/3959790#3959790
Comment: If you learn asymptotic equivalence you can solve basically 90% of limits just in your mind, without pencil and paper! It's terrific!
Comment: Exactly, just a minor typo on the limit, that should be 1 ;)
Answer: Your last step has an error.  The $x$ under the square root shouldn't be there.  So your next step should be
$$= \lim_{x\to 0^+} \frac{6}{2}\sqrt{\sin 2x} = 0.$$
Comment: You are right!! Thank you!! :)

Limit of sin function


While attempting to calculate D for an RSA implementation, D always comes out negative

Question: I am trying to calculate d for RSA using the extended euclidean algorithm. However, everytime I run in it turns out to be negative.
This is the code for the algorithm, provided a is phi and e is 65537:
BigInteger exEuc(BigInteger a, BigInteger b){
    BigInteger x = 0;
    BigInteger prevX = 1; //holds the previous value of x
    BigInteger y = 1;
    BigInteger prevY = 0; //holds the previous value of y
    BigInteger temp, q;
        while(b != 0){
            q = a / b;
            temp = b;
            b = a % b;
            a = temp;
            temp = x;
            x = prevX-q*x;
            prevX = temp;
            temp = y;
            y = prevY-q*y;
            prevY = temp;
      }
      return prevY; //this ends up being d
}

I have tested the result using e*d mod phi which gives 1 like it's supposed to but I know d is not supposed to be negative. Any idea what is going wrong here?
Comment: Have you tried using an *unsigned* big number? Or printing it as an *unsigned* number?
Comment: Have you tried stepping through your code with a debugger, to figure out where, and why, it ends up being negative?
Comment: Yes, that's the way the Euclidean algorithm works. For computing inverses, where you always want the positive remainder, [you need to add `b` to the answer if it is negative](https://en.wikipedia.org/wiki/Extended_Euclidean_algorithm#Computing_multiplicative_inverses_in_modular_structures).
Comment: The number remains negative when I add a OR b back, and in both cases the e*dmodphi=1 test fails when I try
Comment: I don't what you're saying, but if you add `phi` to d it will be positive and e*d will equal 1 mod phi.
Answer: There's nothing wrong with your implementation of the extended Euclidean algorithm, as shown by the fact that it gives a correct modular inverse d for the encryption exponent e.
However, for RSA, you do want a positive value for d.  To get that, simply let:
d = exEuc(phi, e) % phi;

That's assuming that the % operator for your BigInteger library always returns a non-negative result.  If that's not the case, then at least the result should be greater than -phi, in which case you can simply do:
d = exEuc(phi, e) % phi;
if ( d < 0 ) d += phi;

This works because, in modular arithmetic, adding or subtracting (an integer multiple of) the modulus does not change the congruence class of a number.  In particular, if (d*e) % phi == 1, then also ((d+phi)*e) % phi == 1.

BTW, there's a couple of optimizations you can make to your exEuc() code.  The first one is that you're not actually using the x and prevX values for anything in the end, so you can simply get rid of them entirely, like this:
BigInteger exEuc(BigInteger a, BigInteger b){
    BigInteger y = 1;
    BigInteger prevY = 0; //holds the previous value of y
    BigInteger temp, q;
        while(b != 0){
            q = a / b;
            temp = b;
            b = a % b;
            a = temp;
            temp = y;
            y = prevY-q*y;
            prevY = temp;
      }
      return prevY; //this ends up being d
}

The second optimization is that, since we're only interested in the result modulo phi, we can actually reduce y modulo phi any time we want during the calculation to keep it from getting too large (or negative!):
// calculate the modular inverse of x modulo m
BigInteger modInv(BigInteger x, BigInteger m){
    BigInteger a = x, b = m;
    BigInteger y = 1;
    BigInteger prevY = 0; //holds the previous value of y
    BigInteger temp, q;
        while(b != 0){
            q = a / b;
            temp = b;
            b = a % b;
            a = temp;
            temp = y;
            y = (prevY-q*y) % m;
            prevY = temp;
      }
      if ( prevY < 0 ) prevY += m;  // may or may not be necessary
      return prevY; // this ends up being d
}

(Note that I renamed the function and its arguments to be a bit more descriptive of what the optimized version actually does.  Also note that I included an extra test to make sure that the return value cannot be negative even if your BigInteger library's % operator may return negative values.)


Configure Fail2ban for SSH and ldap

Question: I have a fail2ban Installation on two Centos7 machines, and I want to secure SSH with it (The SSH protecting seems active), set the allowed pw-fails to 5, and left the banntime at 3600 secs. I tried to login with my account with a false password numerous times, but I still haven't gotten banned. Am I missing something? Do I need to make a specific setup for ldap-users?
#
# WARNING: heavily refactored in 0.9.0 release.  Please review and
#          customize settings for your setup.
#
# Changes:  in most of the cases you should not modify this
#           file, but provide customizations in jail.local file,
#           or separate .conf files under jail.d/ directory, e.g.:
#
# HOW TO ACTIVATE JAILS:
#
# YOU SHOULD NOT MODIFY THIS FILE.
#
# It will probably be overwritten or improved in a distribution update.
#
# Provide customizations in a jail.local file or a jail.d/customisation.local.
# For example to change the default bantime for all jails and to enable the
# ssh-iptables jail the following (uncommented) would appear in the .local file.
# See man 5 jail.conf for details.
#
# [DEFAULT]
# bantime = 3600
#
# [sshd]
# enabled = true
#
# See jail.conf(5) man page for more information

# Comments: use '#' for comment lines and ';' (following a space) for inline comments

[INCLUDES]

#before = paths-distro.conf
before = paths-fedora.conf

# The DEFAULT allows a global definition of the options. They can be overridden
# in each jail afterwards.

[DEFAULT]

#
# MISCELLANEOUS OPTIONS
#

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8

# External command that will take an tagged arguments to ignore, e.g. <ip>,
# and return true if the IP is to be ignored. False otherwise.
#
# ignorecommand = /path/to/command <ip>
ignorecommand =

# "bantime" is the number of seconds that a host is banned.
bantime  = 3600

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 600

# "maxretry" is the number of failures before a host get banned.
maxretry = 5

# "backend" specifies the backend used to get files modification.
# Available options are "pyinotify", "gamin", "polling", "systemd" and "auto".
# This option can be overridden in each jail as well.
#
# pyinotify: requires pyinotify (a file alteration monitor) to be installed.
#              If pyinotify is not installed, Fail2ban will use auto.
# gamin:     requires Gamin (a file alteration monitor) to be installed.
#              If Gamin is not installed, Fail2ban will use auto.
# polling:   uses a polling algorithm which does not require external libraries.
# systemd:   uses systemd python library to access the systemd journal.
#              Specifying "logpath" is not valid for this backend.
#              See "journalmatch" in the jails associated filter config
# auto:      will try to use the following backends, in order:
#              pyinotify, gamin, polling.
#
# Note: if systemd backend is choses as the default but you enable a jail
#       for which logs are present only in its own log files, specify some other
#       backend for that jail (e.g. polling) and provide empty value for
#       journalmatch. See https://github.com/fail2ban/fail2ban/issues/959#issuecomment-74901200
backend = auto

# "usedns" specifies if jails should trust hostnames in logs,
#   warn when DNS lookups are performed, or ignore all hostnames in logs
#
# yes:   if a hostname is encountered, a DNS lookup will be performed.
# warn:  if a hostname is encountered, a DNS lookup will be performed,
#        but it will be logged as a warning.
# no:    if a hostname is encountered, will not be used for banning,
#        but it will be logged as info.
usedns = warn

# "logencoding" specifies the encoding of the log files handled by the jail
#   This is used to decode the lines from the log file.
#   Typical examples:  "ascii", "utf-8"
#
#   auto:   will use the system locale setting
logencoding = auto

# "enabled" enables the jails.
#  By default all jails are disabled, and it should stay this way.
#  Enable only relevant to your setup jails in your .local or jail.d/*.conf
#
# true:  jail will be enabled and log files will get monitored for changes
# false: jail is not enabled
enabled = false

# "filter" defines the filter to use by the jail.
#  By default jails have names matching their filter name
#
filter = %(__name__)s

#
# ACTIONS
#

# Some options used for actions

# Destination email address used solely for the interpolations in
# jail.{conf,local,d/*} configuration files.
destemail = root@localhost

# Sender email address used solely for some actions
sender = root@localhost

# E-mail action. Since 0.8.1 Fail2Ban uses sendmail MTA for the
# mailing. Change mta configuration parameter to mail if you want to
# revert to conventional 'mail'.
mta = sendmail

# Default protocol
protocol = tcp

# Specify chain where jumps would need to be added in iptables-* actions
chain = INPUT

# Ports to be banned
# Usually should be overridden in a particular jail
port = 0:65535

#
# Action shortcuts. To be used to define action parameter

# Default banning action (e.g. iptables, iptables-new,
# iptables-multiport, shorewall, etc) It is used to define
# action_* variables. Can be overridden globally or per
# section within jail.local file
banaction = iptables-multiport

# The simplest action to take: ban only
action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

# ban & send an e-mail with whois report to the destemail.
action_mw = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
            %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]

# ban & send an e-mail with whois report and relevant log lines
# to the destemail.
action_mwl = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
             %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]

# See the IMPORTANT note in action.d/xarf-login-attack for when to use this action
#
# ban & send a xarf e-mail to abuse contact of IP address and include relevant log lines
# to the destemail.
action_xarf = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
             xarf-login-attack[service=%(__name__)s, sender="%(sender)s", logpath=%(logpath)s, port="%(port)s"]

# ban IP on CloudFlare & send an e-mail with whois report and relevant log lines
# to the destemail.
action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
                %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]

# Report block via blocklist.de fail2ban reporting service API
#
# See the IMPORTANT note in action.d/blocklist_de.conf for when to
# use this action. Create a file jail.d/blocklist_de.local containing
# [Init]
# blocklist_de_apikey = {api key from registration]
#
action_blocklist_de  = blocklist_de[email="%(sender)s", service=%(filter)s, apikey="%(blocklist_de_apikey)s"]

# Report ban via badips.com, and use as blacklist
#
# See BadIPsAction docstring in config/action.d/badips.py for
# documentation for this action.
#
# NOTE: This action relies on banaction being present on start and therefore
# should be last action defined for a jail.
#
action_badips = badips.py[category="%(name)s", banaction="%(banaction)s"]

# Choose default action.  To change, just override value of 'action' with the
# interpolation to the chosen action shortcut (e.g.  action_mw, action_mwl, etc) in jail.local
# globally (section [DEFAULT]) or per specific section
action = %(action_)s

#
# JAILS
#

#
# SSH servers
#

[sshd]

port    = 2431
logpath = %(sshd_log)s

[sshd-ddos]
# This jail corresponds to the standard configuration in Fail2ban.
# The mail-whois action send a notification e-mail with a whois request
# in the body.
port    = ssh
logpath = %(sshd_log)s

[dropbear]

port     = ssh
logpath  = %(dropbear_log)s

[selinux-ssh]

port     = ssh
logpath  = %(auditd_log)s
maxretry = 5

#
# HTTP servers
#

[apache-auth]

port     = http,https
logpath  = %(apache_error_log)s

[apache-badbots]
# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
port     = http,https
logpath  = %(apache_access_log)s
bantime  = 172800
maxretry = 1

[apache-noscript]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 6

[apache-overflows]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-nohome]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-botsearch]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-fakegooglebot]

port     = http,https
logpath  = %(apache_access_log)s
maxretry = 1
ignorecommand = %(ignorecommands_dir)s/apache-fakegooglebot <ip>

[apache-modsecurity]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-shellshock]

port    = http,https
logpath = %(apache_error_log)s
maxretry = 1

[nginx-http-auth]

port    = http,https
logpath = %(nginx_error_log)s

[nginx-botsearch]

port     = http,https
logpath  = %(nginx_error_log)s
maxretry = 2

# Ban attackers that try to use PHP's URL-fopen() functionality
# through GET/POST variables. - Experimental, with more than a year
# of usage in production environments.

[php-url-fopen]

port    = http,https
logpath = %(nginx_access_log)s
          %(apache_access_log)s

[suhosin]

port    = http,https
logpath = %(suhosin_log)s

[lighttpd-auth]
# Same as above for Apache's mod_auth
# It catches wrong authentifications
port    = http,https
logpath = %(lighttpd_error_log)s

#
# Webmail and groupware servers
#

[roundcube-auth]

port     = http,https
logpath  = logpath = %(roundcube_errors_log)s

[openwebmail]

port     = http,https
logpath  = /var/log/openwebmail.log

[horde]

port     = http,https
logpath  = /var/log/horde/horde.log

[groupoffice]

port     = http,https
logpath  = /home/groupoffice/log/info.log

[sogo-auth]
# Monitor SOGo groupware server
# without proxy this would be:
# port    = 20000
port     = http,https
logpath  = /var/log/sogo/sogo.log

[tine20]

logpath  = /var/log/tine20/tine20.log
port     = http,https
maxretry = 5

#
# Web Applications
#
#

[drupal-auth]

port     = http,https
logpath  = %(syslog_daemon)s

[guacamole]

port     = http,https
logpath  = /var/log/tomcat*/catalina.out

[monit]
#Ban clients brute-forcing the monit gui login
filter   = monit
port = 2812
logpath  = /var/log/monit

[webmin-auth]

port    = 10000
logpath = %(syslog_authpriv)s

[froxlor-auth]

port    = http,https
logpath  = %(syslog_authpriv)s

#
# HTTP Proxy servers
#
#

[squid]

port     =  80,443,3128,8080
logpath = /var/log/squid/access.log

[3proxy]

port    = 3128
logpath = /var/log/3proxy.log

#
# FTP servers
#

[proftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(proftpd_log)s

[pure-ftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(pureftpd_log)s
maxretry = 6

[gssftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(syslog_daemon)s
maxretry = 6

[wuftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(wuftpd_log)s
maxretry = 6

[vsftpd]
# or overwrite it in jails.local to be
# logpath = %(syslog_authpriv)s
# if you want to rely on PAM failed login attempts
# vsftpd's failregex should match both of those formats
port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(vsftpd_log)s

#
# Mail servers
#

# ASSP SMTP Proxy Jail
[assp]

port     = smtp,465,submission
logpath  = /root/path/to/assp/logs/maillog.txt

[courier-smtp]

port     = smtp,465,submission
logpath  = %(syslog_mail)s

[postfix]

port     = smtp,465,submission
logpath  = %(postfix_log)s

[postfix-rbl]

port     = smtp,465,submission
logpath  = %(syslog_mail)s
maxretry = 1

[sendmail-auth]

port    = submission,465,smtp
logpath = %(syslog_mail)s

[sendmail-reject]

port     = smtp,465,submission
logpath  = %(syslog_mail)s

[qmail-rbl]

filter  = qmail
port    = smtp,465,submission
logpath = /service/qmail/log/main/current

# dovecot defaults to logging to the mail syslog facility
# but can be set by syslog_facility in the dovecot configuration.
[dovecot]

port    = pop3,pop3s,imap,imaps,submission,465,sieve
logpath = %(dovecot_log)s

[sieve]

port   = smtp,465,submission
logpath = %(dovecot_log)s

[solid-pop3d]

port    = pop3,pop3s
logpath = %(solidpop3d_log)s

[exim]

port   = smtp,465,submission
logpath = %(exim_main_log)s

[exim-spam]

port   = smtp,465,submission
logpath = %(exim_main_log)s

[kerio]

port    = imap,smtp,imaps,465
logpath = /opt/kerio/mailserver/store/logs/security.log

#
# Mail servers authenticators: might be used for smtp,ftp,imap servers, so
# all relevant ports get banned
#

[courier-auth]

port     = smtp,465,submission,imap3,imaps,pop3,pop3s
logpath  = %(syslog_mail)s

[postfix-sasl]

port     = smtp,465,submission,imap3,imaps,pop3,pop3s
# You might consider monitoring /var/log/mail.warn instead if you are
# running postfix since it would provide the same log lines at the
# "warn" level but overall at the smaller filesize.
logpath  = %(postfix_log)s

[perdition]

port   = imap3,imaps,pop3,pop3s
logpath = %(syslog_mail)s

[squirrelmail]

port = smtp,465,submission,imap2,imap3,imaps,pop3,pop3s,http,https,socks
logpath = /var/lib/squirrelmail/prefs/squirrelmail_access_log

[cyrus-imap]

port   = imap3,imaps
logpath = %(syslog_mail)s

[uwimap-auth]

port   = imap3,imaps
logpath = %(syslog_mail)s

#
#
# DNS servers
#

# !!! WARNING !!!
#   Since UDP is connection-less protocol, spoofing of IP and imitation
#   of illegal actions is way too simple.  Thus enabling of this filter
#   might provide an easy way for implementing a DoS against a chosen
#   victim. See
#    http://nion.modprobe.de/blog/archives/690-fail2ban-+-dns-fail.html
#   Please DO NOT USE this jail unless you know what you are doing.
#
# IMPORTANT: see filter.d/named-refused for instructions to enable logging
# This jail blocks UDP traffic for DNS requests.
# [named-refused-udp]
#
# filter   = named-refused
# port     = domain,953
# protocol = udp
# logpath  = /var/log/named/security.log

# IMPORTANT: see filter.d/named-refused for instructions to enable logging
# This jail blocks TCP traffic for DNS requests.

[named-refused]

port     = domain,953
logpath  = /var/log/named/security.log

[nsd]

port     = 53
action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
logpath = /var/log/nsd.log

#
# Miscellaneous
#

[asterisk]

port     = 5060,5061
action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
           %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s"]
logpath  = /var/log/asterisk/messages
maxretry = 10

[freeswitch]

port     = 5060,5061
action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
           %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s"]
logpath  = /var/log/freeswitch.log
maxretry = 10

# To log wrong MySQL access attempts add to /etc/my.cnf in [mysqld] or
# equivalent section:
# log-warning = 2
#
# for syslog (daemon facility)
# [mysqld_safe]
# syslog
#
# for own logfile
# [mysqld]
# log-error=/var/log/mysqld.log
[mysqld-auth]

port     = 3306
logpath  = %(mysql_log)s
maxretry = 5

# Jail for more extended banning of persistent abusers
# !!! WARNINGS !!!
# 1. Make sure that your loglevel specified in fail2ban.conf/.local
#    is not at DEBUG level -- which might then cause fail2ban to fall into
#    an infinite loop constantly feeding itself with non-informative lines
# 2. Increase dbpurgeage defined in fail2ban.conf to e.g. 648000 (7.5 days)
#    to maintain entries for failed logins for sufficient amount of time
[recidive]

logpath  = /var/log/fail2ban.log
banaction = iptables-allports
bantime  = 604800  ; 1 week
findtime = 86400   ; 1 day
maxretry = 5

# Generic filter for PAM. Has to be used with action which bans all
# ports such as iptables-allports, shorewall

[pam-generic]
# pam-generic filter can be customized to monitor specific subset of 'tty's
banaction = iptables-allports
logpath  = %(syslog_authpriv)s

[xinetd-fail]

banaction = iptables-multiport-log
logpath   = %(syslog_daemon)s
maxretry  = 2

# stunnel - need to set port for this
[stunnel]

logpath = /var/log/stunnel4/stunnel.log

[ejabberd-auth]

port    = 5222
logpath = /var/log/ejabberd/ejabberd.log

[counter-strike]

logpath = /opt/cstrike/logs/L[0-9]*.log
# Firewall: http://www.cstrike-planet.com/faq/6
tcpport = 27030,27031,27032,27033,27034,27035,27036,27037,27038,27039
udpport = 1200,27000,27001,27002,27003,27004,27005,27006,27007,27008,27009,27010,27011,27012,27013,27014,27015
action  = %(banaction)s[name=%(__name__)s-tcp, port="%(tcpport)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(udpport)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]

# consider low maxretry and a long bantime
# nobody except your own Nagios server should ever probe nrpe
[nagios]

enabled  = false
logpath  = %(syslog_daemon)s     ; nrpe.cfg may define a different log_facility
maxretry = 1

[oracleims]
# see "oracleims" filter file for configuration requirement for Oracle IMS v6 and above
enabled = false
logpath = /opt/sun/comms/messaging64/log/mail.log_current
maxretry = 6
banaction = iptables-allports

[directadmin]
enabled = false
logpath = /var/log/directadmin/login.log
port = 2222

[portsentry]
enabled  = false
logpath  = /var/lib/portsentry/portsentry.history
maxretry = 1

[pass2allow-ftp]
# this pass2allow example allows FTP traffic after successful HTTP authentication
port         = ftp,ftp-data,ftps,ftps-data
# knocking_url variable must be overridden to some secret value in filter.d/apache-pass.local
filter       = apache-pass
# access log of the website with HTTP auth
logpath      = %(apache_access_log)s
blocktype    = RETURN
returntype   = DROP
bantime      = 3600
maxretry     = 1
findtime     = 1

Edit 1:
So I tried everything which was currently mentioned in the below comments, but it did not work for me. I tried to put the necessary configuration into a jail.local and use it with a default jail.conf. That did not work. The "fakeattackers" werent even put in the jail of fail2ban.
With the current configuration in the jail.conf
[sshd]
enabled = true
port    = 1234
logpath = %(sshd_log)s

[sshd-ddos]
# This jail corresponds to the standard configuration in Fail2ban.
# The mail-whois action send a notification e-mail with a whois request
# in the body.
port    = ssh
logpath = %(sshd_log)s

The attackers are sent in the jail :
fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed: 1
|  `- File list:    /var/log/secure
`- Actions
   |- Currently banned: 1
   |- Total banned: 1
   `- Banned IP list:   192.168.1.1

But even so, there is no changes in the iptables itself:
iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootpc

The banned attackers can still login without problems (while banned).
Does someone know how to fix this?
Comment: If you have SELinux enabled, see if it is blocking fail2ban from reading log files: run `journalctl -lfu fail2ban` and check the output.
Comment: SELinux is not enabled
Comment: Is this your `jail.local` or `jail.conf` file? How have you installed fail2ban?
Comment: its the jail.conf I accually have two machines here. On one I installed it via repository and yum install. For the second machine It somehow was not able to install it via Repo (while there using the same). So I downloaded the packages and installed it with rpm -i . The current installation which we are currently talking about is the yum installed one.
Comment: if you read the beginning of the file, you will read, you should make a copy of jail.conf file. Make copy of it and name it jail.local and customize it according to your need. You don't need all the jails, so enable them, those you need and see if it works. There are many how-to's are available online, check one of those.
Answer: So I found out what the problem was. Fail2ban in Centos defaultly uses the firewalld as tool to block IPs. But firewalld is turned off, so the command which fail2ban executed in back, did nothing.
To get this to work with iptables, one needs to set the necesarry command in the jail.local.
action   = iptables[name=SSH, port=ssh, protocol=tcp]

And it works.
Thanks for the help :)
Answer: It seems that sshd jail is note enabled in your jail.conf
You should uncomment both lines to enable it
# [sshd]
# enabled = true

If this do not solve the issue post in comment the results of this commands:
fail2ban-client status
fail2ban-client status ssh

Thanks.
Comment: bump* Does nobody have an idea ? :(
Comment: [sshd]

enabled = true
port    = ssh
logpath = %(sshd_log)s
d It is enabled, you just checked default comments. The accual section is more one the bottom.
Comment: fail2banclient status : fail2ban-client status
Status
|- Number of jail: 1
`- Jail list: sshd
fail2ban-client status ssh(d?)fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 1
|  |- Total failed: 13
|  `- File list: /var/log/secure
`- Actions
   |- Currently banned: 1
   |- Total banned: 1
   `- Banned IP list: 192.168.5.2   It is actually doing something, but the ban is not "executed" the IP maybe listed there but I can still login. Do I need a certain firewall like firewalld or shorewall for this, or should fail2ban be able to alter iptables on its own?

Configure Fail2ban for SSH and ldap


Characteristic scale degrees

Question: Came across an article at openMsicTheory.com about harmonic function. Below excerpt was about finding function of a chord

Each of the three harmonic functions — tonic (T), subdominant (S), and dominant (D) — have characteristic scale degrees. Tonic's characteristic scale degrees are 1, 3, 5, 6, and 7. Subdominant's characteristic scale degrees are 1, 2, 3, 4, and 6. Dominant's characteristic scale degrees are 2, 4, 5, 6, and 7.

Can anyone advise what the writer is attempting to say?
Answer: It's saying that each harmonic function is built up using different notes of the reference key, or tonic scale. Those are the notes that are used to build harmony that satisfies those functions. So, if you want to imply an harmonic function, or find the function of a set of notes, you'll be using those notes in some form and extent.
In other words, it's describing those three functions using pitches, notes. Each function is different, so it is described by different notes.
In the key of C we have:

1 2 3 4 5 6 7
C D E F G A B

That makes our tonic, or tonal center, or "home", to be C major. The subdominant is in the 4th scale degree, so F major, and the dominant is at the 5th scale degree, so G major. (For the chord qualities, we are just stacking thirds)

The tonic chord is C major, Cmaj7 chord is C E G B, which are the scale degrees 1, 3, 5, 7. Article mentions 1, 3, 5, 6, and 7.
The subdominant is F major, Fmaj7 chord is F A C E, which are the scale degrees 1, 3, 4, 6. Article mentions 1, 2, 3, 4, and 6.
The dominant is G major, G7 chord is G B D F, which are the scale degrees 2, 4, 5, 7. Article mentions 2, 4, 5, 6, and 7.

So, each harmonic function is characterized, described, achieved, by using those notes. Which makes sense, since they are built using mainly the diatonic chords of that scale degree (1 for tonic, 4 for subdominant, 5 for dominant).
On each function we are still missing one note, though: 6 for tonic, 2 for subdominant, 6 for dominant. According to the article, the 6 in tonic is an "associate", 2 in subdominant is an "associate", and 6 in dominant is a "dissonance". It also classifies all notes, but doesn't seem to explain exactly what it means by this categorization.
Comment: I can see why some of the dissonances are what they are (one degree away from a "trigger" or "associate") and some of the associates (5 is the 5 of the 1, 1 is the 5 of the 4, 2 is the 5 of the 5; none are "needed" to establish function), but I don't understand the extra 6 in the tonic or 2 in the subdominant. I also don't see why they collapse this all into "characteristic" degrees.
Comment: I'm putting this here since the conundrum of categorization wasn't in the original question, but the extras come from taking the full set of the harmonic functions rather than just the I, IV and V. I don't know why the 4 is dissonant for D rather than an associate, but I'd guess it's because it's a step away from the 5, which has a strong character for D.
Answer: I guess, he means the following: (I will do it on the example of Tonic characteristic scales)
1, 3 and 5 are obviously characteristic degrees of a tonic, right? The other two triads, namely on the 6th degree (6, 1, 3) and on the 3rd degree (3, 5, 7) have an intersection of two notes each with the tonic triad (1, 3, 5), and apart from that each one has an additional degree 6 and 7. So each of these notes - 1, 3, 5, 6, 7 may point at at chords with a "big" intersection with tonic (triads on 6, 3 and 1 - the tonic itself).
Answer: It looks like the author is drawing this from Riemann's functional theory. In it, each of the triads built with scale degrees has a function: T, D, or S. I, III and VI are all T, II and IV are S, and V and VII are D.

T: I, III, and VI have (1, 3, 5), (3, 5, 7), and (6, 1, 3), which gives the set (1, 3, 5, 6, 7).
S: II and IV have (2, 4, 6) and (4, 6, 1), for (1, 2, 4, 6). The 3 comes from IV7 (4, 6, 1, 3) for the set (1, 2, 3, 4, 6).
D: V and VII7 have (5, 7, 2) and (7, 2, 4, 6), which gives the set (2, 4, 5, 6, 7).

Lyd is right in that 7s are involved in the determination and NickQuant is right in that the intersections are important, but they are not the whole story as that would include the V in T, which cannot be true in functional harmony. VI is included instead, as it is the relative minor to I and can have a tonic function.
There is one oddity about the 7s. For almost all of the triads, you can take them as 7s and it doesn't change anything; they just share more of the same notes. The oddity here is III7, which would include the 2. The discrepancy is due to the fact that the III has a dual nature in the theory, where it can be dominant (as it is dominant to the VI) or tonic. In fact, III7 is very dominant in that it contains the V, so while III can be T, III7 is D.
Comment: I would include VI in the subdominant category as well. That's why 3 is listed, since it's the 5th of the VI chord.
Comment: @ibonyun yeah, it fits in both, but the authors on that site chose to make it tonic rather than share it. It might be due to background: I think jazz wonks tend to analyze VI as a tonic. Maybe they just wanted to keep it simple.
Answer:
Each of the three harmonic functions — tonic (T), subdominant (S), and dominant (D) — have characteristic scale degrees. Tonic's characteristic scale degrees are 1, 3, 5, 6, and 7. Subdominant's characteristic scale degrees are 1, 2, 3, 4, and 6. Dominant's characteristic scale degrees are 2, 4, 5, 6, and 7.

If those scale degrees (indicated by ^) are re-arranged as...
subdominant : ^2 ^4 ^6 ^1 ^3
dominant: ^5 ^7 ^2 ^4 ^6
tonic : ^1 ^3 ^5 ^6 ^7

...or as letters in C major...
subdominant : D F A C E
dominant: G B D F A
tonic : C E G A B

...you get...
subdominant : ii9
dominant : V9
tonic : I6 or IM7

That is very much a tonal view from the world of jazz.
In classical harmony ^3 isn't part of the subdominant, and the tonic is only ^1 ^3 ^5.
The basic idea is the primary chords - the tonic, dominant, and subdominant - are built of thirds above the roots ^1 ^5 ^4. In jazz those thirds are stacked up to make ninth chord. In the classical world the primary chord are triads with seventh chord being used under somewhat strict voice leading procedures.
The description on the linked page may not be clear, because it doesn't make a clear connection between the 'characteristic scale degrees' and their connection to the chord tones.
A final point seems worth mentioning: the subdominant or pre-dominant function can be fulfilled by either the subdominant chord IV (scale degrees ^4 ^6 ^1) or the supertonic chord ii7 - often inverted as ii6/5 - (scale degrees ^2 ^4 ^6 ^1.)

Characteristic scale degrees


How to Programmatically add javascript dynamically from a custom module to modify a popup calendar property

Question: I need to add a JavaScript code to my custom module, to modify the property in a date popup calendar and make it dynamic.
According to the documentation of the date popup calendar, the syntax for modifying the MaxDate is:
$( ".selector" ).datepicker({
  maxDate: "+1m"
});

I have to write "+1m" dynamically because it may be "+2m", "+3m", etc, it is variable.
How can I write a JavaScript code in my custom module and effect the popup calendar? I am wondering if I use script tag and put the JavaScript code inside it.
In case, how can I do it by js?
Answer: You can attach a script to a form using the form's #attached attribute. You can do this in hook_form_alter as follows:
function hook_form_alter(&$form, &$form_state, $form_id)
{
  if($form_id == 'targeted_form')
  {
    $form['#attached']['js'][] = array
    (
      'type' => 'file',
      'data' => drupal_get_path('module', 'my_module') . '/scripts.js',
    );
  }
}

If you need to pass values from PHP to the form, you can attach values using Drupal.settings:
function hook_form_alter(&$form, &$form_state, $form_id)
{
  if($form_id == 'targeted_form')
  {
    $form['#attached']['js'][] = array
    (
      'type' => 'setting',
      'data' => array('myModule' => array('someSetting' => 'some value')),
    );
  }
}

Now the value will be available in your form at:
Drupal.settings.myModule.someSetting

Combine these two techniques to attach a form and pass it some settings.


Spring Data MongoDB can't use @Transactional of my Repository Test: Failed to retrieve PlatformTransactionManager for @Transactional test

Question: I am starting a simple spring web service that uses Spring Data MongoDB. I am used to use Spring Data with MySQL and here using @Transactional in my test classes works fine.
However in this case I receive the following error message:
Failed to retrieve PlatformTransactionManager for @Transactional test: [DefaultTestContext@39277750 testClass = ItemRepositoryTest, testInstance = com.ruckpack.equipmentservice.repository.ItemRepositoryTest@73bbbc73, testMethod = should test@ItemRepositoryTest, testException = [null], mergedContextConfiguration = [WebMergedContextConfiguration@2c217989 testClass = ItemRepositoryTest, locations = '{}', classes = '{class com.ruckpack.equipmentservice.EquipmentServiceApplication}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{org.springframework.boot.test.context.SpringBootTestContextBootstrapper=true}', contextCustomizers = set[org.springframework.boot.test.autoconfigure.actuate.metrics.MetricsExportContextCustomizerFactory$DisableMetricExportContextCustomizer@15112c86, org.springframework.boot.test.autoconfigure.properties.PropertyMappingContextCustomizer@0, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverContextCustomizerFactory$Customizer@3400f6a3, org.springframework.boot.test.context.filter.ExcludeFilterContextCustomizer@39669485, org.springframework.boot.test.json.DuplicateJsonObjectContextCustomizerFactory$DuplicateJsonObjectContextCustomizer@dfa43ef, org.springframework.boot.test.mock.mockito.MockitoContextCustomizer@0, org.springframework.boot.test.web.client.TestRestTemplateContextCustomizer@103a4d41, org.springframework.boot.test.context.SpringBootTestArgs@1, org.springframework.boot.test.context.SpringBootTestWebEnvironment@34fe2a08], resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.boot.test.context.SpringBootContextLoader', parent = [null]], attributes = map['org.springframework.test.context.web.ServletTestExecutionListener.activateListener' -> true, 'org.springframework.test.context.web.ServletTestExecutionListener.populatedRequestContextHolder' -> true, 'org.springframework.test.context.web.ServletTestExecutionListener.resetRequestContextHolder' -> true, 'org.springframework.test.context.event.ApplicationEventsTestExecutionListener.recordApplicationEvents' -> false]]
java.lang.IllegalStateException: Failed to retrieve PlatformTransactionManager for @Transactional test: [DefaultTestContext@39277750 testClass = ItemRepositoryTest, testInstance = com.ruckpack.equipmentservice.repository.ItemRepositoryTest@73bbbc73, testMethod = should test@ItemRepositoryTest, testException = [null], mergedContextConfiguration = [WebMergedContextConfiguration@2c217989 testClass = ItemRepositoryTest, locations = '{}', classes = '{class com.ruckpack.equipmentservice.EquipmentServiceApplication}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{org.springframework.boot.test.context.SpringBootTestContextBootstrapper=true}', contextCustomizers = set[org.springframework.boot.test.autoconfigure.actuate.metrics.MetricsExportContextCustomizerFactory$DisableMetricExportContextCustomizer@15112c86, org.springframework.boot.test.autoconfigure.properties.PropertyMappingContextCustomizer@0, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverContextCustomizerFactory$Customizer@3400f6a3, org.springframework.boot.test.context.filter.ExcludeFilterContextCustomizer@39669485, org.springframework.boot.test.json.DuplicateJsonObjectContextCustomizerFactory$DuplicateJsonObjectContextCustomizer@dfa43ef, org.springframework.boot.test.mock.mockito.MockitoContextCustomizer@0, org.springframework.boot.test.web.client.TestRestTemplateContextCustomizer@103a4d41, org.springframework.boot.test.context.SpringBootTestArgs@1, org.springframework.boot.test.context.SpringBootTestWebEnvironment@34fe2a08], resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.boot.test.context.SpringBootContextLoader', parent = [null]], attributes = map['org.springframework.test.context.web.ServletTestExecutionListener.activateListener' -> true, 'org.springframework.test.context.web.ServletTestExecutionListener.populatedRequestContextHolder' -> true, 'org.springframework.test.context.web.ServletTestExecutionListener.resetRequestContextHolder' -> true, 'org.springframework.test.context.event.ApplicationEventsTestExecutionListener.recordApplication

Now this is my test class
@SpringBootTest
@Transactional
internal class ItemRepositoryTest @Autowired constructor(
    private val itemRepository: ItemRepository
) {
    @Test
    fun `should test`() {
        val expected = Item("001", "Meindl", "SuperBoot 300")
        itemRepository.save(expected)
        val actual = itemRepository.findById(expected.id)

        assertThat(actual).isPresent
        assertThat(actual.get()).isEqualTo(expected)
    }
}

@Repository
interface ItemRepository: MongoRepository<Item, String>

Any my application class looks like this:
@SpringBootApplication
class EquipmentServiceApplication

fun main(args: Array<String>) {
    runApplication<EquipmentServiceApplication>(*args)
}

I saw this tutorial: https://www.baeldung.com/spring-data-mongodb-transactions where the usage was done without any special configuration. That is why I am confused it does not work.
Here is my build.gradle.kt for completeness
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

plugins {
    id("org.springframework.boot") version "2.4.2"
    id("io.spring.dependency-management") version "1.0.11.RELEASE"
    kotlin("jvm") version "1.4.30"
    kotlin("plugin.spring") version "1.4.30"
}

group = "com.ruckpack"
version = "0.0.1"
java.sourceCompatibility = JavaVersion.VERSION_11

repositories {
    mavenCentral()
    maven { url = uri("https://repo.spring.io/milestone") }
}

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-data-mongodb")
    implementation("org.springframework.boot:spring-boot-starter-security")
    implementation("org.springframework.boot:spring-boot-starter-web")
    implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
    implementation("org.jetbrains.kotlin:kotlin-reflect")
    implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")

    testImplementation("org.springframework.boot:spring-boot-starter-test")
    testImplementation("de.flapdoodle.embed:de.flapdoodle.embed.mongo")
    testImplementation("org.springframework.security:spring-security-test")
}

tasks.withType<KotlinCompile> {
    kotlinOptions {
        freeCompilerArgs = listOf("-Xjsr305=strict")
        jvmTarget = "11"
    }
}

tasks.withType<Test> {
    useJUnitPlatform()
}

Answer: MongoDB versions 4 and above supports transactions. But are disabled by default in Spring Data MongoDB, you should define MongoTransactionManager explicitly.
Please read: https://docs.spring.io/spring-data/mongodb/docs/2.1.0.RC2/reference/html/#mongo.transactions
And
https://www.baeldung.com/spring-data-mongodb-transactions also says
"Note that we need to register MongoTransactionManager in our configuration to enable native MongoDB transactions as they are disabled by default."
Not sure about the syntax in Kotlin, for Java is:
@Bean
MongoTransactionManager txManager(MongoDbFactory mongoDbFactory) {
    return new MongoTransactionManager(mongoDbFactory);
}

Comment: is there way via application,yaml ?
Comment: Hey xetra11, any chance you have found a way to configure it via application.yml at the end?
Answer: The Kolin version of the above would be something like:
@Bean
fun txManager(mongoDbFactory: MongoDbFactory) : MongoTransactionManager = MongoTransactionManager(mongoDbFactory)


React router Link doesn't redirect

Question: Please see sandbox.
I'm having this code:
    <Router history={createBrowserHistory()}>
      <Switch>
        <Route path="/items/:itemId">
          <Item />
        </Route>
        <Route exact path="/items">
          <Items />
        </Route>
        <Redirect to="/items" />
      </Switch>
    </Router>

Expecting /items to be default route. Items is loading links for available items:
  const { url } = useRouteMatch();

  return (

Items

      {items.map((item) => (
        <Link to={`${url}/${item}`}>{item}</Link>
      ))}

  );

Once an item link is clicked, I'm expecting the url to have the item in it, and to render Item.
url is /items, so everything should work out for Switch to render:
        <Route path="/items/:itemId">
          <Item />
        </Route>

However, it's staying at /items, and not even rendering Items component anymore. No error is shown. Any idea why this is happening?
Answer: For router you are missing an exact props which is really important for your case
<Route exact path="/items/:itemId" />

and inside your Item component you misspelled the param name, because in router it's itemId where the component has a different one item, so fixing it will make the param accessible(they should be exact match)
export const Item = () => {
  const { itemId } = useParams();

  return (

Item

      {itemId}

  );
};

Comment: Actually, it doesn't work, neither the other answer. Please see the updated sandbox.
Comment: It does work for me, but notice that you have to put the `/items` in URL and after navigation `codesandbox` sometimes doesn't load the new page correctly, so unfortunately you have to hit the refresh button manually (locally you wouldn't have such an issue)
Comment: For me what happens (locally as well) is that when on `/items`, clicking on an item it does change the url to `/items/:itemId`, but doesn't render the component, only if I reload the page.
Comment: I'll test locally then and find out what's going on!
Comment: it did work for me after replacing `Router` with `` imported from `react-router-dom`, I don't know why `Router` didn't work despite I've used it before in some application and it didn't has an issue
Answer: That because when it reaches the /items/:itemId Page the component get redirect back to /items , So the solution to this is just remove the redirect component
  <Switch>
    <Route path="/items/:itemId">
      <Item />
    </Route>
    <Route exact path="/items">
      <Items />
    </Route>
  </Switch>

Also there few other mistake which i saw in your code.In Item.jsx , your using
  const { item } = useParams();

This isn't going to the render the anything since it is undefined , that is because your are passing the /:itemId in Route. So try this instead
  const { itemId:item } = useParams();

here we are renaming the itemId to item inorder to accessable down in the code.
Comment: Actually, it doesn't work, neither the other answer. Please see the updated sandbox.

React router Link doesn't redirect


How to disable cache for some subpath nginx

Question: I try to create a microcaching in Nginx by configuring that only valid for the subpath of some URI that will work.
For example :

https://example.com/path/anotherpath/view/ --> Will be cached
https://example.com/path/anotherpath/view/subview --> not to be cached

I have tried this but it does not come out with luck
if ($uri ~* "/path/anotherpath/[^]+/[^]+") {set $no_cache "1";}
And this:
location ~* "/path/anotherpath/[^]+/[\d]" {set $no_cache "1";}

If you guys have any suggestions, they will be much appreciated.
Comment: Please add the output of `nginx -T` to the question so we can see the full configuration. Without knowledge of all the `location` blocks, the question cannot be answered.
Comment: here :
nginx: configuration file /etc/nginx/nginx.conf test is successful
Comment: That is output of `nginx -t`, not `nginx -T`.
Answer: I manage to fix it by this config :
if ($uri ~* "/path/subpath/(.*)/(.*)$" ) { set  $no_cache "1";}


PHP Syntax error in file when called from Cron Tab

Question: Edited
As I guessed, it´s a php version issue:
From command line:
php -v
...
PHP 5.2.17 (cli) (built: Mar  1 2012 16:36:14)
Copyright (c) 1997-2010 The PHP Group
...

phpinfo in a web page:

PHP Version 5.4.20

As far as I know, I won´t be able to change the configuration of the command line version, as it is in a shared server (dreamhost, if it helps someone to a better response).
I really need to use my php 5.3 dependant libraries, replicating some processes from scratch would take me many days of work, and I have to develop this in 2 weeks.
Does someone knows any trick to run a higher version of php than defined for command line?

I´m having a problem with a Cron Job.
In my project, almost every page includes a file that contains the
  following lines of code:
....
    $called_class = get_called_class();
    $this->error_code = 0;
    $this->error_string = "";

    if ( is_array($parametros) || is_object($parametros) ){
        foreach ( $parametros as $key => $value ) {
          if (!$called_class::isValidKey ( $key ) && !$called_class::isExternalKey( $key ) )
              continue;
          $this->$key = $value;
        }
    }
...

And never showed me any error.
But when I set a Cron Job to run a script that includes the same file,
  it sends me the following message to my email:

Parse error: syntax error, unexpected T_PAAMAYIM_NEKUDOTAYIM in
    /xxx/xxx/xxx/modelo.php on line 22

What refers to this line:
if (!$called_class::isValidKey ( $key ) && !$called_class::isExternalKey( $key ) )

I´m guessing that php version used in command line differs from the
  one used from the services that shows the page, or whatever it´s
  called.
I´d appreciate if someone could point me the problem, and suggest me a
  solution. Btw, I know that I could use a separate file using
  instructions compatible with previous versions of php for the cron
  job, but I´d like to avoid that.
Comment: Yes, what you're suspecting seems to be the case. Have you checked `sudo php -v` from the command line?
Comment: It´s a shared server, and I do not have permissions to run sudo command, thank you anyway
Answer: The problem is likely caused by running the script in the wrong working directory, try changing your crontab command to:
cd path/to/script/directory && php script.php

Comment: Although I´m pretty sure that it´s not a problem with the working directory, I tried to do as you said from the command line with the same error.

Is there usually any place where I can change command line php version? If so, how is it usually called?
Answer: Ok... seems to be so easy that I feel a bit ashamed.
changed the command:

/usr/local/bin/php

for

/usr/local/php54/bin/php

That´s for my dreamhost server, but may be similar in other shared servers.
Thank you for your answers.
Comment: +1 for figuring it out yourself when noone could really help!


S3 redirect to sub-directory

Question: I have site served from S3 with Nginx with following Nginx configuration.
server {
    listen         80 default_server;
    server_name    localhost;
    keepalive_timeout 70;
    gzip on;
    gzip_disable "msie6";
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_types text/plain text/css application/json application/x-javascript application/javascript text/xml application/xml application/xml+rss text/javascript;

    location / {
         proxy_pass http://my-bucket.s3-website-us-west-2.amazonaws.com;
         expires 30d;
    }

At present whenever I build new version, I just delete target bucket contain and upload new frontend files to it.
Since I am deleting bucket contain, there is no way I can go back to previous version of frontend even versioning is enabled on bucket. So want to upload new frontend files into version dir (for example 15) in S3 bucket and then setup a redirect from http://my-bucket.s3-website-us-west-2.amazonaws.com/latest to http://my-bucket.s3-website-us-west-2.amazonaws.com/15
anyone knows how this can be done ?
Comment: Whether you can do this, or how to do it, will depend on how you are using Nginx together with this bucket. Please explain how Nginx is being used.
Comment: Updated with nginx config.
Comment: @roy if any answer solved your problem, please make sure to upvote and accept.  If you perform no action, the full amount of the bounty will be lost.  If no answer is sufficient, please explain why both answers are deficient. Thanks!
Answer: There are multiple ways to do this:

The easiest may be through a symbolic link, provided that your environment allows that.
ln -fhs ./15 ./latest

Another option is an explicit external redirect issued to the user, where the user would see the new URL; this has a benefit in that multiple versions could be accessed at the same time without any sort of synchronisation issues, for example, if a client decides to do a partial download, everything should still be handy, because they'll most likely be doing the partial download on the actual target, not the /latest shortcut.
location /latest {
    rewrite ^/latest(.*)    /15$1   redirect;
}

The final option is an internal redirect within nginx; this is usually called URL masquerading in some third-party applications; this may or may not be recommended, depending on requirements; an obvious deficiency would be with partial downloads, where a resume of a big download may result in corrupted files:
location /latest {
    rewrite ^/latest(.*)    /15$1   last;
}

References:

http://nginx.org/r/location
http://nginx.org/r/rewrite
Answer: One of the simple ways to handle this situation is using variables. You can easily import a file to set the current latest version. You will need to reload your nginx config when you update the version with this method.

Create a simple configuration file for setting the latest version

# /path/to/latest.conf
set $latest 15;

Import your latest configuration in the server block, and add a location to proxy to the latest version.

server {
    listen         80 default_server;
    server_name    localhost;

    # SET LATEST
    import /path/to/latest.conf;

    location / {
         proxy_pass http://s3host;
         expires 30d;
    }

    # Note the / at the end of the location and the proxy_pass directive
    # This will strip the "/latest/" part of the request uri, and pass the
    # rest like so: /$version/$remaining_request_uri
    location /latest/ {
        proxy_pass http://s3host/$latest/;
         expires 30d;
    }

    ...

}

Another way to do this dynamically would be to use lua to script this behavior. That is a little more involved though, so I will not get into that for this answer.

S3 redirect to sub-directory


Expected Number of Loops from Nesting Closed Function

Question: Suppose we have a finite set, $S$, and a one-to-one function such that $f:S\to S$. We define a loop as an instance of $f(...f(a)...)=a$ for some $k$ iterations of nesting the function within itself. We know that for each set, at least $1$ loop must exist, and that no loops intersect since it's a one-to-one function. My question is, how many loops can we expect if the size of $S$ is $n$ (i.e. try to find the explicit form of $L(n)$)? What I know so far is that $L(1)=1$ and probably $\lim_{n\to\infty}L(n)=\infty$. Also, I wrote a little simulation that tested $n$ from $1$ to $100$ for $10,000$ trials each and it appears the relationship is somewhat logarithmic. The results are below. Any help is appreciated. Thanks!
<code>1: 1.0
2: 1.5045
3: 1.8184
4: 2.0929
5: 2.2747
6: 2.4513
7: 2.5833
8: 2.721
9: 2.8445
10: 2.9137
11: 3.0369
12: 3.0975
13: 3.1871
14: 3.2604
15: 3.3159
16: 3.389
17: 3.4196
18: 3.4808
19: 3.5732
20: 3.5994
21: 3.6848
22: 3.7041
23: 3.7628
24: 3.7794
25: 3.8164
26: 3.867
27: 3.8887
28: 3.93
29: 3.9738
30: 4.0105
31: 4.0483
32: 4.0503
33: 4.077
34: 4.1253
35: 4.1476
36: 4.1869
37: 4.1906
38: 4.2298
39: 4.2324
40: 4.267
41: 4.3302
42: 4.3197
43: 4.3616
44: 4.4119
45: 4.3936
46: 4.3861
47: 4.4259
48: 4.4645
49: 4.4597
50: 4.4999
51: 4.513
52: 4.5477
53: 4.5665
54: 4.5713
55: 4.5841
56: 4.6307
57: 4.6295
58: 4.6588
59: 4.7008
60: 4.6451
61: 4.7015
62: 4.6883
63: 4.7326
64: 4.7167
65: 4.7513
66: 4.7705
67: 4.7709
68: 4.7904
69: 4.8286
70: 4.8387
71: 4.8824
72: 4.846
73: 4.8466
74: 4.8527
75: 4.9124
76: 4.9098
77: 4.8895
78: 4.9421
79: 4.9558
80: 4.9605
81: 4.9909
82: 4.9901
83: 5.0157
84: 5.0223
85: 5.03
86: 5.0407
87: 5.0841
88: 5.0697
89: 5.062
90: 5.0585
91: 5.1167
92: 5.0829
93: 5.1316
94: 5.1422
95: 5.1148
96: 5.1581
97: 5.1524
98: 5.1512
99: 5.1728
100: 5.2072
</code>
Comment: What is $L(n)$?
Comment: So we have $S_n \to S_n$. And we want to determine average number of elements in loop (cycle)?
Comment: @Gae.S. $L(n)$ is the expected number of loops given a set of size $n$. The table I provided shows the mapping of $n$ to $L(n)$. I'm trying to find an explicit form of $L(n)$.
Comment: @openspace We can do that too. I'm looking for $L(n)$, but the average number of elements in a cycle can be expressed as $n/L(n)$.
Answer: The reason $L(n)$ appears to logarithmic is because
$$
L(n)=\sum_{k=1}^n \frac{1}{k}
$$
for all positive integers $n$.

Explanation:

For each positive integer $n$, let $L(n)$ be the expected number of cycles (including singleton cycles) for a random element of $S_n$.

For convenience, we set $L(0)=0$.

For a random element of $S_n$, the length of the cycle which contains $1$ is one of the values $1,...,n$, with all $n$ values equally likely.

Thus for each $k\in\{0,...,n-1\}$, the case where there are exactly $k$ elements of $\{2,...,n\}$ which are not in the cycle containing $1$ contributes
$$
{\small{\frac{1}{n}}}{\,\cdot\,}(1+L(k))
$$
 to the expected number of cycles.

If follows that for all positive integers $n$ we have
$$
L(n)
=
\sum_{k=0}^{n-1} \left({\small{\frac{1}{n}}}{\,\cdot\,}(1+L(k))\right)
=
1+{\small{\frac{1}{n}}}\sum_{k=0}^{n-1} L(k)
$$
Then we get
\begin{align*}
L(n+1)
&=
1+{\small{\frac{1}{n+1}}}\sum_{k=0}^n L(k)
\\[4pt]
&=
1+{\small{\frac{1}{n+1}}}\left(L(n)+\sum_{k=0}^{n-1} L(k)\right)
\\[4pt]
&=
1+{\small{\frac{1}{n+1}}}\Bigl(L(n)+\bigl(n(L(n)-1)\bigl)\Bigr)
\\[4pt]
&=
1+{\small{\frac{1}{n+1}}}\Bigl((n+1)L(n)-n)\bigl)\Bigr)
\\[4pt]
&=
L(n)+\frac{1}{n+1}
\\[4pt]
\end{align*}
hence, noting that $L(1)=1$, it follows that
$$
L(n)=\sum_{k=1}^n \frac{1}{k}
\qquad\qquad\qquad\qquad\qquad\;
$$
for all positive integers $n$, as claimed.
Comment: Wow, that's actually pretty cool! How did you derive it?
Comment: I'll edit in an explanation . . .


Opening a ZIP file, scanning a CSV file inside it and transferring certain content to another CSV file

Question: The following code views the CSV file inside the ZIP folder and then using the concept of index setting, tries to extract the row containing the entry "HINDUNILVR" in the column "SYMBOL", of each and every CSV file (this code segment is inside another loop) , for a series of dates (read ZIP FILES). However, there is some problem in the code, as the there is no output in my "stock.csv" file that is present in C: . It is empty after running this code.
Expected Output File Description:
A CSV file named "stock.csv" containing the first column as SYMBOL, with HINDUNILVR as the entry in each cell of that column, followed by the many prices and details associated with the stock for each and every day, in the same row.
Python Code:
if(int(str(sdate.day))<10):

     df1 = pd.read_csv(zf.open("cm" + "0" + str(sdate.day) + MonthCode(sdate.month) + str(sdate.year) + "bhav.csv"))

   else:

     df1 = pd.read_csv(zf.open("cm" + str(sdate.day) + MonthCode(sdate.month) + str(sdate.year) + "bhav.csv"))

   df2 = df1.set_index("SYMBOL", drop = False)

   saved_row = df2.loc["HINDUNILVR",:]

      with open_file('C:\stock.csv','a') as f:
        saved_row.to_csv(f,header=False)

Comment: Is your script running with right privileges to write to c drive?
Comment: if don't have write privileges code will throw exception
Comment: i have changed the code to:

 with open_file(r'''C:\Users\Mukund\Documents\stock.csv''','a') as f:
      saved_row.to_csv(f,mode='a',header=False) , still it is an empty output, turns out I didn't have privileges to write to C drive
Comment: if u didn't have privileges to write to C:\ Drive didn't the code throw exception with permission denied error. Can you post the sample file.
Comment: There was no exception thrown. What do you mean by sample file? The file of expected output?
Comment: This is how the expected output should look, (just one line has been done by me for sample):                                                                                https://drive.google.com/file/d/1dB7II4ItF-p4lwthEwoHYkL2g5Co6X-v/view?usp=sharing
Comment: zip file which contains the source CSV.
Comment: https://drive.google.com/file/d/1i5xlCLAUNWHjzBvjBETza3W_r7aa28k4/view?usp=sharing                                                                                this is one of the zip files ( the one corresponding to the output file posted earlier)
Comment: Please check solution below.
Answer: From what I understand you simply want all rows where SYMBOL='HINDUNILVR' to be copied to a new file.
Below Sample code read the same file and write back to the same file. You can change the file name as required:
import pandas as pd
import os
df1 = pd.read_csv("Sample.csv")
df2 = df1[df1["SYMBOL"]=="HINDUNILVR"]
with open('Sample.csv','a+') as f:
    df2.to_csv(f,header=False,index=False)

or
import pandas as pd
import os
df1 = pd.read_csv("Sample.csv")
df2 = df1[df1["SYMBOL"]=="HINDUNILVR"]
df2.to_csv("Sample.csv",header=False,mode='a+',index=False)

or an Update on your code :
 if(int(str(sdate.day))<10):
    df1 = pd.read_csv(zf.open("cm" + "0" + str(sdate.day) + MonthCode(sdate.month) + str(sdate.year) + "bhav.csv"))
else:
    df1 = pd.read_csv(zf.open("cm" + str(sdate.day) + MonthCode(sdate.month) + str(sdate.year) + "bhav.csv"))
df2 = df1[df1["SYMBOL"]=="HINDUNILVR"]
df2.to_csv("C:\stock.csv",mode='a+',header=False,index=False)

or
    if(int(str(sdate.day))<10):
    df1 = pd.read_csv(zf.open("cm" + "0" + str(sdate.day) + MonthCode(sdate.month) + str(sdate.year) + "bhav.csv"))
else:
    df1 = pd.read_csv(zf.open("cm" + str(sdate.day) + MonthCode(sdate.month) + str(sdate.year) + "bhav.csv"))
df2 = df1[df1["SYMBOL"]=="HINDUNILVR"]
with open('C:\stock.csv','a+') as f:
    df2.to_csv(f,header=False,index=False)

Comment: You can try with below as well:
        `saved_row.to_csv(f,mode='a',header=False)`
default is 'w' but very difficult without sample file.
Comment: `a+`  in case file does not exist it will create file.
Comment: shouldn't it be : df1 = pd.read_csv(zf.open("data.csv")), where data.csv is located within the zip file, and Sample.csv is where we want to store the data
Comment: I mentioned you can change the file name as required. I have updated above as per your code as well.
Comment: if this is correct as per your requirement please accept the solution.
Comment: Still I am not getting the required output, you haven't changed my code in any way in your updates
Comment: Thank You so much, it is working now, there was another error elsewhere that I also had to fix.
Comment: Cheers, always Welcome


How to run automated codeception acceptance tests using docker-compose?

Question: I am trying to run the tests written in codeception using docker-compose. I have written docker-compose.yml. The file looks like below:
The same folder that has docker-compose has tests (folder), codecept.phar and codeception.yml .
I am trying to run the following command:
docker-compose run --rm codecept run acceptance
It tries to execute the acceptance tests but I am getting the following error:
"[ConnectionException] Can't connect to Webdriver at http://127.0.0.1:4444/wd/hub. Please make sure that Selenium Server or PhantomJS is running."
Any ideas?
Expecting the command to start executing the codeception acceptance tests.
docker-compose.yml looks like below:
version: '3'
services:
  codecept:
    image: codeception/codeception
    depends_on:
      - chrome
      - web
    volumes:
      - .:/project
  web:
    image: php:7-apache
    #depends_on:
     # - db
    volumes:
      - .:/var/www/html
  db:
    image: percona:5.6
  chrome:
    image: selenium/standalone-chrome

Command run to execute the tests:
docker-compose run --rm codecept run acceptance
Answer: I see 2 problems in your setup.
First you need to configure links like this:
version: '3'
  services:
    codecept:
      image: codeception/codeception
      depends_on:
        - chrome
        - web
      links:
        - chrome
        - web
      volumes:
        - .:/project
    web:
      image: php:7-apache
      #depends_on:
      # - db
      volumes:
        - .:/var/www/html
    db:
      image: percona:5.6
    chrome:
      image: selenium/standalone-chrome

Then you need to change the configuration that points to 127.0.0.1:4444 to chrome:4444 as docker-compose will create this dns entry using the name of the linked service in the network.
127.0.0.1 will not work because it isn't running inside the codeception container.