How does docker stats output relate to top output?

Question: I've got a process running in a Docker container on Linux, with a 2GByte memory limit. (The container is started with docker run --memory=2g.)
Here is what top says about it after it's been running for a while.
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 9016 root      20   0 7342132 4.652g 4.224g S 100.0  7.4  18828:28 blah

Here's the output of docker stats:
CONTAINER           CPU %               MEM USAGE/LIMIT     MEM %               NET I/O
d7032e5928b6        100.02%             2.076 GB/2.147 GB   96.68%              345 MB/199.1 MB

Then, after restarting it, top:
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 5653 root      20   0 2787800 1.328g 1.030g S 100.0  2.1   0:37.57 blah

docker stats:
CONTAINER           CPU %               MEM USAGE/LIMIT     MEM %               NET I/O
10cefdce241f        99.97%              320.4 MB/2.147 GB   14.92%              36.34 kB/22.48 kB

So that matches up (very roughly) with RES-SHR, which makes some sense to me - but the equivalent calculation for the output from the long-running container doesn't match up in the same way.
What exactly is Docker counting here? Can I match it up somehow with the output from  top, or some other tool?
Answer: docker stats include file cache memory as well. It will be freed upon request, but it is counted as used.
Answer: Docker stats are reporting the usage of memory and CPU within the container's cgroup. This will correlate with the some of the top stats, but not exactly since memory in a process includes shared memory, memory mapped to a filesystem object, etc. The cgroup is going to aggregate the usage of all of the processes within that container. To see the cgroup usage inside the container, you can look at files in /sys/fs/cgroup/memory/:
$ docker run -it --rm --name memtest -m 2g busybox /bin/sh
/ # cat /sys/fs/cgroup/memory/memory.usage_in_bytes
1519616
/ # cat /sys/fs/cgroup/memory/memory.limit_in_bytes
2147483648


Why use V43 over vii°6?

Question: It seems that there is a leap of a 4th when moving from I6 to V43. wouldnt a descending bassline with ^2 having a vii°6 be smoother in connecting I6 with I53? Or perhaps there is another good reason to use V43?
Comment: How do you avoid having a leap with vii°6?
Comment: key signature..?
Answer: When you propose I6 viio6 I you gloss over the main issue to consider: which tones get doubled in triads in four part harmony.
The reason for the leap in the example is because the tonic is doubled. If you double the fifth of I6 then the leap is avoided...

If you did it with three parts, keeping the main bass and soprano, I think the basic voice leading for all step-wise motion would be...

...where FA would resolve to MI and SOL is omitted from the final I chord.
Finally, if you want I6 viio6 I in three parts, complete triads, while maintaining tendency tone movements, you cannot avoid a leap. So, I think the thing to do is put the leap in the inner voice so the outer voices are in smooth steps, like this...

Now, back to the question:

Why use V43 over vii°6?

I think it really is just the same as any other harmony choice. Melodic considerations, wanting to ensure or avoid certain melodic intervals. Harmonic considerations, wanting the "fuller" sound of a seventh chord, the particular resonance of a fourth, the lighter sound of triadic viio6, etc.
Perhaps the point to make is "smooth" voice leading may be defined as step-wise voice movement, but that doesn't necessarily mean the only good voice leading is smooth, step-wise motion.
Answer: Using V4/3 instead of vii°6 allows the tenor to keep the B like it's a pedal tone. It's also possible that the composer just plain likes the sound of V4/3 better than vii°6 there.
Switching to vii°6 does let you use E-F♯ (presumably)-G♯ (presumably) in the alto and B-A-B in the tenor, though...
Comment: ok but doesnt the fact that you have to leap a 4th to make the progression mean it is not as good as using a vii°6?
Comment: @armani - If the composer just plain likes the sound of V4/3 better than vii°6 there, I'd think that the chord progression being "not as good" voice leading-wise would have been no obstacle, and the composer would make V4/3 work no matter what.
Comment: @armani how do you voice the chords?  The tenor has to have ^5 ^4 ^3; ^5 ^4 ^5 frustrates the expected downward resolution of ^4 to ^3.
What then does the alto do?  It can't be ^1 ^2 ^3 because that doubles the third in the last chord.  Stepwise motion ^1 ^2 ^1 or ^1 ^7 ^1 yields parallel octaves with bass or soprano.  With a four voice texture you're going to have a leap of some sort with vii°6, too.
Comment: Phoog see this example https://ibb.co/swLCK0z 2nd bar 2nd chord
Comment: @armani - Note that the example you have uses ^1 ^2 ^3 in the soprano, which dodges all the parallel octave problems phoog mentions while letting an inner voice still use ^5 ^4 ^3. Looks like that soprano in your OP is screwing things up even further than we initially thought.
Comment: Ok but even with ^1^2^3 in the soprano, there will still be a 4th leap in the progression I63 V43 I right? By using vii6 you avoid that.
Comment: @armani - Actually, ^1 ^2 ^3 cannot support V4/3 to harmonize ^2 at all unless you use 5 or more voices, since you need a ^2 in the bass and the soprano and ^5, ^7, and ^4 elsewhere.
Answer: The rule that  in a 6th chord (1st inversion) the root tone has to be doubled has no statistical fundament. If we double here the 3rd (G#) the progression has no leap at all.
Comment: It is not statistical but I do kind of dislike the doubled 3rd... it overtakes the sound of the root of the chord IMO. And perhaps why many use the 3rd as a last resort when doubling
Comment: I've posted here else where an statistical research by Diether De la Motte of the doubled chord tones in the music of Bach, Handel and Haydn.

Why use V43 over vii°6?


Files with cyrillic letters in name public storage could not be retrieved in Laravel

Question: I'm getting Object not found error if file contains Cyrillic letters in it's name.
I'm using standard configuration with public storage and symbolic link to public directory and retrieving url to file using following code:
$url = Illuminate\Support\Facades\Storage::disk(config('filesystems.default'))
           ->url($file->getDiskPath());

Thank you for your help, in advance!
Answer: Finally I come up to some solution which fits my needs. Hope it will help someone who face it in future:
First of all I added support of slug for my item. Luckily I was already using Spartie/laravel-tags package and it was as easy as add HasSlug trait to my model.
Then I wrote function which gave me url like:
baseUrl + '/get-item/' + item.slug.en + '/' + item.item_media.id

After then I created controller to catch GET request to that URL:
public function getItem($slug, $mediaId)
{
    $media = Media::find($mediaId);

    if($media !== null){
        $fileName = $media->filename;
        $filePath = $media->directory;
        $fileExt  = $media->extension;
        $fileMime = $media->mime_type;

    $path = Storage::disk(config('filesystems.default'))->path($filePath . '/'. $fileName . '.' . $fileExt);

    if(!File::exists($path)) abort(404);

    $headers = array(
        'Content-Type: ' . $fileMime,
    );

    return response()->download($path, $fileName, $headers);
    }
}

Of course $slug parameter is redundant in controller definition, but having this string in URL is very useful for SEO purposes.
Update
Problem was not in Russian (or Cyrillic in route). In fact Larave does handle it correctly in route. The actual issue wasn in encoded special characters as per (laravel. Replace %20 in url) and following link gave me a hint to resolve in in very elegant way:
I rewrited url to following:
baseUrl + '/download?file=' + directory + filename + '.' + extension

after that my controller method changed as following:
public function getItem(Request $request)
{
    $media = null;

    if($request->has('file')) {
        $name = $request->get('file');
        $media = Media::forPathOnDisk(config('filesystems.default'), $name)->first();
    }

    if($media !== null){
        $fileName = $media->filename;
        $filePath = $media->directory;
        $fileExt  = $media->extension;
        $fileMime = $media->mime_type;

    $path = Storage::disk(config('filesystems.default'))->path($filePath . '/'. $fileName . '.' . $fileExt);

    if(!File::exists($path)) abort(404);

    $headers = array(
        'Content-Type: ' . $fileMime,
    );

    return response()->download($path, $fileName, $headers);
    }
}


Logic operations - trying to understand basics in forming

Question: I have noted the basics I got from the internet on a sheet of paper to learn but I'm not sure if it's correct because the source isn't really trustable in my eyes, so I'll write it down here what I've written on my paper and I hope this is the correct section for it!
I'm especially not sure if last line is correct. About the others, I think it seems logic and makes sense :)
If possible, please make a little personal task for me and I solve it. You say if I did it right. I would really appreciate that because learning everything myself isn't that easy.
$$A \vee \bar{A} = 1$$

$$A \wedge \bar{A} = 0$$

$$A \wedge 1 = A$$

$$A \vee 1 = 1$$

$$(A\wedge A) \vee (\bar{A} \wedge B) = B \wedge (A \vee \bar{A}) = B \wedge 1 = B$$
Answer: The last line is not correct. The correct expression beggins with '(A∧B)'; not with '(A∧A)'. The rest is OK.
I wish you luck in your learning
Comment: What do you mean? We have the commutativity laws?
Answer: You wrote "$(A∧A)∨(\bar A∧B)=B∧(A∨\bar A)$". From where does it come? You can't just pull it out of thin air. Firstly you can simplify "$A \land A$". Also, use the distributivity rule properly, namely:

$P \lor ( Q \land R ) = ( P \lor Q ) \land ( P \lor R )$.

If you don't like this rule, the other distributivity rule actually suffices, namely:

$P \land ( Q \lor R ) = ( P \land Q ) \lor ( P \land R )$.

The proof will be a little harder, but perhaps make more sense:

$A \lor ( \neg A \land B )$
$\ = ( A \land ( \neg B \lor B ) ) \lor ( \neg A \land B )$
$\ = ( A \land \neg B ) \lor ( A \land B ) \lor ( \neg A \land B )$
$\ = ( A \land \neg B ) \lor ( A \land B ) \lor ( A \land B ) \lor ( \neg A \land B )$
$\ = ( A \land ( \neg B \lor B ) ) \lor ( ( A \lor \neg A ) \land B )$
$\ = A \lor B$.

Draw out the venn diagram to see how each step works.
Comment: Got it :D

$$(A \wedge A) \vee (\bar{A} \wedge B) = (A \vee \bar{A}) \wedge (A \vee B) = 1 \wedge (A \vee B) = (A \vee B)$$
Comment: @cnmesr: That's correct.
Comment: Big thanks to both of you for helping me! :)
http://math.stackexchange.com/questions/1923775/strange-logic-task-from-an-old-exam-proof-required

Here I have another but more advanced task, I'd be very happy and grateful if you could take a look please!


Bulk remove/convert the .WebP files

Question: I downloaded some images from a page in a folder, and have all images that finish by .webp, say, toto.jpg.webp, titi.png.webp...
Is there a way in the Window's Command line to bulk remove the .webp extension from all the files in the folder?
I tried to ren *.jpg.webp *.jpg, but it renamed them in .jpg.jpg, I don't like... )
Answer: I understood that Google's webp is not only a file rename, but a real image format that is not compatible with most of the image editors. So, there's not enough to rename, but also to bulk convert such files.
So, I used XnConevrt to bulk convert these files. You just add your folder, select a filter (.png, by eg) and then it bulk converts for you all the .png.webp to real .png files.
Comment: That app is awesome

Bulk remove/convert the .WebP files


Determine a positive integer $n\leq5$,such that $\int_{0}^{1}e^x(x-1)^ndx=16-6e$

Question: Determine  a positive integer $n\leq5$,such that $\int_{0}^{1}e^x(x-1)^ndx=16-6e$.
I tried to solve it.But since $n$ is given to be $\leq$ 5,my calculations went lengthy.
Applying integration by parts repeatedly,we get
\begin{align}
\int e^x(x-1)^n \, dx &= \left[(x-1)^ne^x-n(x-1)^{n-1}e^x+n(n-1)(x-1)^{n-2}e^x \right. \\
& \hspace{5mm} \left.-n(n-1)(n-2)(x-1)^{n-3}e^x+n(n-1)(n-2)(n-3)(x-1)^{n-4}e^x \right. \\
& \hspace{5mm} \left.-n(n-1)(n-2)(n-3)(n-4)e^x\right]
\end{align}
\begin{align}
\int_{0}^{1}e^x(x-1)^n \, dx &= -n(n-1)(n-2)(n-3)(n-4)e-(-1)^n+n(-1)^{n-1}-n(n-1)(-1)^{n-2} \\
& \hspace{5mm} +n(n-1)(n-2)(-1)^{n-3}-n(n-1)(n-2)(n-3)(-1)^{n-4} \\
& \hspace{5mm} +n(n-1)(n-2)(n-3)(n-4) \\
&=16-6e
\end{align}
Now solving this is very difficult,is there another simple and elegant method to find $n=3.$
Answer: HINT:
Let $x-1=y$
$$\int_0^1e^x(x-1)^n\ dx=\int_{-1}^0e^{y+1}y^n\ dy=e\int_{-1}^0e^yy^n\ dy$$
Now integrating by parts, $$I_n=\int_{-1}^0e^yy^n\ dy=[y^n\int e^y\ dy]_{-1}^0-\int_{-1}^0\left(\dfrac{d(y^n)}{dy}\int e^y\ dy\right)dy$$
$\implies I_n=-(-1)^n\dfrac1e-nI_{n-1}$
Now $I_0=\int_{-1}^0e^y\ dy=1-\dfrac1e$
$\implies I_1=\dfrac1e-I_0$  and so on
Answer: HINT:
Notice, the following property of definite integral
$$\int_{a}^{b}f(x)dx=\int_{a}^{b}f(a+b-x)dx$$
Now, we have
$$\int_{0}^{1}e^x(x-1)^ndx=16-6e$$
$$\int_{0}^{1}e^{1-x}(1-x-1)^ndx=16-6e$$
$$\int_{0}^{1}e^{1-x}(-x)^ndx=16-6e$$
$$\int_{0}^{1}e \cdot e^{-x}(-1)^nx^n dx=16-6e$$
$$(-1)^n e\int_{0}^{1} e^{-x}x^n dx=16-6e$$
I hope you can proceed further.
Comment: Perfect answer :D
Answer: If we set
$$ I_n = \int_{0}^{1}e^x(1-x)^n\,dx = \int_{0}^{1}x^n e^{1-x}\,dx\tag{1}$$
integration by parts gives:
$$ I_{n+1} = -1 + (n+1)\, I_n\tag{2}$$
Since $I_0 = -1+e$, by using the previous formula and induction we have:
$$ \forall n\in\mathbb{N},\qquad I_n = A_n + B_n e,\quad A_n,B_n\in\mathbb{Z}\tag{3}$$
as well as
$$ B_n = n! \tag{4} $$
so the solution to the original problem is clearly $n=\color{red}{3}$.
Answer: I would define
$$a_n:=\int_0^1 e^x (x-1)^n dx.$$
Then we simply get $a_0=e-1$ and for $n>0$ using integrations by parts we get
$$a_n = [e^x(x-1)^n]_0^1 - n\int_0^1 e^x (x-1)^{n-1}dx = (-1)^{n+1}-n a_{n-1}.$$
With that recursive formula you compute $a_1$, $a_2$ and $a_3$ very quickly and get your result $a_3=16-6e$.
Answer: If you have reason to believe the equation is satisfied by a positive integer $n\le5$, we can rule out the even integers $n=2$ and $4$ since $16-6e\approx-0.30969$ is negative.  For odd integers, a change of variable leaves the equation
$$I_n=\int_0^1 x^ne^{-x}dx=6-{16\over e}\approx0.1139$$
Now
$$1-x\le e^{-x}\le1-x+{1\over2}x^2\quad\text{for }0\le x\le1$$
which means
$$I_1\ge\int_0^1(x-x^2)dx={1\over2}-{1\over3}={1\over6}=0.1666\ldots$$
is too big, and
$$I_5\le\int_0^1(x^5-x^6+{1\over2}x^7)dx={1\over6}-{1\over7}+{1\over16}\approx0.0863$$
is too small.  So by the Goldilocks Principle, $n=3$ must be just right.


Question on terminology

Question: If a map $M$ has the feature
$$
M(x+y)=M(y)M(x)
$$
is there a term to describe this kind of map? It's not a linear map, at least. The order matters here - $M(y)M(x) \neq M(x)M(y)$.
Comment: You can't have $M(y)M(x)\neq M(x)M(y)$, because that would imply that $M(x+ y)\neq M(y + x)$, and that violates either what we want functions to be, or what we (usually) want addition to be.
Comment: But what if we have a function on the set of ordinal numbers: Then we have $\omega+1 \neq 1+\omega$, so what's the problem since each of those $2$ ordinals may have an image?
Comment: @Matti: Can you clarify if your question is about arbitrary (possibly non-commutative) addition/multiplication or not? Or is it just about functions from $\Bbb R \to \Bbb R$? Do you assume continuity?
Comment: @MartinR I asked this question over two years ago. It's been answered already. Thank you.
Answer: Its name is an exponential map.
Answer: A map $M: A \to B$ with the property that
$$
\forall x, y \in A: M(x+y)=M(x)M(y) \tag{*}
$$
is called a group homomorphism from the (additive) group $(A, +)$ to the (multiplicative) group $(B, \cdot)$. Example:
$$
 M : \Bbb C \to \Bbb C,  z \mapsto e^{\operatorname{Re}(z)}.
$$
Your condition
$$
\forall x, y \in A: M(x+y)=M(y)M(x) \tag{**}
$$
is equivalent to $(*)$ if $(A, +)$ is commutative (i.e. $A$ is an abelian group), as mentioned in the comments. I am not aware of a term for $(**)$ in the non-commutative case.

In the particular case that $A=B= \Bbb R$, $(*)$ is equivalent to $f = \log M$ being an additive function on $\Bbb R$, i.e. it satisfies Cauchy's functional equation
$$
\forall x, y \in \Bbb R: f(x+y)=f(x) + f(y) \, .
$$
This functional equation has nonlinear solutions, but under some conditions (i.e. continuity) it follows that $f(x) = cx$ for some constant $c$. So in this special case we have that $M$ is an exponential function: $M(x) = e^{cx}$.
Comment: The term you seek is "antihomomorphism".

Question on terminology


show that $y = \cos x$, has a maximum turning point at $(0, 1)$ and a minimum turning point at $(\pi, -1)$

Question:
show that  $y = \cos x$, has a maximum turning point at $(0, 1)$ and a
  minimum turning point at $(\pi, -1)$

Turning points occur when the gradient is 0 or $\frac{dy}{dx} = 0$
$f(x) = \cos x$
$\frac{d}{dx} = - \sin x$
$x = -\sin^{-1}(0)$
$x = 0$
sin is in the 1st and 2nd quadrant so $x = 0$ or $x = -90$
$f''(x) = - \cos x$
$f''(0) = - 1$ which is a maximum turning point
$f''(-90) = 0$, is this a point of inflection so I am not sure if this is right.
When $x = 0, f(x) = \cos(0) = 1$
So we have a maximum turning point at $(0, 1)$.
I'm not sure how to get the minimum turning point?
Comment: Is the value of  $ \ x \ $ given by the inverse sine function really the _only_ value for which $ \ \frac{dy}{dx} \ = \ 0 \ $ ?  (Also, $ \ -90º \ $ is neither in the _second_ quadrant, nor is its sine value zero.)
Answer:
$$f(x)=\cos x \qquad [-1,\pi]$$

Finding the $\min$ and $\max$:
$$f'(x)=-\sin x$$
that's equal to zero when $x=\color{red}{0},x=\color{red}\pi$ (critical points)
$$f''(x)=-\cos x\\
f''(0)=-\cos 0=-1$$
$\Longrightarrow\quad x=0$ is $\max$ of $\cos x$
$$\boxed{\max \text{ point }(0,1)}$$
$$f''(\pi)=1$$
$\Longrightarrow\quad x=\pi$ is $\min$ of $\cos x$
$$\boxed{\min \text{ point }(\pi,-1)}$$
Comment: @dagda1 do you agree that $\sin \pi=0$? multiply this by $-1$, still zero
Comment: $\sin (0) = \pi$. how is that?
Comment: @dagda1 I didn't say that, I said $-\sin 0=0$ and also $-\sin \pi=0$
Comment: $- \sin \pi = 0$?.  This because sin is positive in the 1st and 2nd quarter and 180 - 0 = 180 or \pi?
Answer: The derivative of $f(x) = \cos x$ is $f'(x) = -\sin x$.  The critical points of $f$ occur where $f'(x) = 0 \implies -\sin x = 0 \implies \sin x = 0$.
The sine of an angle $\theta$ in standard position (vertex at the origin, initial side on the positive $x$-axis) is the $y$-coordinate of the point where the terminal side of the angle intersects the unit circle.

Consequently, $\sin\theta = 0$ if the terminal side of angle $\theta$ lies on the $x$-axis.
Hence, $\sin x = 0 \implies x = n\pi$, where $n$ is an integer.  Thus, $x = 0$ and $x = \pi$ are critical points of the function $f(x) = \cos x$.
We can draw the sine graph by tracing out the $y$-coordinates of the point where the terminal side of the angle intersects the unit circle.

The graph of $f'(x) = -\sin x$ is obtained by reflecting the graph of the sine function in the $x$-axis.

From the graph of $f'(x) = -\sin x$, we see that the derivative is positive in the interval $[-\pi/2, 0)$ and negative in the interval $(0, \pi/2]$.  Thus, the function $f(x) = \cos x$ is increasing to the left of $0$ and decreasing to its right.  Hence, $f(x) = \cos x$ has a relative maximum at $x = 0$ by the First Derivative Test. The relative maximum value is
$f(0) = \cos 0 = 1$.
From the graph of $f'(x) = -\sin x$, we see that the derivative is negative in the interval $[\pi/2, \pi)$ and positive in the interval $(\pi, 3\pi/2]$.  Thus, the function $f(x) = \cos x$ is decreasing to the left of $\pi$ and increasing to its right.  Hence, $f(x) = \cos x$ has a relative minimum at $x = \pi$ by the First Derivative Test.  The relative minimum value is $f(\pi) = \cos \pi = -1$, as shown in the graph below.


Mean and variance of a Random Walk with lower boundary

Question: Consider a random walk $S_n$ that starts at $S_0$ and terminates if it hits a lower boundary of $0$.
$$
S_n=\begin{cases}
0, & \text{if } S_{n-1}=0\\
0, & \text{if } S_{n-1}+x_n\le0\\
S_{n-1}+x_n, & \text{otherwise}
\end{cases}
$$
$$n=1,2,3,.. $$
$$x_i \text{ are i.i.d. with } x_i \sim N[\mu,\sigma^2]$$
what are
$$E[S_n]$$
$$\text{VAR}[S_n]$$
as a function of
$$S_0, \mu, \sigma^2 \text{ ?}$$
Comment: Are the $x_i$ IID?
Comment: @NapD.Lover yes! thanks for the clarification - I added it to question.
Answer: The random walk can be written as $S_n=Y_n^+$ where $Y_n=S_0+X_1+\dotso +X_n$, the $X_i$ are IID $\mathcal{N}(\mu, \sigma^2)$ RVs, and $S_0>0$ some positive number and where $x^+=x \vee 0=\max\{x, 0\}$ is the positive part of $x$. At least this holds up until the first time $k$ where $Y_k\leq 0$, afterwards, $S_n=Y_k^+=0$ for all $n\geq k$.
With that in mind, conditional on $Y_n \leq 0$, we have $S_n=0$ hence $\mathbb{E}(S_n|Y_n \leq 0)=0$. Thus, by the law of total probability
$$\mathbb{E}(S_n)=\mathbb{E}(S_n | Y_n >0)P(Y_n>0).$$
Now since on the event $Y_n>0$ we have $S_n= Y_n=S_0+X_1+\dotsc +X_n,$ and the conditional expectation is linear, we get $\mathbb{E}(S_n|Y_n>0)=S_0+n\mu$. All that is left is to compute
$$\mathbb{P}(Y_n>0)=1-\mathbb{P}(Y_n\leq 0)$$
$$=1-\mathbb{P}\left(Z_n \leq \frac{-S_0-n\mu}{\sigma \sqrt{n}} \right),$$
where $Z_n \sim \mathcal{N}(0,1)$ is a standard normal RV. We have used the fact that IID sums of normal RVs are again normally distributed and then standardized by the new mean and standard deviation. All together,
$$\mathbb{E}(S_n)=(S_0+n\mu) \left[1-\Phi \left( \frac{-S_0-n\mu}{\sigma \sqrt{n}}\right) \right].$$
A similar argument should be able to compute the variance, or perhaps via the law of total variance. I'll attempt to add it later if you have trouble but I'm out of free time right now. If something is unclear or mistaken please comment!
Here is some R-code I quickly drafted to verify this numerically. For $n=50$ steps with $\mu=3$, $\sigma=4$ and $S_0=10$, the last run obtained sample mean $158.8442$ (estimated from $N=5000$ simulations of the walk ending $S_{50}$) and exact value $160$.
<code>n <- 50 # number of steps in random walk
N <- 5000 # number of simulations of the RW
mu <- 3 # given mean
volat <- 4 # given volatility
s0 <- 10 # initial localtion
tails <- matrix(0, nrow = N)
for(i in 1:N)
{
  s <- matrix(0, nrow = n+1)
  s[1] <- s0
  for(j in 2:(n+1))
  {
    if(s[j-1] >0 )
    {
      s[j] <- s[j-1]+rnorm(1, mu, volat)
    }
    s[j] <- pmax(s[j], 0)
  }
  tails[i] <- s[n+1]
}
est <- data.frame(approx = mean(tails), exact = (s0+mu*n)*(1-pnorm((-s0-n*mu)/(volat*sqrt(n)))))
print(est)
par(mfrow = c(2, 1))
plot(s, type = "l")
plot(tails, type = "l")
</code>
Comment: @nap-d-lover thanks for your help! Unfortunately your formula seems to diverge from reality when σ ≫ μ . With n=50, s0=10, mu=1, volat=5, I get approx=47, exact=57. It's not a sampling problem. I tested it with N=1 million simulations and also implemented it in Python and C. Identical results approx=47, exact=57. I am not sure where the flaw in your reasoning is. I suspect it is that (>0) erroneously includes the cases where  has gone below zero for some < before getting back up to >0. These case are indeed more common when σ ≫ μ.
Comment: @elemolotiv I will try to fix this and write an edit today if I can work something out. I suspect your hunch is correct so if I can't work out a better answer I'll edit this to say it holds for when $\sigma$ is not too large relative to $\mu$. Thank you for checking carefully different $\mu$ and $\sigma$ and for larger samples and multiple implementations.


Inverted dose-response variables

Question: Context:
Often when we carry out dose-response modelling we want to estimate the dose required to elicit a predetermined response (i.e. response ~ dose). Typically this is done with inverse regression techniques (i.e. after-fitting / reparameterisation), but sometimes these impose constraints or come with a greater degree of uncertainty.
Question:
Is there any statistical reason that we shouldn't instead just swap the terms (I.e. dose ~ response), if the goal of the exercise is fit a model with the most accurate predictions?
I understand that it is very unorthodox (at least) to suggest that an applied treatment is dependent on the response and goes against the underlying theory but is there any mathematical reason we should not do this if it gives more accurate predictions.
Answer: Yes, there is a reason not to do it.
Nonlinear regression (as usually implemented and performed) is based on the assumption that the distribution of the differences between the actual Y values and the predicted Y values (the curve) is Gaussian. If that assumption is reasonable, then it is unlikely that the distribution of the difference between actual concentrations and the predicted concentrations would be Gaussian. If the assumption is far off, then so are the results.
It is easy enough to fit the curve conventionally and then interpolate the values you need.
Comment: @Galen. Thanks. I added a clause to my answer that this assumption (normal residuals) is the basis of commonly used nonlinear regression methods (but other methods could be developed)
Comment: What about with models like SVMs?
Comment: What does SVM stand for?
Comment: Support Vector Machine
Comment: Non-linear regression does **not** assume that the residuals are Gaussian! That is an assumption of least-squares, which can be used in some non-linear cases.

Inverted dose-response variables


How to track scripts that are sending SPAM?

Question: Our server is hosting multiples websites for different clients.
The hosting provider has blocked the port 25 after detecting that our dedicated server is sending spams.
I have checked Qmail logs but found nothing suspicious. Spamassassin at first look seems not giving too much options for my problem.
Answer:
How to track scripts that are sending SPAM?

Since your hosting provider has blocked you from sending mail you can inspect the mail logs and mail queue for clues:

Depending on your system setup scripts are running under a unique client ID that gets logged, which narrows down the hosting account.
Look at the mail queue, it depends on how and which scripts, many of the better ones include the script name in the headers and when that is not the case usually the Reply-to or From headers are set which allows you to narrow down the client account.
Correlate with the web access logs and usually it is pretty easy to find the offending script.

It might be that the mailqueue doesn't contain any spam and your mail server's logs don't indicate that you were sending spam either; then rather than an existing script/mail facility that is getting abused the spammer was able to upload a custom script/program that connects directly to remote mail servers, by-passing Qmail entirely.  In that case: How do I deal with a compromised server?
Answer: Did you know, that Plesk provides as well a very detailed documentation? You might consider to read and follow:
=> Fighting Spam on a Qmail Mail Server ( Plesk Onyx - online documentation - Advanced Administration Guide, Plesk for Linux )
Edit (as requested ):
You might find usefull commands, as for example:
/var/qmail/bin/qmail-qstat

Check how many messages there are in the Qmail queue
/var/qmail/bin/qmail-qread

Read the messages headers, it will show senders and recipients. Two many recipients are mostly a sign for spam.
find /var/qmail/queue/mess/ -name 1234567

Find messages by it's ID and inspect the the headers. You could investigate, that a user sends quite a lot of mails via CGI - scripts for example, which is another indicator for spam.
More can be found on the above mentioned link. :-)
Comment: @[UFHH01](https://serverfault.com/users/405609/ufhh01), could you provide some highlights/specific points from that link you feel would help here (I am not sure the asker is using Plesk, so if you could extract the useful commands, that would be helpful) - [good answers](https://serverfault.com/help/how-to-answer) should provide context for links
Comment: @iwaseatenbyagrue : Edited as suggested - thx for the hint.
Answer: grep Received\: /var/log/maillog | grep --color invoked\ by\ uid

The uid(s) that comes out most should be the same as the compromised vhost(s). If uid 0 you're really in trouble.
You didn't provide enough details for me to be sure you're logging in /var/log/maillog. In plesk, for instance, maillog is elsewhere.
Answer: You might consider talking to your hosting provider to see what they can provide to help you - they presumably cut off your port 25 traffic based on something, and knowing what that was would be helpful.
Notably, I would check with them if this is an issue of volume, or of content - the fix in each case might be a bit different.
I would also check your /var/log/mail (or equivalent file) to try and identify the top sender(s) - notably, I would check entries showing nrcpt greater than 1, that use a domain you host.
I would suggest you ensure each one of your domains/websites runs as its own (local) user - that way, if you find the issue comes from a single domain, you can more easily block that local user from sending anything (if you run all sites under the same user, you are going to have to do domain matching, etc).
As you have SpamAssassin, I would also ensure you are scanning outbound emails (whether or not you block them based on spaminess is up to you, but I would ensure they are scored for spaminess). Review the stats for each domain (or sender if that becomes necessary), and see if that helps you gain some insight.


Are there characteristic classes for symplectic vector bundles?

Question: Given a real vector bundle, say the tangent bundle of a manifold, some obstructions to this being the underlying real bundle of a complex bundle come from characteristic classes. These can prove the lack of complex structure on the underlying manifold. I was wondering if perhaps there was a similar set of classes for symplectic manifolds?
Comment: Given a symplectic manifold, there exists a contractible set of almost complex structures compatible with the symplectic structure. These almost complex structures give a complex vector bundle structure on the tangent bundle. You can define characteristic classes to be the chern classes of this complex vector bundle.
Answer: You have classifying spaces $BSp(n)$ and $BGl(2n)$ and from the inclusion $Sp(n)$ into $Gl(2n)$ you get a map $\iota \colon BSp(n) \to BGl(n)$.
A characteristic class for a real vector bundle is nothing else than a cohomology class of $BGl(n)$ (This follows from naturality). And you can define characteristic classes analogously for symplectic bundles by taking cohomology classes for $BSp(n)$. A symplectic structure of your vector bundle now corresponds to a lift of the classifying map to $BGl(n)$ into $BSp(n)$.
But now if you understand $\iota^*$ you can get that certain characteristic classes can't occure if you have such a lift. For example if you were looking at real bundles and asking for a complex structure you would get that every odd Stiefel-Whitney-Class would have to vanish since the complex grassmanian has no cohomology in odd dimensions.
I'm not aware of how the cohomology ring of $BSp(n)$ looks like, but with that information and a little computation of $\iota^*$ you would get explicit classes, that obstruct such a lift.
Comment: I think $BSP(n)=G_n(\mathbb{H}^\infty)=\mathbb{Z}[a_1,\ldots, a_n]$ with degree $a_i=4$, so all stiefel whitney classes in degree $0,1,2,3 \pmod 4$ have to vanish, and probably $c_n=0$ for $n$ odd as well.


Draw from random distributions so that sum of values is fixed

Question: I want to draw tuples of $N$ random numbers $v_{ij}$ with $i=1,..,N$ the tuple index and $j$ the repetition of the experiment.
The sum of all random numbers in a tuple should be constant $\forall j \sum_i v_{ij}=S$.
The expectation and variance of each $i$-th index in the tuples should be known $\mathrm{E}_j(v_{ij})=e_i>0, \sum_i e_i=S$ and $\mathrm{Var}_j(v_{ij})=v_i$.
The values should be non-negative $0\le v_{ij}\le S$.
The distribution of $v_{ij}$ for each $i$ should look as normal-distributed like as possible.
I'm aware that the condition that the sum is fixed will make them not independent. If possible I would like to distribute the correlations between different indices $i$ as equally as possible.
My idea:
I draw $N-1$ binomially distributed random numbers, scale them to match $e_i$ and $v_i$, then set the last value in the tuple to $S$ minus the sum of these distributed numbers.
However, the variance of the last index value in the tuples is usually much larger than desired and difficult to set independently and I cannot guarantee that the values in the last index stay within the bounds $[0,S]$.
Example: $N=4,e_1=10,e_2=20,e_3=40,S=100$.

The answers of Simulation involving conditioning on sum of random variables didn't help me because they seem to be special to the problem presented there and not applicable to my conditions.
How can I achieve what I want?
Comment: Related (but no duplicates): [Generate uniformly distributed weights that sum to unity?](https://stats.stackexchange.com/q/14059/1352) and [How to simulate a uniform distribution of a triangular area?](https://stats.stackexchange.com/q/289258/1352). The [Dirichlet distribution](https://en.wikipedia.org/wiki/Dirichlet_distribution) is also related, but it doesn't offer much flexibility, since the means and variances of all variables are linked. The [Generalized Dirichlet](https://en.wikipedia.org/wiki/Generalized_Dirichlet_distribution) may be what you are looking for.
Comment: In continuous case you can re-scale the Dirichlet distribution, in discrete, you can use multinomial distribution. You can also check the [multivariate logistic normal distribution](https://en.wikipedia.org/wiki/Logit-normal_distribution#Multivariate_generalization)
Answer: You can get close to what you want by using a singular normal distribution - and if you use equal variances you will get equal negative pairwise correlations.
Using the transformation method (see guy's solution here r - pick 10 random numbers from standard normal distribution whose sum equals 5 for an example), we can find the covariace matrix (I've chosen 3 as the variance)
$$ \Sigma = \pmatrix{\ \ 3 & -1  & -1 & -1 \\
                    -1 & \ \  3  & -1 & -1 \\
                    -1 & -1 & \ \ 3 & -1  \\
                    -1 & -1  & -1 & \ \ 3}.$$
Now we can use singular value decomposition (SVD) to generate the variates (see whuber's code here - Generating samples from singular Gaussian distribution)
Modifying whuber's code to suit your problem, we have:
library(MASS)

# Specify the mean vector for the multinormal distribution
mu <- c(10,20,30,40)

# Use equal variances, which leads to equal correlations
# of -1/3 among all pairwise combinations
sigma <- matrix(c(3.0,-1,-1,-1,-1,3.0,-1,-1,-1,-1,3.0,-1,-1,-1,-1,3.0),4,4)    # Covariance matrix is singular

n <- dim(sigma)[1]
s <- svd((sigma + t(sigma))/2) # Guarantees symmetry
s$d <- abs(zapsmall(s$d))
m <- sum(s$d > 0)

# Generate normals in the lower dimension
n.sample <- 100000   # Number of realizations
x <- matrix(rnorm(m*n.sample),nrow=m)

# Embed in the higher dimension and apply square root
# of sigma obtained from its SVD
x <- rbind(x, matrix(0, nrow=n-m, ncol=n.sample))
y <- s$u %*% diag(sqrt(s$d)) %*% x + mu

# Prepare data for plotting
library(data.table)
x1 <- data.table(y[1,])
x2 <- data.table(y[2,])
x3 <- data.table(y[3,])
x4 <- data.table(y[4,])

# Add in label
x1$var <- 'x1'
x2$var <- 'x2'
x3$var <- 'x3'
x4$var <- 'x4'

all <- rbind(x1,x2,x3,x4)

# Plot histograms of the four components
ggplot(all, aes(V1, fill = var)) + geom_density(alpha = 0.9)

# Check a few totals
sum(y[1:4,1])
sum(y[1:4,2])
sum(y[1:4,10])
sum(y[1:4,n.sample])
sum(y)

# Check to see how close the empirical correlations are
# to negative 1/3
cor(y[1,],y[2,])
cor(y[1,],y[3,])
cor(y[2,],y[3,])

Here is the chart:

Here is output verifying the sum requirement for a few realizations and some of the empirical correlations:
> sum(y[1:4,1])
[1] 100
> sum(y[1:4,2])
[1] 100
> sum(y[1:4,10])
[1] 100
> sum(y[1:4,n.sample])
[1] 100
> sum(y)
[1] 1e+07
>
> # Check to see how close the empirical correlations are
> # to negative 1/3
> cor(y[1,],y[2,])
[1] -0.3294876
> cor(y[1,],y[3,])
[1] -0.331753
> cor(y[2,],y[3,])
[1] -0.336657

If you want to specify non-equal variances among your components, you will have a different covariance matrix. You will also not have equal pairwise correlations.
Part 2 - Unequal Variances
Suppose we want variances of 2, 3, 5, and 6.
If you use the following covariance matrix,
sigma <- matrix(c(3.7838,-0.4865,-1.3514,-1.9459,-0.4865,7.9054,-3.0405,-4.3784,-1.3514,-3.0405,16.5541,-12.1622,-1.9459,-4.3784,-12.1622,18.4865),4,4)    # Covariance matrix is singular

You will get the following chart:

If this is of interest, I will post more on how to find the covariance matrix.
Comment: Thanks for the answer. I will try it out and then report back.


Second order differential equation with constant second member $-g$

Question: I would like to solve this differential equation :
$$\ddot z(t)+A\dot z(t)=-g$$
where $g$ is constant.
I know that the solution of :
$$\ddot z(t)+A\dot z(t)=0$$
is $z(t)=\alpha + \beta e^{-At}$ but I can't find the particular solution...
Comment: Why not just choose a particular solution of the form $$z_p(t) = k t$$ Multiplying by $t$ accounts for the constant solution in the homogeneous solution. Then, substitute and solve for $k$.
Comment: sorry I made a mistake. I edit my equation.
Answer: Let $u(t)=\dot z(t)$ then your ODE is of first order with respect to $u$:
$$\dot u(t)+A u(t)=-g$$
where the solution is $u(t)=\beta e^{-At}-\frac{g}{A}$ (the particular solution is easy to find here).
Finally, in order to find $z$, we integrate $u$ with respect to $t$:
$$z(t)=\int u(t)\,dt=\int\left(\beta e^{-At}-\frac{g}{A}\right)dt=-\beta e^{-At}-\frac{gt}{A}+\alpha.$$
where $\alpha$ and $\beta$ are arbitrary real constants.


configuration files in version control while deploying

Question: I have a particular problem with config files and deployment. When I deploy my project, it eventually moves to two different branches: staging and production (using SVN here)
I make the branch from my local working copy of the project. What this ends up doing is overwriting staging and development's copy of the config files, so I manually have to go in and make the correct changes every time I deploy.
Does it make sense to delete the config files from SVN completely since it's set in stone? Or is there another way in doing this?
Answer: What I usually do is maintain multiple configuration files in the repository, but not under the actual name (e.g. in a dedicated folder that gets excluded from deployment). My deployment scripts then copy the appropriate config file into the final location. So my deployment routine goes:

checkout
compile
run deployment script
Comment: "deployment scripts then copy the appropriate config file into the final location" can you elaborate on this?
Comment: I use scripts to copy the application from an offline folder to its desired location. These scripts create folder structures and copy files, but they can also perform other tasks. For this particular problem, I put a number of different configuration files into a dedicated folder, and the deployment script copies the appropriate one to the live location where the application expects the configuration file.
Answer: You don't want to delete the files from SVN - since that will cause it to delete them from the server when you deploy.
SVN offers a feature called svn: ignore that you can use to exclude one or multiple individual files from version control or match a pattern of filenames to ignore. Basically, ignoring files means that SVN will leave them as they are on each working copy or deployment server, so they can be different in each place.
There are two ways to use ignore from the command line.
A: Edit the ignore properties for a directory.

Use cd trunk/blah to change to the directory that contains files you want to ignore.
Use svn propedit svn:ignore . to edit the ignore properties for this directory. This will open a text editor.
In the file that opens, add or modify lines which will represent the file(s) you want to ignore. For example, adding a line of *.txt will cause SVN to ignore all files ending in .txt. Adding a line of config.php will cause SVN to ignore the config.php file.
Save and exit the file.
Commit the property change. svn commit -m "New Ignores"

B: Use a single command to set the SVN ignore property on a directory.

Use cd trunk/blah to change to the directory that contains files you want to ignore.
Use svn propset svn:ignore config.php . to set the SVN ignore property to ignore the config.php file.

Commit the property change. svn commit -m "New Ignores"

Good introductory article on svn:ignore.
"Official SVN documentation."


A interesting question on Skew-symmetric matrix...finding the determinant.

Question:
Let $a_1,a_2,\cdots ,a_{2n}$ be complex numbers. We construct a $2n \times 2n$ matrix, say $A$ which is skew symmetric and entries are from complex numbers.
  $A=(\alpha_{ij})$, where $\alpha_{ij}=a_ia_j$ for $i<j$. To find the determinant of the matrix $A$.

Since $A$ is a even order skew-symmetric matrix, we have determinant of $A$ a perfect square.

My Intuition: $\det A = a_1^2 \times a_2^2 \times \cdots \times a_{2n}^2$.

I was trying to see what happens when $n=2$, i.e. we have $4 \times 4$ matrix $A$.
Thus we have complex numbers $a_1,a_2,a_3 \  \text{and} \ a_4$ and \
$A=
\begin{bmatrix}
      0 & a_1a_2 & a_1a_3 & a_1a_4 \\
    -a_1a_2 & 0 & a_2a_3 & a_2a_4 \\
    -a_1a_3 & -a_2a_3 & 0 & a_3a_4\\
     -a_1a_4 & -a_2a_4 & -a_3a_4 & 0
\end{bmatrix}
$
We can see
$
A=
\left[
\begin{array}{c|c}
D_1 & B \\
\hline
-B^T & D_2
\end{array}
\right]
$, where
$D_1 = \begin{bmatrix}
0 & a_1a_2\\
-a_1a_2 & 0
\end{bmatrix}
$,
$D_2 = \begin{bmatrix}
0 & a_3a_4\\
-a_3a_4 & 0
\end{bmatrix}
$ and
$
B = \begin{bmatrix}
a_1a_3 & a_1a_4\\
a_2a_3 & a_2a_4
\end{bmatrix}
$
Also I have noted that $\det B =0$.

Can we use the result of determinant of block matrices? Can someone shed some light how to do the problem?

Let
$
D = \text{diag}(a_1,a_2,\cdots,a_{2n})$ , and $C = (c_{ij})$, where $C$ is a skew symmetric matrix with $c_{ij} = 1$ when $i<j$. Then one can easily see that $A=DCD$. So we are let to prove that $\det C =1$.
Comment: Have you tried to find the eigenvalues of $C$? $C$ is real skew-symmetric, so all eigenvalues are pure imaginary. They're the roots of real polynomial, so they come in conjugate pairs. Perhaps if you look at the first 2 or 3 examples, you may find a pattern.
Answer: Yes, the determinant is $a_1^2 \cdots a_{2n}^2$. To see this, notice that if you divide the $i$'th row by $a_i$ for all $i$, and then divide the $i$'th column by $a_i$ for all $i$, then you get a matrix with entries in $\{0,1,-1\}$ whose determinant is easily seen (do some row-reduction!) to be 1.
Comment: When you say *is easily seen to be equal to $1$*, can you elaborate? I tried and I don't find it easy. The easy part is the one you cover factoring $a_1^2 \dots a_{2n}^2$.
Comment: Subtract row 2 from row 1. Then subtract row 3 from row 2, etc.  Then the first row becomes (1 1 0 ... 0) and the second row becomes (0 1 1 0 ... 0), the second last is (0 .. 0 1 1) etc.
Comment: Then at the end, add rows 1, 3, 5,... to the last row so that it becomes (0 ... 0 1).  Then you have an upper triangular matrix with 1's on the diagonal.


Checar se mais abas de site X está aberta

Question: Existe a possibilidade de checar se meu site está aberto mais de uma vez no navegador do usuário?
Meu site exige que seja usada apenas uma aba para nao ocorrer error de comunicação, por isso preciso alertar o usuário quando ele abrir mais de uma aba do meu site, existe essa possibilidade com JavaScript?
Comment: Por favor evitem longas discussões nos comentários; a conversa de vocês foi [movida para o chat](http://chat.stackexchange.com/rooms/54619/discussion-on-question-by-user3163662-checar-se-mais-abas-de-site-x-esta-aberta)
Answer: Como sugerido nos comentários, pode-se usar local storage/session storage para fazer essa verificação. Vou dar um exemplo bem simples:

Verifica se existe uma contagem; se não existe, começa do 1, caso contrário incrementa a que está lá;
Ouve por eventos no storage; se a contagem mudou, é porque tem outra página tentando alterá-la ao mesmo tempo;

Se a página que está tentando alterá-la "chegou primeiro" (i.e. possui uma contagem menor), então ela é quem manda. Retira-se...
Se a página que está tentando alterá-la "chegou depois" (i.e. possui uma contagem maior), então a página atual é quem manda. Reseta a contagem para seu próprio valor.

var contagem = +localStorage.getItem("contagem"); // Antigo dono
window.addEventListener("storage", storageChanged, false);
localStorage.setItem("contagem", contagem+1); // Tenta se tornar o novo dono

function storageChanged(event) {
    if ( event.newValue <= contagem )    // Se o antigo dono ainda estiver por aí
        alert("Já tem uma aba aberta."); // Vai embora
    else                                              // Senão
        localStorage.setItem("contagem", contagem+1); // torna-se o novo dono
}

Se executar o código acima, não vai acontecer nada. Mas se abrir essa resposta numa nova aba e mandar executar o código, o alerta será exibido (Parece que não funciona no StackSnippets, o código está num iframe sandboxed que não permite allow-same-origin... No jsFiddle funcionou, entretanto...).
(Nota: quando uma página altera o storage, um evento é disparado em todas as outras páginas na mesma origem. A própria página que alterou não recebe evento algum.)
Altere a parte onde há o alert para fazer alguma outra coisa (tipo fechar a aba com window.close). Pode-se aprimorar bastante o código acima - por exemplo, fazendo com que quando a aba principal for fechada ela passe o comando para a próxima na fila (supondo que as demais permaneçam abertas, somente inativas) - mas aí já fica um pouquinho mais complexo...
Dito isso, pro seu caso particular sugiro tratar desses "erros de comunicação" em vez de exigir que somente uma aba esteja aberta - se o usuário tentar acessar sua conta em dois browsers diferentes, por exemplo, o que aconteceria? Melhor projetar o sistema de modo que duas abas abertas não causem problemas, nem que pra isso tenha de contar com a ajuda do servidor.


Regarding BigDecimal

Question: i do the below java print command for this double variable
double test=58.15;
When i do a System.out.println(test); and System.out.println(new Double(test).toString()); It prints as 58.15.
When i do a System.out.println(new BigDecimal(test)) I get the below value
58.14999999999999857891452847979962825775146484375
I am able to understand "test" double variable value is internally stored as  58.1499999. But when i do the below two System.out.println i am getting the output as 58.15 and not 58.1499999.
System.out.println(test);
System.out.println(new Double(test).toString());
It prints the output as 58.15 for the above two.
Is the above System.out.println statements are doing some rounding of the value 58.1499999 and printing it as 58.15?
Answer: System.out.println(new BigDecimal("58.15"));

To construct a BigDecimal from a hard-coded constant, you must always use one of constants in the class (ZERO, ONE, or TEN) or one of the string constructors.  The reason is that one you put the value in a double, you've already lost precision that can never be regained.
EDIT: polygenelubricants is right.  Specifically, you're using Double.toString or equivalent.  To quote from there:

How many digits must be printed for
  the fractional part of m or a? There
  must be at least one digit to
  represent the fractional part, and
  beyond that as many, but only as many,
  more digits as are needed to uniquely
  distinguish the argument value from
  adjacent values of type double. That
  is, suppose that x is the exact
  mathematical value represented by the
  decimal representation produced by
  this method for a finite nonzero
  argument d. Then d must be the double
  value nearest to x; or if two double
  values are equally close to x, then d
  must be one of them and the least
  significant bit of the significand of
  d must be 0.
Answer: Yes, println (or more precisely, Double.toString) rounds. For proof, System.out.println(.1D); prints 0.1, which is impossible to represent in binary.
Also, when using BigDecimal, don't use the double constructor, because that would attempt to precisely represent an imprecise value. Use the String constructor instead.
Comment: @arav: No they are not the same. The string constructor does not loose precision but the double one already has lost it. That is, the double is already 58.149999 when you call the constructor. Also, it does not do any rounding, it uses the raw bits to construct exactly what you give it.
Comment: `Double.toString(1.0000000000000001D).equals("1.0")`, precision is lost due to rounding.
Comment: I am assuming  new BigDecimal("52.15") or  double test=52.15;
new BigDecimal(new Double(test).toString())  are one and the same. Pls correct me if i am wrong.  My understanding is  new Double( double variable).toString takes the rounded value 58.15 and not 58.149999999.  Does it do any rounding?  So the results are same if i use either way?
Comment: Yes, `Double.toString(.1D).equals("0.1")`, and yes, rounding did occur here. `.1D` is not exactly `0.1`, but `Double.toString` rounds to it. This is a "two wrongs make it right" kind of scenario which you should not rely on.
Comment: Opps, yes I misread that comment and mixed up with what you were saying in your answer and then OP. In the OP the issue is that he is using a double directly in the BigDecimal constructor and that is the problem.
Comment: poly, so your saying it will give me the same output if i use new BigDecimal(Double.toString(doublevariable))  instead of  Bigdecimal("52.15").  if its going to give me the same output then it's a workaround right?   In what cases this might fail?
Answer: out.println and Double.toString() use the format specified in Double.toString(double).
BigDecimal uses more precision by default, as described in the javadoc, and when you call toString() it outputs all of the characters up to the precision level available to a primitive double since .15 does not have an exact binary representation.

Regarding BigDecimal


Obtain an expression for the error in $ y = x^2 - 1/x^2$. Find the error bound for $y$ when $x = 0.76$

Question: Here's my working:
$y = \left(0.76\right)^2-\left(\frac{1}{\left(0.76\right)^2}\right) = -1.1537$
1) Finding Relative Error:
$R.E = 2\left(\frac{5.10^{-3}}{0.76}\right)+\frac{0.5}{1}+2\left(\frac{5.10^{-3}}{0.76}\right) = 0.5263$
2) Finding Absolute Error
$A.E. = |R.E.| * |y| = 0.5263 * 1.1537  = 0.6071$
3) Finding Error Bounds:
$Error Bound = y +/- A.E. = -1.7608$ to $-0.5466$
but in my book, the answer is: $-1.184$ to $-1.123$
where am i missing?
Answer: Hint
$$y=x^2-\frac 1{x^2}\implies \frac{dy}{dx}=2x+\frac 2{x^3}\implies \Delta y=\left(2x+\frac 2{x^3} \right)\Delta x$$ So, using $x=0.76$ and $\Delta x=5\times 10^{-3}$, $\Delta y= 0.0304$.
Comment: @HQuser. Your solution is perfect ! Probably one more typo in a textbook. Cheers.
Comment: The OP appears to be using error propagation rules rather than calculus.
Comment: @IanMiller. I agree with you but there are so many ways to adress this kind of problems.
Comment: @HQuser. You are very welcome !
Comment: @HQuser I agree with Claude for your second question. The textbook must have a typo.
Comment: @ClaudeLeibovici Thank you for giving me new prospective of solving same problem. I have tried solving the question $y = x-1/x$ when $x=0.24$ and I am getting range $-4.0184$ to $-3.8348$ but my answer on book is: $-3.946$ to $-3.908$. So I assume answer on my book is wrong?
Comment: @ClaudeLeibovici thank you very much!
Answer: The number $1$ has no error associated with it. As such you should not include the $\frac{0.5}{1}$ term in your relative error.
Without it the working would go as follows:
Here's my working:
$y = \left(0.76\right)^2-\left(\frac{1}{\left(0.76\right)^2}\right) = -1.1537$
1) Finding Relative Error:
$R.E = 2\left(\frac{5.10^{-3}}{0.76}\right)+2\left(\frac{5.10^{-3}}{0.76}\right) = 0.0263$
2) Finding Absolute Error
$A.E. = |R.E.| * |y| = 0.0263 * 1.1537  = 0.0303$
3) Finding Error Bounds:
$Error Bound = y +/- A.E. = -1.1840$ to $-1.1234$


What does it mean if an item in Search API has a "-1" flag and is never indexed?

Question: Search API uses flags in the changed column of the search_api_item table to store item-level information relating to indexing. The -1 flag (something to do with queuing items?) appears to be crucial to a problem I'm facing, where it is set on items which are then never indexed.
I've studied the code and all the documentation and relevant issues I can find, and I can't figure out in nearly enough detail what the practical implications of this flag are: what it being set causes or prevent from happening (and therefore, where I should focus next in trying to solve my problem). Since Search API is an API module which plenty of developers work with, and these flags are a core part of its logic and an important part of the contrib-accessible data it records, I'm hoping that someone will be able to help explain what a -1 flag means and what it causes to happen (or what it prevents).
For context, the specific problem spurring me to ask (with Search API using Search API Solr on Drupal 7) is that indexing stops abruptly - cron runs continue to work fine, cron requests reach the search server with no problem, but a large portion of my indexable items are never indexed. There are no error messages or relavant information in any of the appropriate logs (drupal watchdog, cron, apache solr, tomcat). It's a classic debugging nightmare: something is failing silently and leaving no trace. But there's one clue: in the search_api_item table on the database, I can see that, when Cron reaches those items where indexing is failing, the changed column changes to a '-1'. The notes for that column say:

Either a flag or a timestamp to indicate if or when the item was
  changed since it was last indexed.

After much digging in code I found what appears to be the place these flags are set ( in search_api/includes/datasource.inc ) and there appear to be three such flags:

-1, set in trackItemQueued(): the one which looks to be crucial to my problem. According to the comments it means "queued" - but I can't find the crucial details of what this causes to happen (or prevents from happening) next. If I've understood this right, then this flag being set should be a sign that everything is working fine: but the items aren't indexed, and I can't see what the expected next step should be.
0, which apparently means already indexed (presumably causing these items to be passed over)
1, set in trackItemInsert(), and I can't figure out anything about what this one means
(and timestamps apparently signify "changed/dirty", which I imagine means "no longer matches the index data as of [timestamp], needs to be queued for re-indexing")

Search API is an API module, so there will be a good number of Drupal developers who know how it works on the inside. Do you know what the -1 flag indicates - in terms of what then happens to items flagged ("queued", or, "marked to be queued") in this way? What is the role of this flag in the process and what happens next?

(bonus points for any expert advice on the next step to take in debugging a problem like mine: but the focus of my question is general, about understanding the Search API item flags. My issue and attempts at debugging it are intended mainly as context. In case it's relevant further context, I'm also running Ultimate Cron and its dependency Background Process to handle the cron runs, to reduce search server strain by allowing lots of small indexing batches called by cron every few minutes without overdoing other drupal cron functions. This part of the process seems to work fine: the search server is reached perfectly in sync with the Ultimate Cron schedule as expected. It's what happens after the item flags are checked and updated that is leaving me stumped)

(I've also posted a related support request on the Search API issue queue, without a response. I'm judging it's okay to ask here as well because a) this meta discussion explicitly says that also posting questions that are support questions also asked in issue queues here as well is okay and also a desirable way to take the pressure off busy module maintainers, and b) this is particularly true for Search API as the maintainer has said in the past that he doesn't have much time available to help with support issues. As adivsed on Meta, I'll update both threads with any relevant information)
Answer: -1 means that it's added to the queue system. By default, that means the queue table in the database. It should create a queue task for groups of id's matching your batch size.
If they aren't run, this either means that the queue system does not run at all (It should be executed in the default cron implementation) or that some other stuff is filling up the queue.
The following query should give you an idea of what's in the queue: SELECT name, COUNT(*) FROM queue GROUP BY name.
Some hints:

There was a quite major bug in the update.module that caused it to create possibly thousands of duplicate queue tasks. If you can see many fetch_tasks with the above query, you can delete them and clean the cache_update table to have them re-added once. Make sure you're running an up to date Drupal core version to prevent this from re-appearing.
Maybe one of your cron implementations is running into a timeout or fatal error. You could try a module like Elysia cron to get detailed stats about how much time each cron implementation uses. It probably is something that is run after search_api.module because it seems to be able to create the queue tasks. xmlsitemap.module is a candidate that can take a huge amount of time if you have many nodes.
You're using an alternative cron module, that is not properly executing queue tasks.

You can also try http://drupal.org/project/queue_ui, it provides an UI to manage the default queue implementation and provides the same information as the above query, but it's not actively maintained and broken without applying a patch from the issue queue.
Comment: No, -1 does not mean "now", there is no time involved. It just means it has been added to the queue at some point, when it will be indexed, it will be set to the changed timestamp. I don't know about specific problems with the modules that you listed, and I would only look into that when you confirmed that it's not something else (e.g. by confirming that there are only search api related tasks in the queue).
Comment: Great, many thanks for these leads, I'll investigate. Just to make sure I'm not misunderstanding the timing, is this right: -1 being set means it **has just now** been added to the standard Drupal queue?
Comment: Oh and one more clarification, I'm not sure whether with your third bullet, "You're using...not properly executing", you mean this as a possibility to check for, or a definite statement about a known problem with Background Process and/or Ultimate Cron. Do you have specific knowledge of these modules? I believe them to have similar features to those of Elysia Cron and Queue UI that you mention, but I may be niavely expecting too much from them. Thanks again!


Call parent constructor before child constructor in PHP

Question: I was wondering if its possible to call the parents __construct(), before the child's __construct() with inheritance in PHP.
Example:
class Tag {
     __construct() {
         // Called first.
     }
}

class Form extends Tag {
     __construct() {
         // Called second.
     }
}

new Form();

Ideally, I would be able to do something in between them. If this is not possible, is there an alternative, which would allow me to do this?
The reason I want to do this is to be able to load a bunch of default settings specific to the Tag that Form can use when __construct() is called.
EDIT: Sorry forgot to add this.. I'd rather not call the parent class from the child class. It's simply because it exposes some private data (for the parent) to the child, when you pass it as an argument
This is what I want to do:
$tag = new Tag($privateInfo, $publicInfo);
$tag->extend(new Form()); // Ideal function, prob doesn't work with inheritance.

Tag.php
class Tag {
     private $privateInfo;
     public $publicInfo;
     __construct($private, $public) {
         $this->privateInfo = $private;
         $this->publicInfo = $public;
     }
}

Form.php
class Form extends Tag {

     __construct() {
         echo $this->publicInfo;
     }
}

Make sense?
Thanks!
Matt Mueller
Comment: Private data of a parent class won't be exposed to any subclasses. Public or protected data will, private won't.
Comment: Can you elaborate on what you mean by this *'exposes some private data (for the parent) to the child'*?
Comment: Yah, but if the (private) parameters goes through the child constructor first, the child has access to them. I editted it to clarify.
Comment: Why is this a problem?  Simply have your child constructor pass through the arguments it doesn't care about and ignore them.
Comment: Because other people will be extending the Tag and I don't want all this garbage coming through the child class.
Comment: Either the constructor is called or it isn't. You seem to want it to be called but not do its intended job. Maybe magic rather than programming should be your next port of call.
Answer: Just call parent::__construct in the child.
class Form extends Tag
{
    function __construct()
    {
        parent::__construct();
        // Called second.
    }
}

Comment: Ahh sorry. I forgot to add that I would rather not have that in the child's constructor. Please read edit.
Comment: Decided that this was the best of the alternatives. Unfortunately, its not exactly what I wanted - oh well.. Thanks!
Comment: Oh wow Denis, you have been really helping me out the last couple days.. Thanks man!
Answer: yeah just call parent::__construct() in your construct
Comment: This is the way to do, simply read OOP manual if you don't agree. Don't try to reinvent the wheel. If you have concern regarding your model, try to post that too, to see how you did. Probably you are doing a bad design technique.
Comment: Ahh sorry I forgot to include a requirement... Please read the edit for update.
Comment: I'm not trying to reinvent the wheel... I'm trying to get a solution to my requirements.
Comment: I reread your edited question, but still don't understand what problems you have. What data is visible to the child class from parent? Maybe you should post a bad example, with some echo to see what do you see wrong.
Answer: Yes, but only internally (i.e., by writing a PHP extension), so if I were you I'd settle with calling parent::__construct(). See this section on the PHP wiki.
Sorry, PHP is not Java. I think not requiring (implicitly or explictly) the super constructor to be called was a very poor design decision.
Answer: From the sounds of it you may want to rethink your design so that you don't need to pass the parameters in the constructor. If you don't think it can be done, ask it as a question, you might be surprised by some of the suggestions.
The child class has the ability to override the parent constructor without calling it at all. I would recommend having a final method in the parent class. That way everyone knows you don't want this being overriden, and any inherited class (rightly) has access to do whatever it wants in the constructor.
class Father {
    private $_privateData;
    final function setPrivateData($privateData) {
        $this->_privateData = $privateData;
    }
}

Another, not recommended, more "reinventing the wheel", solution would be to define a function in the parent class, say _construct(), that's called in its own construct. Its not really clear, doesn't use language features/constructs, and is very specific to a single application.
One last thing to keep in mind: you can't really hide information from the child class. With Reflection, serialize, var_dump, var_export and all these other convenient APIs in the php language, if there is code that shouldn't do anything with the data, then there's not really much you can do asides from not store it. There are libraries and such that help create sandboxes, but its hard to sandbox an object from itself.
Edit: Somehow I missed Artefacto's answer, and I suppose he is right (I've never tried writing an extension to do that). Still, implementing it breaks developer expectations while making it harder to actually see code to explain what's going in.
Comment: Hey,  thanks for your response. Unfortunately this topic got a little out of hand, which is my fault. Anyways, I am trying to redesign it. I have a question about your answer... how can you do this - "The child class has the ability to override the parent constructor without calling it at all."?
Comment: @Matt You can't using just PHP code. The person writing the derived class has the ability to not call the parent constructor at all. What you could do for a project is have another function defined that gets called by the parent class in its constructor, say _construct() which the parent defines and calls.


How can i close the connection(req) once i received the response from api call Node js

Question: var req = http.request(options, function(res) {
var msg = '';
res.setEncoding('utf8');
res.on('data', function(chunk) {
msg += chunk;
});
res.on('end', function() {
//

});
res.on('error', function() {

});

});
req.write(data);

req.end();

I have to close the connection once the api call successfully completed in node js
Answer: res.on('end', function() { req.end() });
https://nodejs.org/api/http.html#http_request_end_data_encoding_callback
Comment: req.end()-its not closing the connection in our case.i am facing ETIMEDOUT error while calling api.How to close the connection properly


If x is in the derived algebra, show that Tr(ad a)=0

Question: Let $L$ be a Lie algebra, and let $a\in [L,L]$. How to prove that $trace(ad_a)=0$?
Comment: This is a recent duplicate. Give me a second.
Comment: @QiaochuYuan:  Were you perhaps thinking of your answer to [this Question](http://math.stackexchange.com/questions/1632814/computation-of-killing-form)?
Comment: @hardmath: no, someone asked this exact question recently. It might've been deleted though.
Comment: $\operatorname{tr}(AB)=\operatorname{tr}(BA)$, and the fact that $ad:L\to\mathfrak{gl}_n$ is a homomorphism.
Comment: @QiaochuYuan I also remember that we have answered the same question recently, and I think, also posted by the same user (who might have deleted the question along with both answers).
Comment: I don't remember that I asked this question before.. And I didn't delete any question. It is not possible actually to delete a question with answers
Comment: Ronald: sorry, I might be mistaken. But the question definitely was there a few days ago, with tag "Lie algebras". Can you find it ?
Comment: No problem. But you may mean this question .. Which is asked by me and still active http://math.stackexchange.com/questions/1625573/3-dim-simple-complex-lie-algebra
Comment: No, I mean the very same question. My answer was: $ad(L)=ad([L,L])=[ad(L),ad(L)]$. It might have been a user who was removed upon request.
Comment: I think your answer is nice, why you don't add it here . interested people can see different answers
Comment: Ronald, thanks for the compliment! I cannot add an answer, because the question is on hold.
Answer: Hint: If $x=[a,b]$, Jacobi implies that $ad_x=ad_a(ad_b)-ad_b(ad_a)$. Since $trace(fg)=trace(gf)$ the result follows.


Add followers' list at the end of discussion

Question: I want to add following functionalities in Drupal 7.

Any forum-user can have followers, So I want to show followers list on forum-user's profile page. (Like this)
Those followers should be automatically subscribed to all the posts by that author.

I'm using Advanced Forum module. How to do this using Views module or any other possible way?
Answer: You can do this using Flag module.

Flag is a flexible flagging system that is completely customizable by
  the administrator. Using this module, the site administrator can
  provide any number of flags for nodes, comments, users, and any other
  type of entity. Some possibilities include bookmarks, marking
  important, friends, or flag as offensive. With extensive views
  integration, you can create custom lists of popular content or keep
  tabs on important content.

Creating Following & Followers
Answer: @NishantP's link helped me to understand a bit, but this post solved the issue perfectly. Its Author has elaborated nicely.
Modules used : Views, Flag, CTools
Comment: While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes.


Documentation for default configuration for MapReduce jobs

Question: It is stated in books that default input format for MapReduce jobs is TextInputFormat. I cannot find any official documentation that lists the default configuration for MapReduce jobs. It is not available in Job or Mapper class documentation.
I request for reference to some documentation. Thanks!
Answer: Here's a page for some default configuration values:

http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

The default value for the input format is actually defined in the code (if a configuration value hasn't been set, the fall back or default is TextInputFormat, as can be seen in JobContext class, Line 185 (for v1.2.1):

http://svn.apache.org/viewvc/hadoop/common/tags/release-1.2.1/src/mapred/org/apache/hadoop/mapreduce/JobContext.java?view=markup


tracert.exe: Why doesn't RTT increase with each hop?

Question: Internet is slow today. So, I ran tracert.exe against a nearby server (German edition of Windows XP):
>tracert inka.de

Routenverfolgung zu inka.de [188.8.131.52]  über maximal 30 Abschnitte:

  1     2 ms     2 ms     9 ms  speedport.ip [192.168.0.17]
  2  2685 ms  2508 ms  2375 ms  22.214.171.124
  3  1435 ms  1680 ms  2103 ms  220.127.116.11
  4  1302 ms  1729 ms  1856 ms  f-ed5-i.F.DE.NET.DTAG.DE [220.127.116.11]
  5  1759 ms  1688 ms  2153 ms  ffm-b12-link.telia.net [220.127.116.11]
  6  1542 ms  1072 ms  1172 ms  ffm-bb1-link.telia.net [220.127.116.11]
  7   978 ms   999 ms   967 ms  ffm-b2-link.telia.net [18.104.22.168]
  8   895 ms   836 ms  1072 ms  belwue-ic-130164-ffm-b2.c.telia.net [184.108.40.206]
  9  1136 ms  1191 ms  1366 ms  Karlsruhe-RZ-1-10GE-0-3-0-2.belwue.net [220.127.116.11]
 10  1448 ms  1842 ms  1577 ms  fe0-553.cix.ka-ip.net [220.127.116.11]
 11  1609 ms  1901 ms  1830 ms  tapac.inka.de [22.214.171.124]

Ablaufverfolgung beendet.

Line two tells me that RTT for an ICMP package between me and 126.96.36.199 is
more than two seconds on average. However, line seven tells me that RTT between
me and ffm-b2-link.telia.net is less than a second, although there are five
more hops.
I assumed that RTT increases with each hop, at least on average.
Is that a false assumption? If not, then what could be the explanation that RTT
drops considerably for some hops?
Answer: It is very possible that the fowarding devices on the path use some kind of route caching. The first time a pdu for a specific destination arrives at a node, the device must discover which interface to forward the pdu from. When the next traceroute pdu arrives (with a higher TTL) then the node does not need to rediscover which interface to forward the pdu to and can forward the pdu much faster.
It is also possible that a similar thing is happening with Address Resolution Protocol (ARP) - the first time a node needs to forward a frame it must request the next hop via ARP, this is cached so that next time a frame for the same next hop is received it can be forwarded quicker.
Comment: If caching were to blame, I would expect one (probably the first) RTT in each row to be considerably higher than the other two. However, this is not the case here. The RTTs vary considerably by row, and in each row, they are about the same.


Child Class names in logger

Question: I have the following code:
class LogMixin(object):
    def __init__(self, log_name, log_level=logging.INFO):
        self.project_name = self.__class__.__name__
        self.log_level = log_level
        self.log_name = log_name
        self.log_file_path = (r"\\Logging\\{}_{}.log".format(self.log_name, datetime.datetime.today().strftime("%Y%m%d")))
        self.logger = self._get_logger()

    def _get_logger(self):
        logger = logging.getLogger(self.project_name)
        logger.setLevel(self.log_level)
        fh = logging.FileHandler(self.log_file_path)
        ch = logging.StreamHandler()
        fh.setLevel(self.log_level)
        ch.setLevel(self.log_level)

        formatter = logging.Formatter('%(name)12s - %(levelname)s - %(asctime)s - %(message)s')
        fh.setFormatter(formatter)
        ch.setFormatter(formatter)
        logger.addHandler(fh)
        logger.addHandler(ch)
        logger.info("START")
        return logger

    def log(self, log_line, severity=None):
        self.logger.log(severity or self.log_level, log_line)

class Database:
    def __init__(self):
        self._conn = pyodbc.connect("")
        self._cursor = self._conn.cursor()

    def query_df(self, sql, params=None):
        self.cursor.execute(sql, params or ())
        return self.fetchall()

class Query(Database, LogMixin):
    def __init__(self):
        super().__init__()
        LogMixin.__init__(self, project_name='final-project')

    def main_query(self):
        self.log('Test query', severity=logging.DEBUG)
        df = self.query_df(sql="")
        return df

class Processing(Query):
    def __init__(self):
        super().__init__()

    def main_df(self):
        df = self.main_query()
        self.log("Beginning processing")
        return df

class Wrangling(Processing):
    def __init__(self):
        super().__init__()

    def data_wrangling(self):
       self.log("Wrangling data")
       df2 = self.main_df()
       return df2

wr = Wrangling()
wr.data_wrangling()

I am attempting to create one log file at the end of the process in this format:

INFO - 2022-10-26 15:27:46,761 - START

Query - DEBUG - 2022-10-26 15:27:46,761 - Test Query

Processing - INFO - 2022-10-26 15:28:46,761 - Beginning processing

Wrangling - INFO - 2022-10-26 15:29:46,761 - Wrangling data

My current approach creates this output:
Wrangling - INFO - 2022-10-26 15:27:46,761 - START

Wrangling - DEBUG - 2022-10-26 15:27:46,761 - Test Query

Wrangling - INFO - 2022-10-26 15:28:46,761 - Beginning processing

Wrangling - INFO - 2022-10-26 15:29:46,761 - Wrangling data

How can I make the names in front of the log message to be specific to the class that it's coming from instead of just the class where the process is started from?
Comment: Your implementation of a logging class is very troublesome. You could have done it without a logging class. Anyway let me write a solution
Comment: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Answer: Your implementation of classes is a bit troublesome. Avoid calling a separate _get_logger and log methods entirely
class LogMixin(object):

    def __init__(self, parent=None, log_name="", log_level=logging.INFO):
        self.project_name = parent.__class__.__name__
        self.log_level = log_level
        self.log_name = log_name
        self.log_file_path = (r"\\Logging\\{}_{}.log".format(self.log_name, datetime.datetime.today().strftime("%Y%m%d")))
        logger = logging.getLogger(self.project_name)
        logger.setLevel(self.log_level)
        fh = logging.FileHandler(self.log_file_path)
        ch = logging.StreamHandler()
        fh.setLevel(self.log_level)
        ch.setLevel(self.log_level)

        formatter = logging.Formatter('%(name)12s - %(levelname)s - %(asctime)s - %(message)s')
        fh.setFormatter(formatter)
        ch.setFormatter(formatter)
        logger.addHandler(fh)
        logger.addHandler(ch)
        logger.info("START")

class Query(Database, LogMixin):
    def __init__(self):
        super().__init__()
        logmix = LogMixin(self)
        self.logger = logging.getLogger(logmix.project_name+'.Query')

    def main_query(self):
        self.logger.info('Test query')
        df = self.query_df(sql="")
        return df

class Processing(Query):
    def __init__(self):
        super().__init__()

    def main_df(self):
        df = self.main_query()
        self.logger.info("Beginning processing")
        return df

class Wrangling(Processing):
    def __init__(self):
        super().__init__()

    def data_wrangling(self):
    self.logger.info("Wrangling data")
    df2 = self.main_df()
    return df2

wr = Wrangling()
wr.data_wrangling()

Will return
NoneType - INFO - 2022-10-26 22:04:21,491 - START
Wrangling - INFO - 2022-10-26 22:04:21,492 - START
Wrangling.Query - INFO - 2022-10-26 22:04:21,492 - Wrangling data

Child Class names in logger


Is this how to convert odds ratio intervals to risk ratios

Question: I have a binary logistic regression with just one binary factor predictor. I would like to report the 'risk ratio' (probability ratio) with a confidence interval. I'm using R 4.0.4 for statistical computing.
Here's some data:
<code>> dat <- data.frame(record=c(rep(1,11), rep(0,50-11), rep(1,20),
             rep(0,50-20)), group=as.factor(c(rep("A", 50),
              rep("B", 50))))
> tapply(dat$record, dat$group, mean)

   A    B
0.22 0.40
</code>
The risk ratio equals 0.40 / 0.22 = 1.82.
A logistic regression (with logit link) returns an odds ratio (OR):
<code>> mod <- glm(record ~ group, data=mydata, family=binomial)
> antilogit(c(coef(mod)[1], sum(coef(mod)))) # probabilities
(Intercept)
       0.22        0.40
> exp(c(coef(mod)[2], confint(mod)[2,])) # OR and confidence interval
Waiting for profiling to be done...
  groupA    2.5 %   97.5 %
2.363636 0.998533 5.826774
</code>
The antilogit function converts logits to probabilities:
<code>> antilogit <- function(x) {
  exp(x)/(1+exp(x))
}
</code>
I can convert the OR to a risk ratio (RR) using the formula (e.g. Zhang & Yu 1998. What's the relative risk? JAMA 280) RR = OR / (1 - p0 + p0 * OR):
<code>> 2.363636 / (1-0.22+0.22*2.363636)
[1] 1.818182
</code>
Converting the OR confidence interval to a RR one is tricky because the intercept (p0) changes with the slope (OR). This is wrong:
<code>> ORINT=exp(confint(mod)[2,]) # OR interval
Waiting for profiling to be done...
> c(ORINT[1]/(1-0.22+0.22*ORINT[1]), ORINT[2]/(1-0.22+0.22*ORINT[2]))
    2.5 %    97.5 %
0.9988554 2.8259379
</code>
I thought to estimate the intercepts for the OR interval using glm():
<code>> dat$group=as.numeric(dat$group)-1 # convert factor to dummy variable
> ORINT=confint(mod)[2,] # log(OR) interval
> # estimate p0 for log(OR) interval
> p0INT=c(coef(glm(record ~ offset(ORINT[1]*group), data=dat,
                   family=binomial)),
   coef(glm(record ~ offset(ORINT[2]*group), data=dat,
    family=binomial)))
> p0INT=antilogit(p0INT) # convert to probabilities
> p0INT
(Intercept) (Intercept)
  0.3101570   0.1377992
> ORINT=exp(ORINT) # convert to OR
> c(ORINT[1]/(1-p0INT[1]+p0INT[1]*ORINT[1]),
    ORINT[2]/(1-p0INT[2]+p0INT[2]*ORINT[2]))
    2.5 %    97.5 %
0.9989875 3.4992996
</code>
What I have calculated is RR = 1.82 with a 95% confidence interval (1.00, 3.50). Notice that these results are quite different to those for OR because p0 is not small (see the equation above: if p0 is small then OR ~ RR).
For comparison, here are some direct risk ratio estimates from a log binomial and a quasipoisson model (e.g. Naimi & Whitcomb 2020. Estimating risk ratios and risk differences using regression. AJE 189):
<code>> mod=glm(record ~ group, data=dat, family=binomial(log))
> exp(c(coef(mod)[1], sum(coef(mod)))) # probabilities
(Intercept)
       0.22        0.40
> exp(coef(mod)[2]) # RR
   group
1.818182
> exp(confint(mod))[2, ] # RR interval
Waiting for profiling to be done...
    2.5 %    97.5 %
0.9989799 3.5578943
> mod=glm(record ~ group, data=dat, family=quasipoisson) # quasipoisson is used to adjust for underdispersion
> exp(c(coef(mod)[1], sum(coef(mod)))) # probabilities
(Intercept)
       0.22        0.40
> exp(coef(mod)[2]) # RR
   group
1.818182
> exp(confint(mod))[2, ] # RR interval
Waiting for profiling to be done...
    2.5 %    97.5 %
0.9940394 3.4519283
</code>
My conversion from OR interval to RR interval seems OK. Any comments? I haven't thought about how this method could be implemented for more complicated models (and it's not a priority now).
Yes, this is a lot of work for a simple dataset (but it's easy to write a function to do the conversion). The reason I want to convert ORs to RRs is because I sometimes encounter complete separation (p = 0 for one of the groups) and then use penalized-likelihood (Firth 1993. Bias reduction of maximum likelihood estimates. Biometrika 80) to obtain sensible estimates. The R packages logistf 1.24 and brglm 0.7.2 do penalized likelihood but will not accept a log link (and can't estimate RR directly). The R package brglm2 0.7.1 does penalized likelihood and integrates well with the glm() function (it can accept a log link) but doesn't compute profile penalized-likelihood intervals (and Wald tests and intervals for logistic regression are quite unreliable). The R package brglm2 0.7.1 also will not accept quasipoisson. I also had a look at the probratio() function in the R package epitools 0.5.10.1, which converts OR to RR but returns Wald intervals.
I could have simply used a chi-squared test or prop.test() to estimate differences:
<code>> tab=table(dat)
> tab
      group
record  0  1
     0 39 30
     1 11 20
> chisq.test(tab, correct=FALSE)

        Pearson's Chi-squared test

data:  tab
X-squared = 3.7868, df = 1, p-value = 0.05166

> prop.test(c(11,20), c(50,50), correct=FALSE)

        2-sample test for equality of proportions without continuity correction

data:  c(11, 20) out of c(50, 50)
X-squared = 3.7868, df = 1, p-value = 0.05166
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.357828257 -0.002171743
sample estimates:
prop 1 prop 2
  0.22   0.40
</code>
However, RR has been reported in previous work and is the most informative measure of the contrast between groups for my study.
Answer: The above proposal is statistical overkill.
I finally discovered that unconditional score statistic methods can be applied to the difference in two independent binomial proportions AND relative risk AND the odds ratio. Unconditional score statistic methods have been demonstrated to perform well for all of these estimates, e.g.  Agresti & Min 2002, Biostatistics 3, Newcombe 1998 Stat. Med. 17.
Unconditional score statistic methods (and others) can be computed using the PropCIs R package. I found that my logistic regression riskratio intervals were more conservative (wider than) those from the riskscoreci function in PropCIs.
Answer: The OP started with odds ratios and I assume a logit link best fits the data (log links are unlikely to fit as well in general because they require constraints to keep probabilities legal).  Once you recognize that risk ratios are covariate-dependent (covariate-specific) it's easy.  Just pick settings for all the covariates except for the one you are changing, let the one change, computed estimated risks directly from the logistic model, and take ratios.  Getting uncertainties is the only hard part.   For Bayesian models, posterior intervals for covariate-specific risk ratios may be computed exactly from the posterior draws of the underlying log-odds-scale parameters.   In the frequentist domain you could use the bootstrap or the $\delta$ method.
The OP started with the no-covariate case but it's best to learn more general methods IMHO.  For one thing there is likely to be risk heterogeneity due to baseline variable distributions, and simple ratios were intended only for the homogeneous case.


Laravel 5 Use standard view as an email template

Question: I've a Laravel 5.0 project, that I need to send emails to users (such as registration confirmation mail, ... etc).
My problem is I want to create a view (that contains the header and footer of my App), and I want to use this view's html output as a template for the email message body (I need HTML output so I can get Full URLs for images and links to be sent in the email body).
My Question is, Is there any method that can work like curl, when I pass the view URL, This method grab the HTML source of this view?
I don't want to use custom packages for that, I need a native solution
Answer: Create basic email template in resources/views/mails/template.blade.php
header html...

@yield('content')

footer html...

And then in particular emails, for example resources/views/mails/confirmation.blade.php
@extends('mails.template')

@section('content')
    content of confirmations email...
@endsection

To store html of rendered view in a variable you can do
$view = View::make('mails.confirmation');
$html = $view->render();

Comment: Thank you so much, those two lines are what's needed $view = View::make('mails.confirmation'); $html = $view->render();


Trial Solutions for Differential Equations

Question: I have a question from my Differential Equations & Linear Algebra class. When you're trying to find the general solution to an nth order linear non-homogeneous differential equation, you have to find a trial solution to solve it (at least until you get to variation of parameters later in the same chapter) and I assume that the lack of information is due to people usually preferring variation of parameters, but can someone just explain to me the general way to find the form of a trial solution for a problem? Say you have the equation:
<code> y''-y'-2y = 5e^(2x)
</code>
I know the complementary solution would be
<code> Yc = c1*e^(-x)+c2*e^(2x)
</code>
when you're treating the left-hand side as a homogeneous equation, but then once you get to the right hand side, I'm pretty lost here. I know it has the annihilator (D - 2) where D is the differential operator, but how do I turn that into the trial solution? Sorry for the length, I just wanted to provide as much information as possible for accuracy. Thanks in advance everyone.
Answer: The equation is
$$
         (D-2)(D+1)y = 5e^{2x}
$$
$D-2$ annihilates the right side, which gives
$$
                 (D-2)^{2}(D+1)y = 0.
$$
Applying $D-2$ introduces additional solutions, but the general solution of the second equation has the form
$$
                     y=Ae^{2x}+Bxe^{2x}+Ce^{-x}
$$
Plug this back into the original equation, and notice that $(D-2)(D+1)$ annhilates the terms with $A$ and $C$. So $B$ is determined by the equation
$$
               (D-2)(D+1)Bxe^{2x}=5e^{2x}
$$
A convenient trick is to write $D+1=D-2+3$:
\begin{align}
          (D-2)(D+1)Bxe^{2x}&=\{(D-2)^{2}+3(D-2)\}Bxe^{2x}\\
          &=3(D-2)Bxe^{2x}=3(e^{2x}De^{-2x})Bxe^{2x} \\
          &=3Be^{2x}=5e^{2x} \implies B=5/3.
\end{align}
The general solution is
$$
                y=Ae^{2x}+\frac{5}{3}xe^{2x}+Be^{-x}.
$$
Comment: So can you just take the right hand side, drop the constant, replace it with an "A" and that be the trial solution for all cases? Because it seems so simple here, but I want to make sure that isn't an oversimplification.
Comment: @RoryHector : The solution of the original equation must be found among the general solutions of $(D-2)^{2}(D+1)y=0$, even though there may be more solutions of this last equation than of the original, which is why you have to plug back in to the original equation and solve for some constants. Any remaining (undetermined) constants are allowed to be anything.
Answer: Lets try a particular solution of the form $Axe^{2x}$.
$y_p(x) = Axe^{2x}$
$y_p'(x) = A e^{2x} + 2Axe^{2x}$
$y_p''(x) = 2A e^{2x} + 2Ae^{2x} + 4Axe^{2x} = 4Ae^{2x} + 4A xe^{2x}$
$y_p'' - y_p' -2y_p = (4A - 2A - 2A) xe^{2x} + (4A - A)e^{2x} = 3Ae^{2x} = 5e^{2x}$
After which I can solve for $A$, and declare the general solution:
$y = C_1 e^{-x} + C_2 e^{2x} + \frac{5}{3} xe^{2x}$.

The intuition behind this is the following: normally I'd try $e^{2x}$ for the form of the particular equation, but this is already a solution to the homogeneous one. When the characteristic equation admits roots with multiplicity $>1$, the standard ansatz is $e^{ax}, xe^{ax}, \dots$, so let's try that, and see where we get. In this case this works pretty well. I'm sure someone else can point to exactly how general this can get, but it certainly will work for equations of the form above. There's an obvious extension, that the initial ansatz should be $Ax^n e^{ax}$, where both $n$ and $A$ are to be determined.


Is it valid to simulate a shot noise Poisson process with discretization (R)

Question: I have a two-factor shot noise process as follows (sorry for the picture of text):

I want to simulate from this process. Is it valid to do so in a discretized way? The solution to these SDEs is a compound Poisson process, so I simulated it as follows:
<code>shot_noise <- function(T, n, alpha, lambda, gamma, x0){
  dt = T/n
  dN = rpois(n, lambda*dt)
  x = c(x0)
  j = rep(0, n)
  for (i in 2:(n+1)){
    jumps = sum(rexp(dN[i-1], rate = gamma)*exp(-alpha * runif(dN[i-1], max = dt)))
    j[i-1] = jumps
    x[i] = x[i-1] - alpha*x[i-1] + jumps
  }
  return(x)
}
</code>
This would be the code to simulate x_+ and x_-, given the parameters. It seems correct to me, and the output looks like what I think it should. However, the other examples of simulating compound Poisson processes that I see tend not to discretize. I am doing so because this simulation will be used to test whether maximum likelihood estimation for this model is valid, and the observed data is discretized in that case ($\epsilon_t$ at each time-step).
Is the method I've used valid and correct? Is there a better way?
Comment: Your simulation seems nearly good, but I think that the "skeleton
part" of the transition is $x(t_i) = e^{-\alpha \Delta} x(t_{i-1})$
where $\alpha$ (an inverse time) is $a$ in equations, and $\Delta$
stands for the time interval `dt` in code.  Your question could be
much simplified by using only one process $x(t)$ (as in the
simulation) and possibly the notation $x(t) = \sum_{\{i: \,S_i \leq
t\}} \eta_i$ where the $S_i$ are the Poisson arrivals.  Note that
$[x(t)]_t$ is continuous-time Markov, and $[x(t_i)]_i$ is
discrete-time Markov both with gamma margin.
Comment: Sorry I missed the impluse in my formula $x(t) = \sum_{\{i: \,S_i \leq t\}} e^{-\alpha (t- S_i)} \eta_i$.
Comment: @Yves Thank you! I modified that line in the code to:
`x[i] = x[i-1]*exp(-alpha*dt) + jumps`

I should have also mentioned, in my case, the jump times are latent, which is where the uniform RV appears (using EM algorithm)
Comment: Conditional on their number $n_{i-1} := N(t_i) - N(t_{i-1})$, the
successive arrivals $S_k$ falling in the interval $(t_{i-1},\,t_i)$
are such the r.vs $S_k -t_{i-1}$ have the same distribution as order
statistics of a sample of the the uniform distribution on
$(0,\,\Delta)$. This is a classical property of the PP. Note that
your code works when $n_{i-1} =0$ beacuse `sum(numeric(0)) == 0`.
Comment: I appreciate your exposition.
Answer: Let us derive the simulation. It will be enough to consider one of the two
process $x^{+}_t$ and $x^{-}_t$, that we can denote by $x(t)$. It has the representation $$ x(t) = \sum_{\{k: \, T_k \leq t\}}
e^{- \alpha [t - T_k ]} \eta_k $$ where $T_k$ are the arrivals of the
Poisson Process. For $\Delta > 0$ we have $$ x(t + \Delta) =
\sum_{\{k: \, T_k \leq t\}} e^{- \alpha [t + \Delta - T_k ]} \eta_k
   + \sum_{\{k: \, t < T_k \leq t + \Delta\}} e^{- \alpha
[t + \Delta - T_k ]} \eta_k = e^{-\alpha \Delta} x(t) + \varepsilon . $$
In the second sum denoted as $\varepsilon$, the
arrivals $T_k$ and the jumps $\eta_k$ are independent of the history
$\{x(s);\, s \leq t \}$, and so is $\varepsilon$. So $x(t)$ is
a Markov process. Moreover, conditional on the number of
arrivals on $(t,\,t+ \Delta)$, say $J$, we know
that the r.vs $T_k - t$ have the distribution of the order statistics
of a sample of size $J$ of the uniform distribution on $(0,\,\Delta)$. So
\begin{equation}
   \varepsilon =  \sum_{j=1}^J e^{-\alpha U_j}
\eta_j' \tag{1}
\end{equation}
where the $\eta_j'$ are i.i.d. from the
distribution of $\eta_k$ and the $U_j$ are uniform on $(0,\,\Delta)$. The sum is understood as $0$ when $J =0$.
Therefore the simulation of $x(t + \Delta)$ conditional on $\{x(s);\,
s \leq t \}$ can be as follows

Draw the number of jumps $J$ on $(t,\,t+ \Delta)$ from the Poisson distribution with mean
$\lambda \Delta$.
Draw $J$ i.i.d r.vs $U_j$ from the uniform on $(0,\,\Delta)$ and $J$
i.i.d r.vs $\eta_j'$ from
the jump distribution.
Take $x(t + \Delta) = e^{-\alpha \Delta} x(t) + \varepsilon$ where
$\varepsilon$ is obtained by (1).

If $t_i^\star$ is an increasing sequence of observation times which is
independent of the process $x(t)$, then we can simulate the sequence
$x(t_i^\star)$ by simulating $x(t_i^\star)$ conditional on the history
at $t = t_{i-1}^\star$ for $i > 0$, as in the code of the
question. The sequence $x(t_i^\star)$ is a Markov chain, and it is time-homogeneous if $t_i^\star-t_{i-1}^\star$ is constant.
Interestingly, it can be shown that when the jump distribution is
exponential with scale $1/ \gamma$, the stationary distribution of
$x(t)$ is gamma with shape $\lambda / \alpha$ and scale $1 / \gamma$.
This distribution can be used to draw the initial value
$x(t_0^\star)$, and we have an example of Markov chain with gamma margin
as in this question.
A classical reference for shot noise is section 5.6 in D.R Cox and
V. Isham Point Processes, Chapman & Hall (1980).
Comment: Thanks @Yves. Seems like that's more or less what I did, with the suggestion you made earlier. Much appreciated.
Comment: @ArvindShrivats Yes I just detailed the algorithm that you implemented, up to the autoregression coefficient.


Remove comma from unlimited items body in content type

Question: i have a content type with multiple body fields. It's works but there is a single comma between the body fields on the rendered page. I can't for the life of me figure out how to remove that god forsaken comma. Pls help!!!
Edit, here is an image to hopefully clarify the output issue I am having:

So, the body is an array:

I need to figure where the array is being rendered so that I can remove the comma that is being used to separate the array.
Comment: Are they different body fields, or a single multi-value field? How do you render the fields? In a view?
Comment: It's a single mult value. It's being rendered by the standard node template. No views. Hth
Comment: How does the bootstrap classes get in there, and where does node-page-multi-row class originate from? You are using some support bootstrap module for that, and I bet that module is involved. On a default bootstrap themed node page those classes are not present.
Answer: for anyone else using the progressive theme:
template.php, line 458:
from
<code>  $output .= drupal_render($variables['items'][$i]) . (($i == count($variables['items']) - 1) ? '' : ', ');
</code>
to
<code>  $output .= drupal_render($variables['items'][$i]) . (($i == count($variables['items']) - 1) ? '' : '');
</code>
Comment: +1. You can shorten it even more to this: $output .= drupal_render($variables['items'][$i])
Comment: Eric, where is the difference in your suggestion? I just don't see it... from and to look the same to me...


Finding the lenght of a parameterized curve

Question: I am doing a homework and I am stuck on this problem. I need to find the lenght of a curve defined by
$$
x = \sqrt{2}t,\ y = e^t,\ z = e^{-t}. \quad 0 \leqslant t \leqslant 1
$$
I know that I need to differentiate each component, square them and finally add them up under a square root that I need to integrate. I come to this:
$$
\int_{1}^{0} \sqrt{2+e^{2t}+e^{-2t}} \, \mathrm{d}t,
$$
but then I am stuck… I am pretty sure I did not make a mistake while doing differentiation, but I am not sure how to integrate this… If someone could help, I would be really grateful.
Answer: Hint:
$$\int_0^1\sqrt{2+e^{2t}+e^{-2t}}dt=\int_0^1\sqrt{\left(e^t+e^{-t}\right)^2}dt$$