How to evaluate patient significant improvement?

Question: Imagine the following study where we want to teach a group of $N$ patients to improve their capacity to hold their breath under the water. We have data for the patients before teaching them and after teaching them. Something like
patient_id | duration before (s) | duration after (s)
-----------------------------------------------------
1          | 53                  | 78
2          | 28                  | 65
...        | ...                 | ...
N          | 67                  | 70

What I've seen in the literature is to report the fold increase, ie: the ratio of an increased number to the original number. For the above table it would be
patient_id | duration before (s) | duration after (s) | FI
------------------------------------------------------------
1          | 53                  | 78                 | 1.47
2          | 28                  | 65                 | 2.32
...        | ...                 | ...                | ...
N          | 67                  | 70                 | 1.04

And the reported FI could be $FI = 1.61 \pm 0.53$.
However, I would like to compute the percentage of patients that got a significant improvement in their holding breath duration. For example, I would say that patient N got an insignificant improvement, since moving from 67 to 70 doesn't look like a huge advance, while patient 2 doubled their duration. My objective is to have a result like "X% of the patients showed a significant increase in their holding breath performance".
Which evaluation methodology should I use to that end?
I thought of computing the average FI and its standard deviation and then marking the patients with $FI_i < \langle FI \rangle - k \sigma$ as "not improved". However, I feel that using the same data to calculate the average fold increase and to filter the patients could lead to data leakage and bias the results.
Comment: You use the word "significant" in the sense of "functionally important."  Although statistics can help you quantify what that is, it's up to you to specify how great a relative improvement is "important."  I fear your use of "significant" is likely to mislead people into misinterpreting your question by assuming you mean this word in its conventional sense of a difference that is detectable with your data, which is a different matter altogether.  Please, then, clarify your question and include a characterization of *how much* of an improvement you would deem "significant."
Comment: By some definitions those ratios aren't "fold increases." From [Wikipeida](https://en.wikipedia.org/wiki/Fold_change): 'a change from 30 to 60 is defined as a fold-change of 2. This is also referred to as a "one fold increase".' Maybe call those values the "post/pre ratio" to avoid confusion. Note that averages of ratios don't always work well to describe the distribution of outcomes. Please edit your question to say more about what might be considered an "important" improvement, as @whuber has asked: is a number of seconds improvement, reaching some duration_after, or some post/pre ratio?
Comment: @whuber With significant I meant that you can reject the null hypothesis, which is "the duration of the participant X hasn't changed"
Comment: That contradicts the ways in which employ the word "significant" in your question.
Answer: I think you would need repeated measurements of the patients before and after the treatment to determine if the effect is significant for a single patient. If patient N has values of 67, 66, 67 before and 70, 71, 70 after the treatment, it would probably be still significant. If you have an idea of how large this variance is, you could use the effect size.
Alternatively, you could rewrite your result from:
"X% of the patients showed a significant increase in their holding breath performance".
to
"X% of the patients showed a increase in their holding breath performance of at least Y%".
Answer: The question seems to conflate functional improvement with some type of statistical improvement.

I would like to compute the percentage of patients that got a significant improvement in their holding breath duration... moving from 67 to 70 doesn't look like a huge advance, while patient 2 doubled their duration. My objective is to have a result like "X% of the patients showed a significant increase in their holding breath performance".

That sounds like you want something that would be significant for an individual.
The particular measure that you propose, however:

I thought of computing the average FI* and its standard deviation and then marking the patients with $FI_i < \langle FI \rangle - k \sigma$ as "not improved".

would simply identify individuals whose measure of improvement was in some low quantile among the results of your study sample. If training was very successful, many of those in a low quantile of improvement could nevertheless have a magnitude of improvement that is functionally important for them.
You need to apply your understanding of the subject matter to decide first on a measure of functional importance. If one ever had to hold breath under water to survive, the achieved number of seconds or the increase in the number of seconds of breath-holding following training might be more functionally important than the post/pre ratio of breath-holding times that you propose.
A plot of the empirical distribution function of improvement scores would then describe your results nicely. A reader could decide for herself what a functionally significant improvement score would be and determine what fraction of individuals did better than that.
One caution: as others have noted, anything that you find with this particular design could not demonstrate that your "teaching" had anything to do with improved performance. It might just be that people tend to do better when they try a second time. Without a control group, for example individuals who had a second test without the training, you can't say anything about the effect of the "teaching."

*I'm reluctant to call the post/pre ratio a "fold increase" (FI) due to a different definition for that phrase used in a Wikipedia entry. Furthermore, ratios often have skewed distributions that might not be well represented by the standard deviation in the original scale.
Answer: Agree with most of the points from Elsa12. To be able to test whether two measurements for the same individual are significantly different from one another, you would need repeated measurements absent treatment and absent any other changes. This would tell you how much random measurement error there is associated with the outcome and allow you to estimate whether two values differ by an amount greater than we would expect by chance alone.
As you can't do that, I would recommend the following:
Firstly, make an average conclusion for the sample using whatever test you prefer. i.e., on average, the duration participants can hold their breath significantly increases 1.7 fold.
Second, choose some cutoffs that make sense and say something like: 75% of respondents saw increases in their time by at least 20% (FI of at least 1.2). You could also report how many had FIs below one.
Thirdly, you could create a plot showing the range of FIs across the sample. This is essentially calculating and plotting a treatment effect for every individual and would show the reader how much variability there is in the effects of the program.
Finally, if you wanted to, you could take those individual treatment effects (FIs) and regress them on sample characteristics to see which characteristics predict stronger and weaker effects (treatment effect heterogeneity analysis).
Hope some of that is helpful!
Comment: For reasons set forth in a comment to the question, I believe this answer misinterprets the question.
Answer: I you are trying to determine whether subject X's duration improved significantly relative to their own baseline pre-test duration, the simple answer is that you cannot. While technically the study should use a control group, but I digress. The simple reality is that scenario would be the same as conducting a pre-test post-test prospective analysis on a sample size of 1. As mentioned by others, if you wanted to determine if the treatment effects are significant for a single individual, you would need to do a repeated measures study with an adequate sample of observations before and after. In your scenario, you are determining if the expected values of the distributions are far enough apart to be significant enough.
Now if you are trying to determine what percentage of the post treatment obserivations are significantly different from the pre-treatment observations, that's as simply as counting the observations of $\mu_{x_2}$  that are $\ge \bar{\mu_{x_1}}+1.96\sigma$.
Now, just to be thorough, if you were simply trying to determine if a subject x's improvement was signifcantly greater than the expected value of the cohort, that is as simple as a doing a one sample z-test but using the individual's score instead of the sample mean and use the sample mean in place of teh population mean. I.e., use $\frac{x - \bar{x}}{\frac{S_x}{\sqrt{n}}}$ instead of $\frac{\bar{x} - \mu_x}{\frac{\sigma_x}{\sqrt{n}}}$. In that instance you are using the post test sample as a population. In that instance, you don't need a large sample to generate a second mean because you're not trying to draw any inferences. This is how they determine percentile scores for standardized tests and the IQ test.
But for determining how many people's post treatment performance was significantly better than that same person's pre-treatment performance using post-hoc analysis simply isn't possible. To determine the answer to that question would require basically performing an interrupted time-series analysis or with the requisite number of observations before and after the treatment. Then doing that for the entirety of your study population (I wouldn't suggest a repeated-measures design for that even if there are multiple pre- and post-treatment observations because in time-series the time variable is a consistent interval, which about as close as one could reasonably get to a random sample of an individual's performance before and after treatment).


Please correct my calculation for the differentiation of an extremely basic yet simple single-symbol abstract function

Question: (Foreword: Sorry to bother this platform with this extremely simple confusion, but I've been stuck on this trivial question for a good 3.5 hours, leading to anguish/frustrated to the point that again (and in the past countless times) want to change another self-study textbook, or even give up (although it will not come true because of unwillingness).)

In the book "Introduction to Analysis" (by 高木貞治) → Chapter 3 "Integral Method" → Section 38 "Approximate Calculation of Definite Integrals" → "[Integration of cubic polynomials]":
( Scanned images 1 and 2 of the two pages about this)

$$
\varphi(h)=
  \int_{-h}^h f(x) \,\mathrm{d}x
  - \dfrac{h}{3}(f(h)+f(-h)+4f(0)),
$$
by a simple calculation get
$$
\begin{gather*}
\varphi(0)=\varphi'(0)=\varphi''(0)=0,\\
\varphi'''(h)=-\dfrac{h}{3}(f'''(h)-f'''(-h))
\end{gather*}
$$

I am fairly confident that the following set of expressions should be correct (actually incorrect; eventually ${\color{red}{\texttt{correct}}}{\color{maroon}{\texttt{ed}}}$) by referring to the First Fundamental Theorem of Calculus etc. (and even verified them by giving two small examples):
$$
\newenvironment{rcases}
  {\left.\begin{aligned}}
  {\end{aligned}\right\rbrace}
\begin{rcases}
\left( \int_  0 ^h  f(x)  \,\mathrm{d}x \right)' =  f( h) \\
\left( \int_{-h}^0  f(x)  \,\mathrm{d}x \right)' = -f(-h){\color{red}{(-h)'}}{\color{maroon}{=f(-h)}} \\
(f(-h))' = -f'(-h)
\end{rcases}
$$
Based on them, the following calculations are derived ($\color{orange}{\texttt{Corrected and solved}}$ based on answer):
$$
\require{cancel}
\newcommand{\C}[1]{{\color{orange}{#1}}}
\begin{align*}
  \varphi'  (h) &= f(h) \boldsymbol{\C+} f(-h) - \dfrac{1}{3}(f(h)+f(-h){\color{gray}{+4f(0)}}) - \dfrac{h}{3}(f'(h)-f'(-h)) \\
                &= \dfrac{2}{3}f(h) \C+ \dfrac{\C2}{3}f(-h){\color{gray}{ - \dfrac{4}{3}f(0)}} - \dfrac{h}{3}(f'(h)-f'(-h)), \\[2ex]
  \varphi'' (h) &= \dfrac{2}{3}f'(h) \C- \dfrac{\C2}{3}f'(-h) - \dfrac{1}{3}(f'(h)-f'(-h)) - \dfrac{h}{3}(f''(h)+f''(-h)) \\
                &= \dfrac{1}{3}f'(h) \C- \dfrac{\C1}{3}f'(-h) - \dfrac{h}{3}(f''(h)+f''(-h)), \\[2ex]
  \varphi'''(h) &= \dfrac{1}{3}f''(h) \C+ \dfrac{\C1}{3}f''(-h) - \dfrac{1}{3}(f''(h)+f''(-h)) - \dfrac{h}{3}(f'''(h)-f'''(-h)) \\
                &= \C{\xcancel{\color{red}{-2f''(-h)}}} - \dfrac{h}{3}(f'''(h)-f'''(-h))
\end{align*}
$$
However, compared with the book,

❎ There is an extra item (marked in red) in my calculation result;
❎ (Of course then) I don't get $0$ by substituting $h=0$ into $\varphi^{(1, 2)}$.

So is there some error in my cognition of some points? Thanks for the help!

A lesson I learned
 Originally, from the perspective of antiderivative ($F(u)$), it is very natural and logical to apply the chain rule ($(F(u))'=F'(u)u'$) when differentiation.
 However, when faced with the integral symbol, when take the derivative of the definite integral form ($\int_0^{u(h)}{\color{gray}{f(x) \,\mathrm{d}x}}$) of integral function $F(u)$, I actually regard $u(h)=-h$ as "a symbolic whole" (that is, the illusion "no need to separate out the minus sign"), and thus "drop out" the chain rule – what a serious essential misunderstanding!
 [Conclusion] For the derivative of integrals, the perspective of antiderivatives is more beneficial.
Answer: There's a minus sign error here:
$$\left(\int_{-h}^0 f(x) \; dx\right)^\prime = +f(-h).$$
I'm guessing you forgot the last step of the chain rule which would be to multiply by the derivative of $-h$, which gives you another minus sign.  I think causes your red term to disappear.
EDIT:
Let $f(x) = x$.  Then
$$\left( \int_{-h}^0 x \; dx\right)^\prime = \left(\left.\frac{x^2}{2}\right|_{-h}^0\right)^\prime = \left(\frac{0}{2} - \frac{(-h)^2}{2}\right)^\prime = - \left(\frac{h^2}{2}\right)^\prime = -h. $$
But
$$-f(-h) = -(-h) = h.$$
Not the same.
Comment: It doesn't look correct, because it's not correct.  When you take the derivative you have to use the chain rule and the derivative of $-h$ is $-1$.  You have to multiply the final result by that $-1.$
Comment: The way I am seeing this : the Previous Integral gets f(h) with limits from 0 to h ; If the Previous Integral limits were from 0 to -h , we will get f(-h) ; Now if we reverse the limits from -h to 0 , we will get -f(-h) ; That looks correct !!
Comment: Oh, I think I got where I went wrong (I hope!) : the Previous Integral gets f(h) with limits from 0 to h ; If the Previous Integral limits were from 0 to -h , we will get f(-h)(derivative of -h) = -f(-h) ; Now if we reverse the limits from -h to 0 , we will get -(-f(-h))=f(-h) ; That matches your calculation now !!
Answer: I have not gone over all the calculations, but I have a concern here:

I think here the first term is fine , the middle term is missing $4f(0)$ withing the brackets , while the last term is fine because Derivative of Constant is 0.
Comment: Well, I made a slight correction accordingly, but it did not affect the subsequent steps at all, not to mention the results, alas


Setting up libvirt with remote access on gentoo: not listening tls port

Question: I'm trying to set up libvirtd on my headless gentoo server. I'd like to have remote control via tls. I've followed libvirt remote access guide, but it don't work for me: client virt-manager shows following error:
unable to connect to libvirtd at 'xxx.xxx.info': Connection refused

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/connection.py", line 1055, in _try_open
    None], flags)
  File "/usr/lib/python2.6/site-packages/libvirt.py", line 107, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirtError: unable to connect to libvirtd at 'xxx.xxx.info': Connection refused

Here is my current server libvirtd config (comments stripped):
listen_tls = 1
tls_port = "16514"
auth_tls = "none"
tls_no_verify_address = 1

And it does not listens to port 16514: lsof -i tcp:16514 prints nothing.
I also have following in my log just after libvirtd start:
Mar 30 14:46:02 fs dnsmasq[27523]: started, version 2.52 cachesize 150
Mar 30 14:46:02 fs dnsmasq[27523]: compile time options: no-IPv6 GNU-getopt DBus I18N DHCP no-TFTP
Mar 30 14:46:02 fs dnsmasq-dhcp[27523]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Mar 30 14:46:02 fs dnsmasq[27523]: reading /etc/resolv.conf
Mar 30 14:46:02 fs dnsmasq[27523]: using nameserver 192.168.1.3#53
Mar 30 14:46:02 fs dnsmasq[27523]: bad address at /etc/hosts line 2
Mar 30 14:46:02 fs dnsmasq[27523]: read /etc/hosts - 15 addresses
Mar 30 14:46:10 fs kernel: IN=eth1 OUT= MAC=01:00:5e:00:00:01:00:18:82:ed:d1:df:08:00 SRC=172.28.36.1 DST=126.96.36.199 LEN=32 TOS=0x00 PREC=0xC0 TTL=1 ID=62606 PROTO=2
Mar 30 14:46:33 fs kernel: IN=virbr0 OUT= MAC=01:00:5e:00:00:01:26:a8:55:dc:5f:05:08:00 SRC=0.0.0.0 DST=126.96.36.199 LEN=32 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2

"kernel: IN=virbr0" are dropped packets logged by -j LOG rule in my iptables config. However, I've tried with all iptables rules allowed - nothing changed.
Certs seems to be fine as well (maybe not?):
# ls -R /etc/pki/
/etc/pki/:
CA  libvirt

/etc/pki/CA:
ca.info  cacert.pem  cakey.pem

/etc/pki/libvirt:
private  servercert.pem

/etc/pki/libvirt/private:
server.info  servercert.pem  serverkey.pem

I'm on gentoo running
[ebuild   R   ] app-emulation/libvirt-0.8.8-r1  USE="json libvirtd lvm lxc network nls parted python qemu udev virt-network -avahi -caps -debug -iscsi -macvtap -nfs -numa -openvz -pcap -phyp -policykit -sasl (-selinux) -uml -virtualbox -xen" 0 kB

libvirtd is running:
# ps ax | grep [v]irt
30484 ?        Sl     0:00 /usr/sbin/libvirtd -d
30735 ?        S      0:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file=  --except-interface lo --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-lease-max=253 --dhcp-no-override

Anything else I should check. How do I debug this issue?
SOLVED:
I'd run libvirtd with --lsiten flag. It's possible to specify it in /etc/conf.d/libvirtd file.
Also, I've found another error in my configuration: when adding connection in virt-manager, you must specify an hypervisor, so url would look like quemu+tsl://user@host/system, not just quemu+tsl://user@host/
Answer: The starting point of your investigation should be here :

And it does not listens to port 16514: lsof -i tcp:16514 prints nothing

This is quite a low level issue : the daemon does not listen on the tcp port.
You should check for any log messages that could indicate an error, or launch the daemon manually in a terminal to see what it says...
Comment: The problem that it does not prints any messages to logs or stdout even when launching from console with -v flag.
Comment: Oh... There is --listen switch, that should be specified for libvirt to listen network. You've not answered my question, but forced me to run libvirtd with --help. Thanks!
Comment: well, actually, sort of. Would you have run the daemon as executable if I didn't suggested it? ;)


What makes Dm/E sound Phrygian?

Question: I was told by a jazz pianist that you can play Dm/E if you play in E Phrygian.
What makes Dm/E sound Phrygian? To me Am or Am7 would be a better chord for Phrygian but then I probably think of Phrygian in terms of church music theory and not jazz theory.
Comment: Note that you can also play Am7 in E Aeolian. Listeners will default to Aeolian over Phrygian if not exposed to that distinctive flattened supertonic.
Answer: The characteristic tones of phrygian are flat 2 and perfect 5. They distinguish phrygian from aeolian and locrian.
Dm/E is E - D - F - A. Note E in bass suggests E is root. Other notes come from phrygian scale, in particular F is flat 2.
Note B or perfect 5 is missing, so in principal Dm/E doesn't distinguish phrygian from locrian (which would have Bb, flat 5), but as for most ears locrian sounds exotic, phrygian would be the first interpretation.
Am7 is certainly a valid chord for E-phrygian (or A-phrygian, I'm not sure which one you meant), but it doesn't contain its characteristic tones so it doesn't emphasize the mode.
Comment: Yes, I agree you can, it "fits". What I said that Am by itself won't suggest E phrygian strongly.
Comment: You can play Am and have E as Mi (which is the finalis). This works in chant accompaniment. If it works in jazz I don't know.
Answer:
Dm/E contains the pitches DEFA.
One can "fill in the blanks" with passing tones: say, GB and C.
The resulting scale is DEFGABC.
"Permuting" the scale gives EFGABCD, which is E Phrygian (in the modern sense, rather than the Gregorian sense).

The pitches of the Dm chord are "embedded" within the E Phrygian scale, just as they are with C Major or D Dorian. But the bass E suggests the possibility of E as the tonic.
Depending on context, your could "fill" Dm/E with other passing tones. For example, EFGABbCD, which would be E Locrian, origination from F major. Alternatively, EFG#ABCD, which would come from the A harmonic minor scale.
Comment: No.3 is a dead ringer for D Dorian!

What makes Dm/E sound Phrygian?


Find the matrix of the total derivative of $f$ at $(0,0)$.

Question: Let $f: \Bbb R^2 \to \Bbb R$ be defined by
$$ f(x,y) = (x^2+y^2)\sin(\frac{1}{\sqrt{x^2+y^2}}) $$
for $(x,y) \not= (0,0)$ and $f(0,0) = 0$. I'm trying to find the matrix of the total derivative of $f$ at $(0,0)$. I think it's just $\begin{pmatrix} 0&0 \end{pmatrix}$ but I'm not sure. Here is my reasoning:
For all $\hat{h} \in \Bbb R^2$ we have
$\begin{align*}
f( (0,0) + \hat{h} ) - f( (0,0) ) &= f( \hat{h} ) \\
&= ||\hat{h}||^2 \sin{1/||\hat{h}||}\\
&\leq ||\hat{h}||^2
\end{align*}$
Since $||\hat{h}||^2 = o(||\hat{h}||)$, we have
$$ \lim_{\hat{h} \to (0,0)} \frac{f(\hat{h})}{||h||} = 0 $$
Thus the total derivative is the zero function $\hat{0} : \Bbb R^2 \to \Bbb R$. So its matrix is $\begin{pmatrix} 0&0 \end{pmatrix}$.
Is this correct? If so, how exactly should I interpret the matrix $\begin{pmatrix} 0&0 \end{pmatrix}$? Does it have to do with the fact that the tangent plane to the graph of $f$ at $(0,0)$ is $z=0$ (assuming this is correct)?
Answer: Yes this is fine. Consider the graph map given by $F(x,y) = (x,y,f(x,y)$. We know that the differential $DF(p): T_pU \to T_{F(p)}\Gamma(f|_U)$ where $\Gamma(f|_U)$ is the graph of $f$ but restricted to $U$ i.e the tangent plane at $F(0,0) = \textbf{0}$ is the image of $DF(0,0)$.
$$\textbf{im}(DF(0,0)) = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ f_x(0,0) = 0 & f_y(0,0) = 0 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} x & y & 0 \end{pmatrix}$$

and this corresponds to the plane $z = 0$. Without the "matrix" approach, you can use the fact that the linearization $L(x,y)$ of $z = f(x,y)$ is the best linear approximation to the graph of $f$ i.e;
\begin{align*} L(x,y) &= f_x(0,0) x + f_y(0,0) y + f(0,0) \\& = f(0,0) \\ & = 0 \end{align*}
Hence, $f(x,y) - L(x,y) \approx \epsilon$ if $(x,y)$ is sufficiently close to the origin, and so $z = f(x,y) = 0$ is the tangent plane.
Comment: I think you were confusing that step with problems in which you try to show that a function is not continuous at the origin.
Comment: Thanks, I accepted your answer. One small question--I thought in this case the partial derivatives $f_x(0,0)$ and $f_y(0,0$ don't exist. This is because at $y=0$, $f_x = 2x\sin(1/x) - \cos(1/x)$ which has no limit as $x$ approaches $0$. Or do $f_x$ and $f_y$ represent something different?
Comment: Ah, now I see where I went wrong. At $(0,0)$, the partial derivatives exist, but are not continuous.
Comment: Yes. This is the example in which you see partials don't have to be continuous for a function to be differentiable.


Using session attributes in spring MVC

Question: I am developing a web application using spring MVC. I just want a simple example of how to do session management in this. I have seen lot of forums but I am not able to get a clear picture of this
My requirement is
I have an object, which I would like to be accessible in all controllers and JSP's I
would like to set that in the controller and get that in JSP
I am looking for something like
    Session.setAtribute();

Could you please let me know a very simple instance . Thank you
Answer: There are different ways of accessing servlet session in Spring MVC. But I think this one is the one that best suits your problem. You can create a session scoped bean, which holds your desired info:
@Component("myObjectHolder")
@Scope(WebApplicationContext.SCOPE_SESSION)
public class MyObjectHolderImpl implements MyObjectHolder {

    private long userId;
    private String username;
    private Theme theme;

    // Getters & Setter
}

Then, you can access to it from other beans:
@Controller
public class MyController {

    @Autowired private MyObjectHolder myObjectHolder;

    @RequestMapping
    public ModelAndView switchTheme(String themeId) {
        ...
        Theme newTheme = themeService.get(themeId);
        myObjectHolder.setTheme(newTheme);
        ...
    }

}

You can access directly from your view too, but you must configure it:
<bean class="org.springframework.web.servlet.view.InternalResourceViewResolver">
    ...
    <property name="exposedContextBeanNames" value="myObjectHolder" />
</bean>

And in your JSP:
Hi ${myObjectHolder.username}, you switched
application theme to ${myObjectHolder.theme.name}

Comment: Thank you for your reply.So MyObjectHolderImpl is where I set my seesion object and I can get it from anywhere rite ,.. COz I dun have to set the session object multiple times, I just need to set it once in one of the controlldersin the My
Comment: @user1562262: No, you probably don't really need to access to session object. You can save desired info (such as user id, name, theme, preferences...) in MyObjectHolder. I'll update my answer.
Answer: The simplest approach is to access HttpSession directly by injecting it into your handler method:
@RequestMapping("/page")
public ModelAndView page(HttpSession session) {
    session.getAttribute("foo");
}

Comment: Thank s for your reply  .. I just want to know like, do I have to inject HttpSession session in the controller that sets the session or both only in the controllers that get the session
Comment: @user1562262: inject `HttpSession` whenever you need it :-). Also read a little bit more, there are much more convenient ways to access HTTP session attributes in Spring MVC, but requiring a little bit of setup upfront.

Using session attributes in spring MVC


SocketException in Mongo logs

Question: We are running mongo for 4 months so far , But lately I am seeing a lot of
SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:58996]
How can I know the cause of this error ? Is it related to code issue or administrator configuration?
We have Fedora 16 server , mongo version 2.0.7
Answer: First off, take a look and follow the steps outlined here:
http://www.mongodb.org/display/DOCS/Troubleshooting#Troubleshooting-Socketerrorsinshardedclustersandreplicasets
Next, look for any ulimit issues on the target host (a new file descriptor is required for a new socket and can cause the error):
http://www.mongodb.org/display/DOCS/Too+Many+Open+Files
Finally, there are a couple of issues related to idle connections being used when they should not be, and that can also contribute to this type of issue:
https://jira.mongodb.org/browse/SERVER-5793
Until SERVER-5632 is complete, the only remedy here is to flush the connections by restarting the mongod/mongos processes.
Answer: I got this to stop happening by adding a batchSize option arg to my cursor query
e.g.
before:
var cursor = db.collection('images').find( {}, {"_id": 1} ).sort({"_id": 1});

after:
var cursor = db.collection('images').find( {}, {"_id": 1} ).sort({"_id": 1}).batchSize(1000);

This made it run smoother (not as many start/stops) and with fewer nasty messages. None in fact.

SocketException in Mongo logs


Prove that $(\log_2n)^{\log_2n} \in o( 2^{(\log_2n)^2})$

Question: Let $f(n) = (\log_2n)^{\log_2n}$ and $g(n) = 2^{(\log_2n)^2} $
I want to show that $f(n) \in o(g(n))$.
$f(n) = O(g(n))$ if there exist positive constants $c$ and $n_0$ such that
$0 \leq f(n) \leq cg(n)$ for all $n ≥ n_0$
I guess it makes intuitive sense if we perform a a substitution $z = \log_2n$ we get $\lim_{z \to \infty} \frac{z^z}{2^{z^2}}$ which I remember as converging to $0$. But I'm not sure exactly why. How do you prove this rigorously?
I guess taking derivatives comes to mind which gets me $\frac{d}{dz}z^z = z^z(\log z+1)$ and $\frac{d}{dz}2^{z^2} = 2^{z^2}2z\log(2) $ which isn't that helpful for me.
Comment: Hint: $$a^b=2^{b\log_2a}$$
Answer: Check the log of the quotient tends to $-\infty$:
$$\log\frac{f(n)}{g(n)}=\log_2n\log_2(\log_2n)-(\log_2n)^2\log2=-(\log_2n)^2\biggl(\log2-\frac{\log_2(\log_2n)}{\log_2n}\biggr).$$
The second factor tends to $\log2\;$ since $\;\log_2u=o(u)$.


Show that the series $\sum\limits_{n=1}^\infty \frac{\left(1-\frac{e}{\sqrt{n}}\right)^ne^{t\sqrt{n}}}{n^{3/2}}$ is convergent

Question: Show that the following series
$$\sum\limits_{n=1}^\infty \frac{\left(1-\frac{e}{\sqrt{n}}\right)^ne^{t\sqrt{n}}}{n^{3/2}}$$
is convergent, where t is a real parameter.
Comment: Shouldn't $t$ be $<e$?
Comment: It should be...
Answer: We have for $0<u<1$,
$$\ln(1-u) = -u + \frac{u^2}2 - \frac{u^3}3 + \ldots \le -u + \frac{u^2}2$$
Then:
$$\left(1-\frac{e}{\sqrt n}\right)^n = \exp\left(n \ln\left(1 - \frac{e}{\sqrt n}\right) \right) \le \exp\left(n\left(-\frac{e}{\sqrt n} + \frac{n}{2e^2}\right)\right) = \underbrace{e^{e^2/2}}_{:=c} e^{-e \sqrt n}$$
Then the whole term in the sum is bounded above by:
$$\frac{ce^{(t-e)\sqrt n}}{n\sqrt n} \le \frac{c}{n\sqrt n}$$
By comparison test, the series converges.
Comment: When it is divergent?
Comment: When we have t > e.
Answer: Hint. One may use a Taylor series expansion giving, as $n \to \infty$,
$$
\begin{align}
\frac1{n^{3/2}}\left(1-\frac{e}{\sqrt{n}}\right)^ne^{t\sqrt{n}}&=\frac1{n^{3/2}}e^{\large n\ln\left(1-\frac{e}{\sqrt{n}}\right)}e^{t\sqrt{n}}
\\\\&=\frac1{n^{3/2}}e^{\large n\left(-\frac{e}{\sqrt{n}}+O\left(\frac1{n}\right)\right)}e^{t\sqrt{n}}
\\\\&=\frac1{n^{3/2}}e^{\large (t-e)\sqrt{n}+O\left(1\right)}
\end{align}
$$ then one sees that the given series is convergent iff $t-e<0$.
Comment: When it is divergent?
Comment: @meteomath When $t-e\ge0$.
Comment: And, what happens when t-e>0 ?
Comment: @meteomath We have $t-e>0 \implies t-e\ge0$, isn't it?
Comment: When t-e=0, t=e. Then we have 1/n^3/2. It is a convergent series because this series is Dirichlet's series, 1/n^s, and it  converges , when s>1.
Comment: I think you make a confusion, when $t=e$ we **don't** have $\frac1{n^{3/2}}\left(1-\frac{e}{\sqrt{n}}\right)^ne^{e\sqrt{n}}=\frac1{n^{3/2}}$, the precedent equality is *not* true.
Comment: @meteomath You are welcome.


Enviar input para salvar no arquivo JSON

Question: Tenho uma tela onde existe um auto-complete para itens de venda, o usuário pesquisa o item por nome ou código, até aqui tudo funciona.
Preciso captar o item escolhido e enviar para o backend (Django) para poder salvar em um arquivo JSON. A ideia é ter um JSON para cada conta.
Mas não sei como enviar os inputs para o backend de modo que possa gravar isso em um arquivo, já que vou carregar esse mesmo arquivo para preencher uma table dos itens registrados\escolhidos.
Esse é meu código de autocomplete.
$(function() {

    $("#itemsearch").autocomplete({
      source: "/menus/autocomplete_search/",

      select: function (event, ui)
       {
       AutoCompleteSelectHandler(event, ui)
       {
            ui.item.value = "";
        }

        },
        minLength: 1,

    });
});

var result = 0
var price = 0

  function AutoCompleteSelectHandler(event, ui)
  {
    var selectedObj = ui.item;
    var name = selectedObj.value
    var codigo = selectedObj.item_num
    price = selectedObj.item_price
    var table = document.getElementById('tableItemVenda');
    var row = table.insertRow(1);
    row.innerHTML = ""+codigo+"	 "+name+"	 R$ "+price+"	";
    inserirTotal(price);

}

Comment: O que significa escolhido? Select na página? Submit de um form? Isso vai determinar qual evento você vai chamar e a hora que vc chama o backend pra gravar a seleção
Comment: @LeonardoPessoa o item selecionado é o 'ui.item' do autocomplete.  Funciona assim, ao digitar o nome ou o codigo, o(s) itens aparecem na lista suspensa do autocomplete, para escolher o item, o usuario navega com as setas e aperta enter no item que desejar.
Answer: Consegui captar as escolhas do usuario chamando a nova funcao 'SaveInvoice' assim que um novo item é selecionado. Sempre que um item é selecionado, é salvo na variável selectedObj.

$(function() {

    $("#itemsearch").autocomplete({
      source: "/menus/autocomplete_search/",

      select: function (event, ui)

       {
       AutoCompleteSelectHandler(event, ui)
       {
            ui.item.value = "";
            SaveInvoice(); //chama funcao para salvar item para o json
        }

        },
        minLength: 1,

    });
  });

function AutoCompleteSelectHandler(event, ui)

  {
    selectedObj = ui.item;
}

function SaveInvoice(){
        $.ajax({
            url: "/menus/gerar_conta/",
            type: "POST",
            dataType: 'json',
            data: {selectedObj, csrfmiddlewaretoken: '{{ csrf_token }}'},
        });
    }


Proving triangular inequality in general

Question: I want some advice on approaching proofs of the triangular inequality for various metrics. I have a suspicion the answer is simply 'improve mathematical maturity' but hopefully there is a general procedure that this comes down to.
Examples: Proving $d(z_1,z_2)=|z_1-z_2|$ on $\Bbb C$, or $d(z_1,z_2)=\min\{|z_1|+|z_2|,|z_1-1|+|z_2-1|\}$ if $z_1\ne z_2$ and $d(z,z)=0$ on $\Bbb C$ etc. I just find triangular inequality tricky. Thanks
Answer: Metric spaces are so common place and so diverse that you can't really expect some general trick that will save you simply checking the property for any given particular definition of metric. There are some general tricks, like transforming one metric by a certain concave function always yields a new metric, but really, most often, you simply check the definition directly. Quite often it's not very hard, but sometimes, as in normed spaces where the triangle inequality is the non-trivial Minkowski inequality, you have to work harder.


Get a difference between 2 dates in js

Question: Currently I'm working with an API which returns me a date(created_at) in this format:
Y-m-dTH-m-sZ.

Actually I can't find out how can I calculate a difference between this format of a date and a current date that I can receive from new Date().
And the difference should be in the format like this for ex.:
2 hours ago, 10 days ago.

Is it a built in function in JS? Or how can I make this calculation and get an answer in the appropriate format?
Please help.
Comment: I always recommend `moment.js` for these sorts of things in JS. Doing it natively is quite cumbersome.
Comment: The questions you're asking are duplicates. On parsing dates: [*Why does Date.parse give incorrect results?*](https://stackoverflow.com/questions/2587345/why-does-date-parse-give-incorrect-results) On getting date difference in "friendly" format: [*Javascript fuzzy time (e.g '10 minutes ago') that's in exact seconds*](https://stackoverflow.com/questions/11479170/javascript-fuzzy-time-e-g-10-minutes-ago-thats-in-exact-seconds).
Comment: Yes actually the same RobG.
Please read my question again. I was talking about difference in datetime formats...
Answer: The code below will work assuming dateB is after dateA, otherwise the difference will be negative and you would have to handle that.
This may be a good headstart for you, sadly there's nothing inbuilt for that, as you are trying to get a difference (which comes in miliseconds) and convert that to a unity that will vary.
Good luck, hope it's helpful.

var dateA = new Date("07/06/2017");
var dateB = new Date("10/06/2017");

var difference = dateB - dateA;

var differenceUnit = "milisecond";

if(difference >= 1000) {
 difference = difference/1000;
 differenceUnit = "second";

 if(difference > 60) {
  difference = difference/60;
  differenceUnit = "minute";

  if(difference > 60) {
   difference = difference/60;
   differenceUnit = "hour";

   if(difference > 24) {
    difference = difference/24;
    differenceUnit = "day";
   }
  }
 }
}

difference = difference.toFixed(0);

console.log("difference is: ", difference, (difference > 1 ? differenceUnit + "s" : differenceUnit))

Get a difference between 2 dates in js


For $0 \le x \le \frac{π}{4}$,let $f(x) = \sin(x) + \cos(x)$. Find the infimum of $f(x)$ over the interval $x \in [0,\frac{π}{4}]$

Question: For $0 \le x \le \frac{π}{4}$,let $f(x) = \sin(x) + \cos(x)$. Find the infimum of $f(x)$ over the
interval $x \in [0,\frac{π}{4}]$.i found the critical points and the second derivative but it seems there is no infimum for this function in the defined interval.Am i right ? or should i have to use an other method?
Comment: Thers *must* be a minimum and a maximum, because $f$ is *continuous* on a *compact* interval, so by Weierstrass's theorem it must reach its extremes.
Comment: differentiated found the critical point applied it in the second differential
Comment: No infimum for a bounded continuous function on a compact interval? Really? Just an hint: an infimum is not necessarily a critical point, even if the function is differentiable everywhere, as here. There is another possibility that should be cristal clear if you draw the curve.
Comment: Here "compact interval" just means bounded interval in the real numbers.  The second derivative is not terribly important.  Compare the values at endpoints and at critical points, and choose the smallest of these.
Answer: HINT:
$$\sin x+\cos x=\sqrt2\sin\left(\dfrac\pi4+x\right)$$
$$0\le x\le\dfrac\pi4\iff\dfrac\pi4\le x+\dfrac\pi4\le\dfrac\pi2$$
$$\implies\sin\dfrac\pi4\le\sin\left(\dfrac\pi4+x\right)\le\sin\dfrac\pi2=1$$
Answer: Note $f'(x) = \cos x - \sin x$. Since $\cos x > \sin x$ in $[0, \frac {\pi}{4})$ $f$ is monotone increasing in $[0, \frac{\pi}{4})$, which implies we have a minimum of $1$ at $x=0$.
Answer: Using trigonometric identities:
Recall that
$$a \sin x + b \cos x = \sqrt{a^2 + b^2} \sin (x + \varphi)$$
where $\tan \varphi = \frac ba$. Thus,
$$f(x) = \sin x + \cos x = \sqrt{2} \sin (x + \frac {\pi}4).$$
$$x \in [0, \frac {\pi}4] \Rightarrow (x + \frac {\pi}4) \in [\frac {\pi}4, \frac {\pi}2].$$
Observe that $f(x) = \sin x$ is monotonically increasing on $[\frac {\pi}4, \frac {\pi}2]$, thus
\begin{align}
f(x)_{min} & = f(0) \\
 & = \sqrt 2 \sin \frac {\pi}4 \\
 & = 1. \\
\end{align}
Using calculus:
$$f(x) = \sin x + \cos x.$$
On the interval $[0, \frac {\pi}4)$, $f'(x) = \cos x - \sin x \gt 0$, and $f'(\frac {\pi}4) = 0$, thus,
\begin{align}
f(x)_{min} & = f(0) \\
 & = \sqrt 2 \sin \frac {\pi}4 \\
 & = 1. \\
\end{align}


Posterior distributions and loss functions.

Question: I am unsure why I got this question wrong:

So if my posterior probability of a head-based coin is currently 60%, then that means I'll choose heads and so my EV of a loss is 4 (40% * 10). However, if I wait and flip another coin, I immediately incur a loss of 1 + whatever my new EV is. What's my new EV seems to be the heart of the question.
How do I know what my new posterior is after I flip another coin?
Comment: Your question is answered in the pop-up commentary: "Using Bayes rule."  Each possible outcome can lead to a different "new posterior," of course--but you only need to consider two cases.
Comment: Sure: the coin can land heads.  Compute the posterior and make a decision accordingly.  It can land tails.  Compute the posterior and make a decision accordingly.  Assess your prospects in light of these possible outcomes.
Comment: can you flesh it out for me?
Comment: With numbers? If you write the answer and why I can credit you?
Comment: Does the question provide me with enough information?
Answer: As indicated in the hint, you can find the posterior using Bayes rule. For example, if $P\left( B_H \right)=0.6$ is your current posterior probability that the coin is head-biased, and $P\left( {H|B_H} \right)=0.75$ the likelihood of obtaining head in the next flip (under the hypothesis of a head-biased coin), then the posterior probability of a head-biased coin after having observed head in the next flip is
$$
P\left( {{B_H}|H} \right) = \frac{{P\left( {H|{B_H}} \right)P\left( {{B_H}} \right)}}{{P\left( {H|{B_H}} \right)P\left( {{B_H}} \right) + P\left( {H|{B_T}} \right)P\left( {{B_T}} \right)}} \approx0.82
$$
Hence the expected loss (after having observed head and decided that the coin was biased toward heads) would be
$$L(B_H,H)=1 + \left[1-P\left( {{B_H}|H} \right) \right] \cdot 10 \approx 2.8$$
You can do the same calculation and compute the expected loss after having observed $T$ (tail). I find that $P\left( {{B_T}|T} \right) \approx 0.67$, and that the expected loss (after having observed tail and decided that the coin was biased toward tails) is $L(B_T,T)\approx 4.3$.
However I am not entirely sure of what they mean with minimum posterior expected loss in the question, in particular when they write that the "minimum expected loss of making the decision now is 4": I think this is confusing, and should be just the "expected loss of making the decision now". The minimum expected loss is instead associated with the other possible course of action, that is making the decision after another flip. You can compute the expected loss of flipping another time the coin by taking the average of the expected loss of each of the outcomes of the flip (which we computed above), weighted by their probability. The probability $p(H)$, for example, is given by the denominator of the equation above
$$ p(H) = {P\left( {H|{B_H}} \right)P\left( {{B_H}} \right) + P\left( {H|{B_T}} \right)P\left( {{B_T}} \right)} = 0.55$$
The expected loss of flipping the coin another time is then
$$
p(H)\cdot L(B_H,H) + p(T)\cdot L(B_T,T) = 3.5
$$
So the correct answer should be the third one.
Comment: The outcome of the coin flip is a Bernoulli random variable, so the $p$ is the parameter that indicates the expected proportion of heads over a large number of coin flips. The question asks you to decide between two options: either the coin is head-biased ($p=0.75$, hence heads are obtained on 75% of the times on average) or it is tail-biased ($p=0.25$, meaning that tails are obtained 75% of the times).
Comment: Is this the right way to interpret the question? What does the question mean when it says the p of the coin being tail biased is p(0.25)
Comment: I don't see another way to interpret the question. However I am not entirely sure about what it means with "_minimum_ expected posterior loss of making the decision now is 4". I think this should be called just the "expected posterior loss" of the first possible course of action (that is: deciding now). Bayes theorem allows then to compute the expected loss of the other possible course of action (flipping the coin and deciding after). Only then one could compare the two expected loss and know which one has the _minimum expected loss_
Comment: I have edited the answer. let me know if the third is the correct one :-) as I explain in the edited answer I find a bit odd the way how the word _minimum_ is used in the text of the question.


How to test two lines from the same model are parallel

Question: If I have two sets of data each with 5 entries for X&Y, I built a model to incorporate this using a dummy/indicator variable for when X is in Set B, so obviously I get two parallel lines;
$$y=\alpha + \beta_{1}x +\beta_{2}\mathbb{1}_{x \in B}+\epsilon$$
 where epsilon is N(0,1).
How I do formally test the hypothesis that the two lines are parallel??
Comment: .........ST221?......................................
Comment: Why is $\epsilon$ *standard normal*? It would be unusual to now the variance, let alone for it to be 1.
Answer: A common way to test for parallelism of the two lines is to add an interaction term:
$$y=\alpha + \beta_{1}x +\beta_{2}\mathbb{1}_{x \in B}+\beta_{3}\,x \cdot\mathbb{1}_{x \in B}+\varepsilon$$
and then test whether $\beta_3$ differs from zero. If you reject the null, the conclusion is that the lines are not parallel. (You can't actually prove they are parallel.)
This can be done either as a t-test in the usual regression output or as an F test.
Comment: But I am asked do this as a hypothesis in R, I was thinking F test or T test as the smaller model is nested in the larger?
Comment: See my edit. Is this for some class? If so, you should really have mentioned that.
Comment: OK thank you! Ive done the T test, but how would you go about doing the F test...apologies I am a beginner at R :)
Comment: One way would be to fit one model via `lm`, then the other\* and supplying both `lm` objects to `anova` (smaller model frst). $\:$ \* `update` could be used when fitting the second model but isn't necessary. $\qquad$ See `?lm`, `?anova` (and `?update`, if you use that)


A clarification about BV functions.

Question: From definition, a locally integrable $ u \in BV(\Omega) $ if its distribution derivative is given by a signed Radon measure. That is there exists $ \mu $ such that for any $ \phi \in C^\infty_c(\Omega) $ we have
$$ -\int_\Omega u\phi' dx = \int_\Omega \phi d\mu $$ Now for any $ u \in L^1_{loc}(\Omega) $ it is clear that $ \Lambda(\phi) = \int_\Omega u\phi' dx $ is in dual of $ C^\infty_c(\Omega) $ hence from Riesz Representation Theorem such a signed measure $\mu$ should always exist, this implies $ L^1_{loc}(\Omega ) = BV(\Omega) $. Where am I mistaken ?
Comment: The Riesz Representation theorem is only about the dual of $C_c(\Omega)$.
Comment: Oh yes...Thank you
Answer: It is true that for any $u\in L^1_{\rm loc}$ the formula $\Lambda(\phi)=\int_{\Omega} u\phi'\,dx$ defines a linear functional on the space of test functions. The functional is controlled by the norm of $\phi'$ and therefore is a distribution of order 1. However, this makes it more singular (in general) than measures, which are distributions of order $0$.
(CW answer to fill this box).

A clarification about BV functions.


How can I hide tkinter window when I opened tkinter save file dialog?

Question: My program uses tkinter.asksaveasfilename function to save a text into another location but when user opens save as dialog, tkinter's empty square window also pops-up. My program works on console so I don't need the tkinter window.
I searched the web and stackoverflow but only found root.withdraw funtion to hide tkinter window but when I use it before asksaveasfilename function, save as dialog never pops up. If I use root.withdraw after asksaveasfilename function, square empty tkinter window closes after user closes the save as dialog.
Is there a way to hide tkinter window when asksaveasfilename function is active?
Comment: Can you provide the code that uses `root.withdraw()` before the `asksaveasfilename`? I think something is wrong with your code.
Comment: @TheLizzard I tried copying the save as part from my code into a fresh .py file and runned it, interestingly it worked. But with same part of code, it's not working with the whole code. If I give link to my whole code, can you examine it? https://github.com/Yilmaz4/Encrypt-n-Decrypt/blob/python/Encrypt.py

The part is starting from line 116 and finishing in 133.
Comment: Just after line `117` add `root.withdraw()`. I can't find a reason for why it wouldn't work
Comment: @TheLizzard I moved save as lines between 116 and 133 to the beginning of the code and it worked, I moved them inside the first while loop, it worked again, it continued working till I put these lines into the if statement in line 27 and save as dialog doesn't popped up. After this line, save as dialog never popped up in below lines. Very weird. Do you think it's a Python bug or what?
Comment: Right now there are too many variables to keep track of. Please look at how to define and use python functions. Also your code can be simplified greatly eg: `if exit == True: exit = True; break` can be simplified into: `if exit: break`. I don't get what most of the code is doing.
Comment: @TheLizzard I will try to remove most of variables and test. Also my code is a encryption program. You can learn more from my github page. Also yes you're right. I can simplyfie my code like this. Have a nice day.
Answer: Does this work for you?:
from tkinter.filedialog import asksaveasfilename
import tkinter as tk

# Create the window
root = tk.Tk()
# Hide the window
root.withdraw()

# Now you are free to popup any dialog that you need
print(asksaveasfilename())

# Destroy the window
root.destroy()

It works for me (Windows 10). I think that it should work for Linux and MacOS too.
Comment: Interestingly, this works for me. I tried the same thing in my code but it didn't worked. I mean I am simply creating the root, withdrawing it, then opening save as dialog after that destroying root. But save as dialog never pops up. I'll write save as dialog part again. If it works, then problem is with my code.
Comment: The problem with this solution is that it hides the file dialog from the task bar. If other windows are in front of the file dialog, you have to minimize each of those windows one by one, or look for the file dialog in the Task View rather than task bar.
Comment: @root I didn't check for that when I was writing this answer. Just now I tested it on Ubuntu and the file dialog is in the task bar. Did you test it on Windows?
Comment: @TheLizzard Yes, on Windows it's not in the taskbar. Here's a solution without this problem: https://stackoverflow.com/a/68424608/5231110


Quantize a continuous random variable

Question: Suppose we have a continuous random variable $X$. We do not know its distribution function, but have $n$ i.i.d. samples.
I am looking for methods that quantize (discretize) $X$ into a categorical variable $\tilde{X}$ with $k$ level.
The most naive method is the uniform quantization, where each quantization bin has the same length. However, it seems more reasonable to set the length of bins according to empirical density of $X$, as shown in the below figure.

I think there must be some standard text that deals with my problem, but have not found one yet. I would be really grateful if you could point to an authoritative reference that covers this topic.
Comment: [Why do you want to discretize your data?](https://stats.stackexchange.com/q/68834/1352)
Comment: You can bin them by quantiles, then each bin will have equal density.
Comment: Every textbook on cartography I have seen (and I have a bunch) discusses this.  Standard methods are equal intervals, equal areas (not relevant here, but hinting at the possibility of exploiting additional variables for the binning), quantiles, and "Jenks' method," which is just k-means clustering in 1D.  Of course there are many other methods -- infinitely many.  Which of these might be legitimate depends (very much) on the circumstances and your objectives.
Comment: If you want to approximate the density, bins of equal length seem the most reasonable choice (check `?hist` in R)
Comment: Thanks, binning by quantiles work! One may use ```pandas.qcut``` for implementation.
Answer: Binning a variable by percentiles with equal spacing will ensure intervals with the same density.
Answer: It appears that you are looking for change points.  Each change point becomes a bin boundary.  The density is the number of points between change points, divided by the length of the bin.
Alternatively, you can do nearest neighbor smoothing, or lowess smoothing.
Comment: The usual meaning of "change point" requires *paired* data.  It's unclear how smoothing would address the question.
Comment: No, you can have change points in a univariate list (like a list of photon arrival times).  See https://docs.astropy.org/en/stable/api/astropy.stats.bayesian_blocks.html#astropy.stats.bayesian_blocks for example.
Comment: That requires the list to be *ordered* and for you to create a variable corresponding to the index of the ordering.  Although you could order a sample from a random variable, almost any changepoint technique would be inapplicable due to the correlations induced by that ordering.
Comment: Sure, but nothing in the original question says such an ordering is not possible.  In fact, the presence of "value->" along the x-axis says that it is necessary.

Quantize a continuous random variable


Set layout with JPanels

Question: I haven't really worked with Swing at all in Java. I'm experimenting with it. I want to make a set layout that the size can't be changed. I've seen alot of things suggesting to use Layout managers to add multiple JPanels into a JFrame.
However, all the tutorials I've seen involving layout managers say it allows for the user to resize the screen. The layout I want has a rectangle going along the left hand side, a  thin rectangle going along the bottom, and a third rectangle taking up the rest of the space. I attempting it using an Absolute layout but it just doesn't want to work for me.
import java.awt.Color;

import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.SwingUtilities;

public class Test extends JFrame {

    public Test() {
        JPanel rect1 = new JPanel();
        rect1.setBounds(101, 650, 900, 50);
        rect1.setBackground(Color.RED);
        getContentPane().add(rect1);

        JPanel rect2 = new JPanel();
        rect2.setBounds(0, 650, 100, 1000);
        rect2.setBackground(Color.BLUE);
        getContentPane().add(rect2);

        JPanel rect3 = new JPanel();
        rect3.setBounds(101, 700, 900, 950);
        rect3.setBackground(Color.GREEN);
        getContentPane().add(rect3);

       setTitle("TEST");
       setSize(1000, 700);

       setLocationRelativeTo(null);
       setDefaultCloseOperation(EXIT_ON_CLOSE);
    }

    public static void main(String[] args) {

        SwingUtilities.invokeLater(new Runnable() {
            @Override
            public void run() {
                Test ex = new Test();
                ex.setVisible(true);
            }
        });
    }
}

Can someone help me properly make three Jpanels in a Jframe in this layout (all with different colors)?
Answer: You might be able to achieve the same thing using a BorderLayout as the bases or even a GridBagLayout.
The main piece you are missing is the fact that layout managers use (or can use depending on the layout manager) the component's preferred/minimum/maximum size
Basically, what you would do is define a custom component (extending from something like JPanel) and override it's getPreferredSize method and return the required value you need.  Depending on the layout manager, you may also need to override the getMaximumSize and getMinimumSize methods as well.
For example...
Basically, this shows the "default" size and what happens when the screen is resized...

import java.awt.BorderLayout;
import java.awt.Color;
import java.awt.Dimension;
import java.awt.EventQueue;
import java.awt.GridBagConstraints;
import java.awt.GridBagLayout;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.UIManager;
import javax.swing.UnsupportedLookAndFeelException;

public class FixedSizeLayout {

    public static void main(String[] args) {
        new FixedSizeLayout();
    }

    public FixedSizeLayout() {
        EventQueue.invokeLater(new Runnable() {
            @Override
            public void run() {
                try {
                    UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
                } catch (ClassNotFoundException | InstantiationException | IllegalAccessException | UnsupportedLookAndFeelException ex) {
                }

                JFrame frame = new JFrame("Testing");
                frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
                frame.setLayout(new BorderLayout());
                frame.add(new TestPane());
                frame.pack();
                frame.setLocationRelativeTo(null);
                frame.setVisible(true);
            }
        });
    }

    public class TestPane extends JPanel {

        public TestPane() {
            setLayout(new GridBagLayout());
            GridBagConstraints gbc = new GridBagConstraints();
            gbc.gridx = 0;
            gbc.gridy = 0;

            add(new ContentPane(), gbc);
            gbc.gridx++;
            add(new LeftPane(), gbc);
            gbc.gridwidth = 2;
            gbc.gridx = 0;
            gbc.gridy = 1;
            add(new BottomPane(), gbc);

        }

        @Override
        public Dimension getPreferredSize() {
            return new Dimension(200, 200);
        }

    }

    public class ContentPane extends JPanel {

        public ContentPane() {
            setBackground(Color.GREEN);
        }

        @Override
        public Dimension getPreferredSize() {
            return new Dimension(150, 150);
        }

        @Override
        public Dimension getMinimumSize() {
            return getPreferredSize();
        }

        @Override
        public Dimension getMaximumSize() {
            return getPreferredSize();
        }

    }

    public class BottomPane extends JPanel {

        public BottomPane() {
            setBackground(Color.RED);
        }

        @Override
        public Dimension getPreferredSize() {
            return new Dimension(200, 50);
        }

        @Override
        public Dimension getMinimumSize() {
            return getPreferredSize();
        }

        @Override
        public Dimension getMaximumSize() {
            return getPreferredSize();
        }

    }

    public class LeftPane extends JPanel {

        public LeftPane() {
            setBackground(Color.BLUE);
        }

        @Override
        public Dimension getPreferredSize() {
            return new Dimension(50, 150);
        }

        @Override
        public Dimension getMinimumSize() {
            return getPreferredSize();
        }

        @Override
        public Dimension getMaximumSize() {
            return getPreferredSize();
        }

    }
}

Now, if you prefer, you can make the screen non-resizable, but I for one won't like you.  I prefer to use the power of the layout managers and allow users to make decisions about how they want to view the content ... where I can ... but that's just me (I don't like non-resizable windows except in the case of some dialogs)
Comment: answer me 1 Question. When do you sleep ?
Comment: Now, i really feel sad for not being able to post the question such that i could upvote and accept the answer. :P. Anyway challenge accepted. :| :))
Comment: @sage Keep putting up quality answers as you are ;)
Answer: You can use setResizable(). Please refer to below code fragment.
   setResizable(false); // this will not allow resizing
   setLocationRelativeTo(null);
   setDefaultCloseOperation(EXIT_ON_CLOSE);

Answer: AbsoluteLayout is no go. Don't do that. You are probably looking for BorderLayout. Check the tutorial: How to Use BorderLayout for details. If you don't want your JFrame to be able to re-size use frame.setResizable(false); on it.
Answer: Just use simple BorderLayout and it will resize automatically:
JPanel panel = new JPanel(new BorderLayout());
panel.add(greenPanel, BorderLayout.CENTER);
panel.add(redPanel, BorderLayout.SOUTH);
panel.add(bluePanel, BorderLayout.EAST);

Answer: You can use MigLayout and replace BorderLayout.CENTER by "dock center":
JPanel panel = new JPanel(new MigLayout(" insets 0"));
panel.add(greenPanel, "dock center");
panel.add(redPanel, "dock south");
panel.add(bluePanel, "dock east");

Read more about MigLayout here: http://www.miglayout.com/QuickStart.pdf

Set layout with JPanels


getting the max of each group in a MySql query

Question: Whilst I realise this question has been asked in many forms before, sadly the threads that deal with this problem have no helped my situation.
I have a table that looks like this for example:
id|forename|surname|house|gen|academic|sense|dedicated|balance|respect
-------------------------------------------------------------------------
1 |dav1    |jone1  |n    |1  |2       |3    |4        |5      |18
2 |dav2    |jone2  |s    |1  |2       |3    |4        |5      |9
3 |dav3    |jone3  |w    |1  |2       |3    |4        |5      |10
4 |dav4    |jone4  |w    |1  |2       |3    |4        |5      |11
5 |dav5    |jone5  |n    |1  |2       |3    |4        |5      |12
6 |dav6    |jone6  |h    |1  |2       |3    |4        |5      |13
7 |dav7    |jone7  |h    |1  |2       |3    |4        |5      |14

And the output I would like is:
id|forename|surname|house|total
-------------------------------
1 |dav1    |jone1  |n    |33
4 |dav4    |jone4  |w    |26
2 |dav2    |jone2  |s    |24
7 |dav7    |jone7  |h    |29

So it gets the combined total of the six integer columns, then takes the highest from each of the houses and returns the total, the name, the house and the id.
I have done quite a few iterations, and it seems I need to do a nested table (I dont know the exact term for it), but this is what I have been working along the lines of:
SELECT utable.id, utable.forename, utable.surname, utable.house, (utable.gen+utable.academic+utable.sense+utable.dedicated+utable.balance+utable.respect) AS utable.points
FROM (
    SELECT house, max(`gen`+`academic`+`sense`+`dedicated`+`balance`+`respect`) AS pts FROM users             GROUP BY house
) AS ntable
INNER JOIN users AS utable ON utable.house = ntable.house AND utable.points = ntable.pts

However it isnt working, and in particular right now is says the utable.points has an alias that was previously found. I think it has something to do with how I am summing the columns to make the total points but im not sure.
Any help is greatly appreciated :)
(while I know this will get flagged up as a repeat request, I have tried the other threads and a few websites to try and get this to work to no avail. I am testing it in phpMyAdmin server type is MariaDB 5.5.46-MariaDB-1ubuntu0.14.04.2 - (Ubuntu))
Thank you!!!
Answer: You are nearly there, basically you need a sub-query to get information like above:
SELECT u.id, u.forename, u.surname, u.house, max_u.pts
FROM users u, (
    SELECT house, max(gen+academic+sense+dedicated+balance+respect) pts
    FROM users
    GROUP BY 1
) AS max_u
WHERE u.house = max_u.house
and gen+academic+sense+dedicated+balance+respect = max_u.pts

Comment: Thats perfect, thank you - I never thought to put the sum in the WHERE part of the statement. Really appreciate it, thank you :)
Answer: Rather than doing joins you can simplify it by ordering on the inner query and then grouping on the outer query:
select * from (select id, forename, surname, house, (gen + academic + sense + dedicated + balance + respect) as total from Table1 order by house, total desc) x group by house;

This is a bit of a trick because if you group in descending order by total it will automatically rank the max at the top of each house group. Then when you group by house it will take the max value for each house.
The only issue I can forsee that you might have to think about is what if two rows have are tied for the same maximum value?
Here's an SQL Fiddle: http://sqlfiddle.com/#!9/25bd8e/14
The above is a MySQL solution without needing joins. If you are using PostgreSQL instead you can use this:
select distinct on (house) id, forename, surname, house, (gen + academic + sense + dedicated + balance + respect) as total from Table1 order by house, total desc;

Comment: @SIDU If you are going to claim my code is incorrect please provide a counter example of how my code will fail.
Comment: He wants the join with original row based on the max points - you can try to run your SQL and it will not work as expected.
Comment: Can you provide an example of values where my code will fail. The only situation where I can see an issue is with totals that tie for the max value.
Comment: You can simply use above data into your test table, and run your SQL, it will not produce the result as expected.
Comment: Actually if you look at my SQL Fiddle that I posted it uses the same data and gets the exact results that he wanted plus is a bit faster in execution time.
Comment: I see what you mean now. The SQL itself is not kind-of-valid, however MySQL works in this case. If you run this SQL in PostgreSQL, it should be syntax error
Comment: Thank you for explaining. I am not used to PostgreSQL.
Comment: I have also added a PostgreSQL alternative that seems to do the job, though SQL Fiddle was being a bit buggy so couldn't fully test it.
Comment: Hi kojow7, I have copied your codes from SQL Fiddle into my local postgres (v9.4) it shows exactly same error. I understand standard SQL requires you to include all columns names (except the aggregated) into the group by -- luckily or Brain-washingly by MySQL, such invalid SQL does not return an error
Comment: I think you tried the MySQL code option with your PostgreSQL. I also posted a valid PostgreSQL one that should work with your local install using "distinct on". However, I agree with what you are saying and did not realize MySQL and PostgreSQL do not follow the standards.


How to include or hit a php file from custom module in Drupal 8

Question: I am working on a custom Drupal 8 module. I have a seperate php script as a .php file. I want to include that file in my Drupal 8 module, in simple words I want to hit that .php file by running my module.
What is the best way to do this?
Here is my module code:
    function mymodule_cron(){
     //include('myphpscript.php');
    }

Thanks
Comment: Drupal 8 is built on OOP and use PS4 autoloading with classes. In drupal 8 you create a custom service and call that. The autoloader will handle the loading of it, you don't need to include it yourself.
Comment: thanks for the comments. Please upvote it so that other will get benefits from this question.
Comment: What's in your script?
Comment: @jaypan: Can you provide code example?
Comment: @leymannx: Script can hold any functionality like a loop which runs 10 times and echo message.
Comment: You can see an example of creating a custom service here: https://www.drupal.org/docs/8/api/services-and-dependency-injection/structure-of-a-service-file
Comment: Consider implementing a mini custom form with a button to start a batch process. https://opensenselabs.com/blogs/tech/how-use-batch-api-drupal-8
Comment: Thanks for the comments. I think I cannot understand why people mark my question negative?
Comment: I don't know why it was downvoted. I upvoted it. I think a good answer would help explain a history lesson about drupal 7 and how files were included there. Heck, that's still available in drupal 8 - `module_load_include`. But implementing your own cron manager / plugin system is probably a bit too broad.
Answer: Standard PHP include, require, and require_once need a known path. Often times this is not known when making Drupal contributed modules.
In Drupal 7, the standard way to include a file inside a module was using the module_load_include function. This is still available, but may not be suitable for test-driven development.
In Drupal 8, as @Jaypan references in a comment above, Structure of a a service file, we can create services and use methods on a class to do things. This has the benefit of injecting dependencies to do things.
function mymodule_cron() {
  // This still works, but shouldn't be used for classes in the autoloader. Generally the ".inc" file extension was used instead of ".php", but it would work.
  module_load_include('php', 'mymodule', 'myphpscript');

  // Implementing your own service if you want to use \Drupal.
  \Drupal::service('mymodule.myservice')->callMethodToDoThingOnCron();
}


Purpose of uppercase VOID macro & INT typedef in winnt.h

Question: Anyone any clue as to why there is an uppercase VOID macro defined in the winnt.h header?
To make matters more confusing, VOID is a macro, whereas CHAR, SHORT, INT, and LONG are typedefs.
See the relevant excerpt from winnt.h:
#ifndef VOID
#define VOID void
typedef char CHAR;
typedef short SHORT;
typedef long LONG;
#if !defined(MIDL_PASS)
typedef int INT;
#endif
#endif

A historical reason perhaps for doing VOID* pointer instead of void* pointer?
EDIT: What is even more troubling is to see people using VOID instead of void when doing Windows programming today. You can also see it as part of MSDN docs, e.g. http://msdn.microsoft.com/en-us/library/bb205867(v=vs.85).aspx
Answer: The Windows API is old. Really old. Older than the official C standard old.
This means that, at the start, the Windows API had to deal with all kinds of ancient C compilers, with different levels of language support. Some might not support void. Some might have an int type that's not compatible with what windows thought was int. Some might not understand short. As a workaround, the Windows API provides upper-case portable equivalents that are aliased to whatever works for that particular compiler.
Of course, with modern compilers, things have settled down quite a bit. Everyone supports void, for example. However, in order to maintain compatibility with old code that uses these upper-case macros, the #defines and typedefs have to remain.
Comment: Raymond Chen has a lot of interesting blog posts about the history of Windows.  Although he hasn't written about `VOID` as far as I'm aware, he has written about all of the boolean variants in Win32: http://blogs.msdn.com/b/oldnewthing/archive/2004/12/22/329884.aspx and http://blogs.msdn.com/b/oldnewthing/archive/2011/03/28/10146459.aspx .
Comment: The two blog posts linked from Adam Rosenfield's comment are good, but they're specific to booleans, which is a specialized area.  Raymond Chen also wrote one that specifically addresses this question:  https://blogs.msdn.microsoft.com/oldnewthing/20080325-00/?p=23013
Answer: The original reason is that the Win32 API was originally supposed to be language independent, so they made up their own names (following their naming convention), and then provided a C implementation of that. In reality, much of what they defined tends to be ignored by most people using other languages (and largely even by people using C, for that matter).
As far as why VOID is a #define instead of a typedef, that's pretty simple: the required typedef would be typedef void VOID;, but C (as "defined" by Microsoft's C compilers from that time) simply doesn't allow that, so they use a macro instead.
Comment: I did have the typedef typed in backwards. Even when corrected, however, it's not allowed by Microsoft's compilers from that time, which (of course) is what Microsoft treated as the definition of C. Most compilers allowed it then, and Microsoft's do now, but I'm reasonably certain their older ones didn't.
Comment: No, you have it backwards.  The required typedef would be `typedef void VOID`, which is perfectly allowed.  Try it.


Qual a finalidade do atributo global contentEditable no elemento html?

Question: Em qual situação real ou quando será necessário o uso do atributo global contentEditable, pelo elemento html, visto que esse é um elemento raiz e que editá-lo seria o mesmo que modificar, inserir ou apagar os elementos folha? Exemplo:

   ...

Obrigada!
Answer: Isso quer dizer que é possivel editar/alterar o conteúdo do elemento.
O atributo contentEditable aceita 3 estados:

"true" o conteúdo do elemento pode ser editado.
"false" o conteúdo do elemento não pode ser editado.
"inherit" indica que o estado é herdado do elemento ancestral que tenha esse atributo defenido.

O ultimo caso, combinado com o teu exemplo faz com que o conteúdo de elementos fique ou não editável dependendo de mudar num só lugar (exemplo).
Mais informação (em Inglês) na MDN: contentEditable


How to properly delete music file in SD card and how to create new file?

Question: I have created one folder and some JSON file in SD card via programmatically, It's creating the folder and file no problem, But it's showing only on mobile file directory, It's not showing in the system after connecting my mobile to the system. if I copy and paste the same folder in mobile then it's showing in the system the copied folder only original folder not yet showing. for reference, I have added my mobile and system directory images.
here in mobile original and copied folders are showing
here only copied folder only showing
Same like that, some music files I have in SD card, that one I am trying to delete.

path = (sdcard file path )
  File SongFile = new File(path);
  songFile.delete();

This function deletes the file correctly on my SD card.
But after deleting also in my system it's showing if I trying to play its showing some error. This deleted file not deleted properly. After delete also I am getting the same file programmatically in my application.
In my application also showing deleted files name and details.
Answer: I suspect that you didn't delete the entry from the Media Store and this is the reason why you still see it.

Get the _id of the entry searching it by data parameter with a
ContentResolver
Delete the entry using the above _id

Here a snippet:
public void deleteFromMediaStore(Context context, String pathToDelete) {
        ContentResolver contentResolver = context.getContentResolver();
        if (contentResolver != null) {
            Cursor matchingIndex = contentResolver.query(Media.EXTERNAL_CONTENT_URI, new String[]{"_id", "_data"}, "_data = ?", new String[]{pathToDelete}, null);

            if(matchingIndex != null && matchingIndex.getCount() > 0) {
                matchingIndex.moveToFirst();
                while (!matchingIndex.isAfterLast()) {
                    context.getContentResolver().delete(ContentUris.withAppendedId(Media.EXTERNAL_CONTENT_URI, (long) matchingIndex.getInt(matchingIndex.getColumnIndex("_id"))), null, null);
                    matchingIndex.moveToNext();
                }
                matchingIndex.close();
            }
        }
    }

Comment: Hi, thanks for your reply. Now it's deleting properly and delete function working fine.
Comment: I have created one folder and some JSON file in SD card via programmatically, It's creating the folder and file no problem, But it's showing only on mobile file directory, It's not showing in the system after connecting my mobile to the system. if I copy and paste the same folder in mobile then it's showing in the system the copied folder only original folder not yet showing. for reference, I have added my mobile and system directory images. In my question, I have added reference images
Comment: @RegunathSiva I think you should not worry about this, it's pretty common. Reboot the device and show show fine


Finding a route along points of interest

Question: I have a map with a destination (the red dot below), and a number of points of interest (the yellow, green and blue dots).

I'm trying to find a path to the destination, but the starting point is undefined - I just want it to pass as many of the points of interest as possible, without the route being to circuitous.
For example the following (pink line) would be a good route in this case:

The yellow dot is the POI furthest away from the destination (not useful in this case), the green ones are the four next furthest away.
Can anyone suggest an algorithm which would be suitable for this?
Is this a suitable problem to turn into a graph? The "not too circuitous" requirement seems to suggest that, but how would I reconcile that with wanting to pass as many POIs as possible along the way?
Edit: To clarify the "not too circuitous" requirement. I just want it to be a plausible route, for example turning a maximum of 90 degrees for the sum of all the corners. The POIs will always be nearby the destination so length isn't really an issue.
Comment: Finding an optimal path is NP-Hard, it is pretty much a variation of Traveling Salesman Problem, are you familiar with it?
Comment: As @amit states above, that would be a [TSP](http://en.wikipedia.org/wiki/Travelling_salesman_problem).  "As many POIs as possible" is always all of them, unless you set conditions to dismiss points, ie. maximum length of journey, etc.
Comment: `...the starting point is undefined...` use the destination as starting point if the graph is undirected.
Comment: "As many POIs as possible". Do you have a constraint like elapsed distance?
Comment: Thanks for the quick feedback! @Orbling Let's say I need to pass at least 15 POIs - "maximum length of journey" does sound like my kind of restriction, could maybe have maximum heading change of 90 degrees or something.
Comment: @UmNyobe using the destination as a starting point does sound like a good idea, thanks :)
Comment: Suggest if you write an algorithm, it can take basic constraints like number of points min and max journey length.  Of course, if you specify both, there may be no solution.
Answer: The problem can definetly be reduced to a graph G=(V,E) where V are all your POI, and E in this case is V x V (edge between all vertices). You also need to produce a weight function w:E->R such that w(u,v) = distance between u and v
The problem is actually a variation of the Traveling Salesman Problem, so it is NP-Hard (So there is no known polynomial solution) - but have a look around, there are many heuristics for this problems.
Also - if you do not expect many POIs (say 20-30) - a dynamic programming solution can be used to find the optimal path between all points.
Answer: It seems like you could turn it into a graph, assigning a negative weight to each of the points of interest (probably by reducing the value of any edge leading into that point) and then plugging said graph into the Bellman-Ford algorithm]1, which permits negative-length edges. The only problem might come if two POIs are very close together, so some sort of pruning method might possibly be required.


Python 'pip install' not working/recognised

Question: I've been trying to use the "pip install" python command for a while but have had no luck.
I have checked that pip came in the python installer package, tried repairing the program on the installer and have even tried installing the Python 3.10 app from the Microsoft store (instead of using IDLE) and running the command from there. Nothing has worked at the moment.
This is what I get (IDLE Python 3.10 in the main idle shell terminal):
pip install auto-py-to-exe
SyntaxError: invalid syntax

Comment: `SyntaxError` is likely the python interpreter telling you that what you pasted into the terminal is not valid py code.
Comment: Where are you running this command? `pip` is a command line program. What happens when you run `pip` in a command line?
Comment: Yes I was writing the commands on the main terminal (IDLE shell).
Comment: https://stackoverflow.com/search?q=%5Bpip%5D+SyntaxError
Answer: You're running pip inside of Python. That's not how it should be used. Open a terminal and try
python -m pip install auto-py-to-exe

or
pip install auto-py-to-exe

Comment: @aPythonUser You were running the command in a **Python** terminal. Open command prompt, or the bash terminal, etc... The error you recieved is specifically a Python error, you would not see it unless you wrote the command in Python. If you are truly in a terminal, you've managed to get yourself into python (probably by writing `python`). Type `quit()` and try again.
Comment: Yes I was running the command in the terminal but it does not work for both commands.


Install a distro from an online repository core make file

Question: I would like to use the hosting tool Aegir to install this distribution from a make file directly:
http://cgit.drupalcode.org/elmsmedia/plain/drupal-org.make
what is the "workaround" to be able to insert the above URL as a string in Aegir platform make file field here:

I have tried a lot of distributions but all failed.
thanks
Answer: Unfortunately the Drush Make infra does not support this use-case well.
The quickest wordaround is to clone the project Git repository and provide a local path to the makefile in the Aegir UI.
For elmsmedia I guess the best makefile would be http://cgit.drupalcode.org/elmsmedia/tree/local.make.example
OpanAtrium has http://cgit.drupalcode.org/openatrium/tree/build-openatrium.make for this same purpose.
Comment: What did you priovide as a makefile path? ```/tmp/myopenatriumclone/build-openatrium.make```
Comment: thanks, I have tried to use the OpenAtrium make file, maybe it's my outdated version of Aegir, but it didn't work like expected: http://www.awesomescreenshot.com/image/221652/db00d11630403085789da2f668dfe6f5
Comment: i used the path you provided in the answer.
Comment: Which Aegir and Drush versions do you have?
Comment: ah of course!! i have Drush 4.6-dev and Aegir BOA-2.0.8. they must be too outdated, right?
Comment: Yes, fairly outdated


Extract audio frequency from instrument to find a musical note

Question: I'm trying to develop an Android app that extracts audio frequency from an instrument. I'm using Fast Fourier Transform method with Jtransforms. Here is what I have so far:
<code>public class MainActivity extends Activity {

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    new readFrequencies().execute();
}

@Override
public boolean onCreateOptionsMenu(Menu menu) {
    // Inflate the menu; this adds items to the action bar if it is present.
    getMenuInflater().inflate(R.menu.main, menu);
    return true;
}

private class readFrequencies extends AsyncTask<Void,Integer,Integer> {

        @Override
        protected Integer doInBackground(Void... arg0) {
            AudioRecord recorder = null;
            int bufferSize = 0;
            boolean recording = true;

            int rate = 8000;
            short audioFormat = AudioFormat.ENCODING_PCM_16BIT;
            short channelConfig = AudioFormat.CHANNEL_IN_MONO;

            try {
                bufferSize = AudioRecord.getMinBufferSize(rate,channelConfig, audioFormat);

                recorder = new AudioRecord(AudioSource.DEFAULT, rate,
                    channelConfig, audioFormat, bufferSize);

                if (recorder.getState() == AudioRecord.STATE_INITIALIZED) {
                    /*
                     *  Android 4.1.2
                     *
                    int recorder_id = recorder.getAudioSessionId();
                    if (NoiseSuppressor.isAvailable()) NoiseSuppressor.create(recorder_id);
                    */
                }
                else {
                    Toast.makeText(getApplicationContext(),
                            "Error en la inicialización", Toast.LENGTH_SHORT).show();
                }
            } catch (Exception e) {}

            short[] audioData = new short[bufferSize];

            if (recorder != null) {
                while (recording) {
                    if (recorder.getRecordingState() == AudioRecord.RECORDSTATE_STOPPED) {
                        recorder.startRecording();
                    }
                    else {
                        int numshorts = recorder.read(audioData,0,audioData.length);
                        if ((numshorts != AudioRecord.ERROR_INVALID_OPERATION) &&
                            (numshorts != AudioRecord.ERROR_BAD_VALUE)) {

                            //  Hann
                            double[] preRealData = new double[bufferSize];
                            double PI = 3.14159265359;
                            for (int i = 0; i < bufferSize; i++) {
                                double multiplier = 0.5 * (1 - Math.cos(2*PI*i/(bufferSize-1)));
                                preRealData[i] = multiplier * audioData[i];
                            }

                            DoubleFFT_1D fft = new DoubleFFT_1D(bufferSize);
                            double[] realData = new double[bufferSize * 2];

                            for (int i=0;i<bufferSize;i++) {
                                realData[2*i] = preRealData[i];
                                realData[2*i+1] = 0;
                            }
                            fft.complexForward(realData);

                            double magnitude[] = new double[bufferSize / 2];

                            for (int i = 0; i < magnitude.length; i++) {
                                double R = realData[2 * i];
                                double I = realData[2 * i + 1];

                                magnitude[i] = Math.sqrt(I*I + R*R);
                            }

                            int maxIndex = 0;
                            double max = magnitude[0];
                            for(int i = 1; i < magnitude.length; i++) {
                                if (magnitude[i] > max) {
                                    max = magnitude[i];
                                    maxIndex = i;
                                }
                            }

                            int frequency = rate * maxIndex / bufferSize;
                            publishProgress(frequency);
                        }
                        else {
                            if (numshorts == AudioRecord.ERROR_BAD_VALUE) {
                                Toast.makeText(getApplicationContext(),
                                        "ERROR_BAD_VALUE", Toast.LENGTH_SHORT).show();
                            }
                            else {
                                Toast.makeText(getApplicationContext(),
                                        "ERROR_INVALID_OPERATION", Toast.LENGTH_SHORT).show();
                            }

                            return -1;
                        }
                    }
                }

                if (recorder.getState() == AudioRecord.RECORDSTATE_RECORDING)
                    recorder.stop(); //stop the recorder before ending the thread
                recorder.release();
                recorder=null;
            }
            return 0;
        }

        protected void onProgressUpdate(Integer... f) {
            TextView texto = (TextView) findViewById(R.id.texto);
            texto.setText(String.valueOf(f[0]));
        }

        protected void onPostExecute(Integer f) {
            TextView texto = (TextView) findViewById(R.id.texto);
            int frecuencias = f.intValue();
            texto.setText(String.valueOf(frecuencias));
        }
}
</code>
}
With this code I am able to obtain the exact frequency from a frequency generator which produces pure signals. However, when I try the same with an instrument, I obtain random frequencies. I know that, when it comes to real instruments, the signals produced contain harmonics which may influence on the final result, but I don't know to obtain the real frequency in this case. Could anybody help me?
I've used TarsosDSP and I also have tried the Autocorrelation method, but I haven't been able to get what I wanted.
Thanks in advance.
Comment: This question has been asked many many times. Here is a tutorial that may help with the basics. http://blog.bjornroche.com/2012/07/frequency-detection-using-fft-aka-pitch.html
Comment: Ignore any tutorial that uses only a single FFT peak magnitude for musical pitch estimation.  That method most often does not work on real sound.
Answer: There is a huge difference between the most powerful frequency component in a spectrum and the perceived pitch by a human listener.
This academic paper is probably the definitive review of approaches to solving some of the problems of pitch detection, but does not address the perceptual issues you will need to deal with using real signals.  At the very least you'll need to account for string and wind-column instruments that jump octave at the beginning and end of notes, and missing fundamentals.  This will be a particular lottery on Android where microphones on devices are not in any way consistent.
Searching for the peak frequency in the spectrum is no better than using autocorrelation and frequently wrong with real signals.
The FFT approach (more properly, using STFT and frequency estimation from the phase differences) can be made to work, but only with a lot of post-processing of the spectrum. Several commercial chromatic tuner apps use an FFT-based approach (I can attest to this having written one).
You might like to have a look how some of the feature extraction plugins for Sonic Visualiser go about solving this problem.
I also recommend reviewing the vast number of previous questions on here asking pretty much the same thing - usually questioners wanting to build Guitar tuners.
Answer: For what it's worth - I believe electronic tuners for musical instruments (e.g. guitar tuners, etc.) don't do it this way.  Instead of doing a FFT like you're doing, they are simply measuring the period of the waves - (i.e. the time between the zero crossings), then computing the frequency from the period.
Comment: There are a variety of methods, but zero-crossing is not one of them.
Comment: See:
http://stackoverflow.com/questions/65268/music-how-do-you-analyse-the-fundamental-frequency-of-a-pcm-or-wav-sample
Comment: Older cheap guitar tuners do work this way, but don't work very well.  Bandpass filtering before measuring zero crossings sometimes makes them slightly less inaccurate.
Comment: Not a method to be emulated, nor a method that simply works by itself. For a tone of any complexity, filtering is required.
Comment: @HOtpaw2 of course, with a guitar tuner, it only has to detect one pitch at a time - in in total only 6. Lots of (real-world) tuners used all kind of tricks such as bandpass filters, phase-locked loops, or distorting the signal into a square-wave and then counting clock edges.  It's very instructive to look for patents.


Limit of a composite function approaching infinity.

Question: I always have problem in applying limits in a composite function. The theorem says that
if $f$ is continuous at $b$ and $\lim_{x \to a}g(x) = b$, then $\lim_{x \to a}f(g(x)) = f(\lim_{x \to a}g(x))$. But I never understood this theorem completely. For example, I tried to solve the following question using the theorem but the answer I got was wrong.
$$
\lim_{x \to \infty} \cos^2\Bigl(\pi\sqrt[3]{n^3+n^2+2n}\Bigr) = \cos^2\Bigl(\lim_{x \to \infty}\Bigl(\pi\sqrt[3]{n^3+n^2+2n}\Bigr)\Bigr)
$$
Comment: Well the first limit doesn't exist, and $\cos^2(\infty)$ is not defined. More generally, the theorem can't be applied here since for it to be applied, not only must $\lim\limits_{n\to\infty}g(n)$ be finite, but $f$ must be continuous at that point.
Comment: @DonThousand The limit exists and it is 1/4.
Comment: Is this a integer limit or a real limit?
Comment: @DonThousand What does that mean?
Comment: In the limit definition, is it something like $\forall \varepsilon>0, \exists n\in\mathbb N$ such that for all $y\in\mathbb N$, $y>n$, ... or such that for all $y\in\mathbb R$, $y>n$ ...
Answer: The theorem you're stating is the definition of continuity for a function. Notice that if
$$\lim_{x \to a}f(g(x)) = f(\underbrace{\lim_{x \to a}g(x)}_{\color{blue}{b}}) = f(b)
$$
you're just saying that $f(g(x))$ is approaching $f(b)$ as it gets closer to said value.

Now, notice that if you want to apply this to your question, you'd be implicitly assuming that $\sqrt[3]{n^3+n^2+2n}$ is a continuous function. With this assumption, it is true that
$$
\lim_{n \to \infty } \sqrt[3]{n^3+n^2+2n} = \infty
$$
but since $\cos^2(\infty)$ is undefined, if you're dealing with a continuous limit then the limit does not exist.

There is one more possible interpretation of your problem. In this assumption, you're assuming that $\sqrt[3]{n^3+n^2+2n}$ is a sequence instead of a function. This means you're only evaluating $\sqrt[3]{n^3+n^2+2n}$ at values of $n \in \mathbb{N}$, instead of any real value. In this case, you'd be working with a limit of a sequence instead of the limit of a function, where here you can get a finite answer as follows.
You can show that
$$
\sqrt[3]{n^3+n^2+2n} \sim n + \frac{1}{3} + \mathcal{O}\left(\frac{1}{n}\right)
$$
as $n \to \infty$. You can show this taking the Laurent expansion of $\sqrt[3]{n^3+n^2+2n}$ at infinity, for example. From here we would see that
\begin{align}
\lim_{n \to \infty} \cos^2\left(\pi\sqrt[3]{n^3+n^2+2n}\right)  &=  \lim_{n \to \infty} \cos^2\left(\pi n + \frac{\pi}{3} \right)=  \left[\frac{1}{2}\lim_{n \to \infty}\cos\left(\pi n\right) - \frac{\sqrt{3}}{2}\lim_{n \to \infty} \sin\left(\pi n\right)\right]^2
\end{align}
And here's where you use the hypothesis that you're just taking integer values. Since $n \in \mathbb{N}$, then $\sin(\pi n) = 0$ and $\cos(\pi n) = (-1)^n$ for any $n \in \mathbb{N}$, even if we take the limit. With this we then get that
$$
\lim_{n \to \infty} \cos^2\left(\pi\sqrt[3]{n^3+n^2+2n}\right)   =  \left[\frac{1}{2}(-1)^n- \frac{\sqrt{3}}{2} (0)\right]^2= \boxed{\frac{1}{4}}
$$
where you get your desired result.
Note that the limits $\lim_{n \to \infty}\sin\left(\pi n\right)$ and $\lim_{n \to \infty}\cos\left(\pi n\right)$ don't exist if $n$ can be any real value as both these functions oscillate infinitely between $-1$ and $1$, but by restricting the values of $n$ to only integer values we can give some values to these limits.
Comment: Are you sure that $\cos(\lim_{n\to\infty}\pi n)$ is meaningful?
Comment: @egreg, you're correct. The sequence hypothesis should have been used with the limit taken outside, again valid by the continuity of $\cos^2(x)$.
Answer: Suppose $f$ is continuous at $c$ and defined in a suitable neighborhood of $c$. Then, if $\lim_{x\to a}g(x)=c$, we can deduce that
$$
\lim_{x\to a}f(g(x))=f(c)
$$
so you can "push in" the limit. This can also be extended to the case when $c$ is infinity, but in this case you need that $\lim_{y\to\infty}f(y)$ exists.
The case of sequences is similar. But in your case $\lim_{y\to\infty}\cos^2y$ does not exist.
How do you deal with this limit? The fact that you have the cosine squared is the key. Indeed, you can see that
$$
\cos^2(\pi x_n)=\cos^2(\pi x_n-\pi n)
$$
In your case $x_n=\sqrt[3]{n^3+n^2+2n}$ and
$$
x_n-n=\frac{n^3+n^2+2n-n^3}{\sqrt[3]{(n^3+n^2+2n)^2}+n\sqrt[3]{n^3+n^2+2n}+n^2}
$$
and you can use the trick of pulling $n^3$ from the cube roots in the denominator to get
$$
x_n-n=\frac{1+(2/n)}{\sqrt[3]{(1+(1/n)+(2/n^2))^2}+\sqrt[3]{1+(1/n)+(2/n^2)}+1}
$$
Now
$$
\lim_{n\to\infty}\pi(x_n-n)=\pi/3
$$
and you can push the limit inside:
$$
\lim_{n\to\infty}\cos^2(x_n)=\lim_{n\to\infty}\cos^2(\pi(x_n-n))=
\cos^2\Bigl(\lim_{n\to\infty}\pi(x_n-n)\Bigr)=\cos^2\frac{\pi}{3}=\frac{1}{4}
$$


What is rude jazz?

Question: In a lecture supporting his new book Stomp and Shout, Northwest rock historian Peter Blecha mentioned a genre from the 1940s or 50s called rude jazz. I have never heard of this before, and it might be a specifically Northwest thing. Is anyone else aware of such as genre? What is it? He described it as using jazz instrumentation and methods, but "it's not nice".  I asked him if it was related to jump blues, and he seemed to think that was plausible, but nothing more.
This is a thing??
Answer: "Rude Jazz" appears to be Blecha's own term for a musical trend localized in the Pacific northwest, but which is not explicitly defined in his books. It seems to have been a hard-driving, jazz-inflected version of 1950s R&B and rock 'n' roll.
From Stomp and Shout, "Introduction: The Sea-Port Beat":

Once upon a time—decades prior to grunge rock being even a gleam in Sub Pop's eye—the Pacific Northwest was the birthing ground to a rockin' R&B scene that created its own unique strain of rock 'n' roll. This was a distinctive rude-jazz–tinged mutation that erupted in the seaport towns of Seattle, Tacoma, and Portland between the approximate years of 1958 and 1965 and would come to be called the original Northwest Sound—or in some circles the Sea-Port Beat or Sea-Port Sound. (via book preview on Scribd)

From Sonic Boom, "Chapter 6: That Sound":

The task of creating a Northwest Sound would be left to ... players whose musicality was admittedly informed by jazz traditions, but who would lean more toward the emerging trends of rhythm and blues, and then rock 'n' roll. In fact, this regional musical mutation — which was forged in the rowdy dancehalls of the areas biggest seaport towns of Seattle, Portland, and Tacoma — was initially referred to ... as the "Sea-Port Beat." In its earliest phase, the music was based on a shared repertoire of recent and current R&B hits and a lot of "rude jazz" saxophone-led tunes.

The "Sea-Port Beat," or "original Northwest Sound," as first defined by Seattle's Dave Lewis Combo was essentially a regional sub-strain of rock 'n' roll that was firmly anchored in the West Coast R&B previously honed by Ray Charles and Richard Berry — along with the horn-driven sounds of the bands of Big Jay McNeely, Bobby "Blue" Bland, and Hank Ballard. But the critical ingredient ... was a tendency to ... alter their overall vibe ... in a way that rocked hard and had no room for rhythmic subtlety. (page 39)

Little Green Thing (album) by Dave Lewis

And from Music in Washington:

Just consider some of the local African-American "rude jazz" and rhythm and blues whose names alone evoke good times gone by, like Tacoma's Swinging Esquires and Seattle's Savoy Boys and Billy Tolles and the Vibrators. (pages 8–9)

Aside from these, I found no other references to "rude jazz" via Google or Google Scholar.
Comment: Thanks for that!  For what it's worth, he says he got the term from older musicians he has interviewed, so I don't think it's his own term, but maybe not widespread.

What is rude jazz?