What algorithm should I use to detect anomalies on time-series?

Question: Background
I'm working in Network Operations Center, we monitor computer systems and their performance. One of the key metrics to monitor is a number of visitors\customers currently connected to our servers. To make it visible we (Ops team) collect such metrics as time-series data and draw graphs. Graphite allows us to do it, it has a pretty rich API which I use to build alerting system to notify our team if sudden drops (mostly) and other changes occur. For now I've set a static threshold based on avg value but it doesn't work very well (there are a lot of false-positives) due to different load during the day and week (seasonality factor).
It looks something like this:

The actual data (an example for one metric, 15 min time range; the first number is a number of users, the second - time stamp ):
[{"target": "metric_name", "datapoints": [[175562.0, 1431803460], [176125.0, 1431803520], [176125.0, 1431803580], [175710.0, 1431803640], [175710.0, 1431803700], [175733.0, 1431803760], [175733.0, 1431803820], [175839.0, 1431803880], [175839.0, 1431803940], [175245.0, 1431804000], [175217.0, 1431804060], [175629.0, 1431804120], [175104.0, 1431804180], [175104.0, 1431804240], [175505.0, 1431804300]]}]

What I'm trying to accomplish
I've created a Python script which receives recent datapoints, compares them with historical average and alerts if there is a sudden change or drop.
Due to seasonality "static" threshold doesn't work well and script generates false-positives alerts. I want to improve an alerting algorithm to be more precise and make it work without constant tuning the alerting threshold.
What advise I need and things I discovered
By googling I figured that I'm looking for machine learning algorithms for anomaly detection (unsupervised ones). Further investigation showed that there are tons of them and it's very difficult to understand which one is applicable in my case.
Due to my limited math knowledge I can't read sophisticated scholar papers and I'm looking for something simple to a beginner in the field.
I like Python and familiar with R a bit, thus I'll be happy to see examples for these languages.
Please recommend a good book or article which will help me to solve my problem.
Thank you for your time and excuse me for such long description
Useful links
Similar questions:

Time series and anomaly detection
Time Series Anomaly Detection with Python
Time series anomalies
Algorithms for Time Series Anomaly Detection
Application of wavelets to time-series-based anomaly detection algorithms
Which algorithm should I use?

External resources:

Machine Learning - Anomaly Detection Playlist on YouTube
Detection Algorithms for Biosurveillance: A tutorial
Comment: Guys from Twitter wrote very interesting article on this topic. Check it out: https://blog.twitter.com/2015/introducing-practical-and-robust-anomaly-detection-in-a-time-series
Comment: Did you take a look at one of the simplest algorithms like CUSUM?
Comment: That's a great question, @m-a-ge. I have a similar scenario. My approach was to setup alerts by building continual periodic forecasts using the `auto.arima` function from R's excellent `forecast` package (see http://www.jstatsoft.org/v27/i03/paper). You can tune the confidence levels by adjusting the `level` parameter, e.g. `data.model <- auto.arima(data.zoo, ic = c("bic")); data.prediction.warningLimits <- forecast(data.model, h=1, level=0.99)`.
Comment: @xeon, not yet. I'm new to the subject and need some time to digest everything.

Thanks for bringing this up, it's a good starting point, I can implement it right now
Comment: Hey @IlyaKhadykin Hope you are doing fine! did you ever got any solution for this problem. I am doing something exactly same in which every minute we are having certain users and we also get lots of false positive. As of now we are calculating score for every 5 minute of interval data and matching it with historical pattern.    IF YOU GOT ANY PARTICULAR ALGORITHM WORKING, CAN YOU PLEASE SHARE HOW YOU DID IT. Thanks in advance!
Comment: Also how do you detect a flatline pattern with some issues. Like when the users are showing a normal pattern. or we can say churn is stationary.
Answer: I think the key is "unexpected" qualifier in your graph. In order to detect the unexpected you need to have an idea of what's expected.
I would start with a simple time series model such as AR(p) or ARMA(p,q). Fit it to data, add seasonality as appropriate. For instance, your SAR(1)(24) model could be: $y_{t}=c+\phi y_{t-1}+\Phi_{24}y_{t-24}+\Phi_{25}y_{t-25}+\varepsilon_t$, where $t$ is time in hours. So, you'd be predicting the graph for the next hour. Whenever the prediction error $e_t=y_t-\hat y_t$ is "too big" you throw an alert.
When you estimate the model you'll get the variance $\sigma_\varepsilon$ of the error $\varepsilon_t$. Depending on your distributional assumptions, such as normal, you can set the threshold based on the probability, such as $|e_t|<3\sigma_\varepsilon$ for 99.7% or one-sided $e_t>3\sigma_\varepsilon$.
The number of visitors is probably quite persistent, but super seasonal. It might work better to try seasonal dummies instead of the multiplicative seasonality, then you'd try ARMAX where X stands for exogenous variables, which could be anything like holiday dummy, hour dummies, weekend dummies etc.
Comment: This approach assumes a specific ARIMA model which will have biased parameters based on the anomalies that have been implicitely assumed to be non-existent. A  more general approach would be to ALSO identify the anomalies first and then an optimal ARIMA model leading to inline tests of significance. Additionaly anomalies can be level shifts, seasonal pulses and local time trends which require a more general solution than proposed here. See  http://www.unc.edu/~jbhill/tsay.pdf  for a comprehensive procedure. You can also Google "Automatic Intervention Detection" for more information.
Comment: @m-a-ge, one more thing: you may want to use overlapping intervals. Let's say you collect data every minute, but for modeling you can pick a step in 10 minutes. It creates some issues for estimaton (due to autocorrelation), but the resulting model most likely will be more robust.
Comment: @IrishStat I suggested ARIMAX with dummies for events. OP can account for known events such as web site crashes with dummies. This will decrease the error variance, and there will be more alerts. There's no reason to build the complicated model, because it's simply impossible to account for everything when it comes to web site traffic. The simple models will work best.
Comment: @Aksakal  Models should be as simple as necessary BUT not too simple.
Answer: On the Netflix tech blog there is an article on their Robust Anomaly Detection tool (RAD).
http://techblog.netflix.com/2015/02/rad-outlier-detection-on-big-data.html
It deals with seasonality and very high volume datasets so it may fit your requirements.
The code is open source Java and Apache Pig https://github.com/Netflix/Surus/blob/master/resources/examples/pig/rad.pig
The underlying algorithm is based on robust PCA - see original paper here: http://statweb.stanford.edu/~candes/papers/RobustPCA.pdf
Answer: Most outlier detection algorithms in open source package are for business time series data with low frequency, daily/weekly/monthly frequency data. This data appears to be for a specialized area that is captured in minutes, so I'm not sure if open source outlier detction would be helpful. You could try to adapt this approaches to your data.
Below I outline some available packages approaches in open source R:

tsoutliers: Implements Chen and Liu's outlier detection algorithm within arima framework. see my earlier question on this site. Fantastic approach, but very slow not sure if it will be able to handle high frequency data like yours. It has the advamtage of detecting all types of outliers as mentioned in my earlier question/post.
Twitter's Anomaly detection: Uses Rosner's algorithm to detect anomalies based in time series. The algorithm decomposes timeseries and then detects anomalies. In my personal opinion, this is not efficient and accurate in detecting outlires in time series.
tsoutlier in forecast package: Similar to twitter's algorithm in terms of decomposing time series and then detecting outliers. Only will detect additive outliers or pulses.

There are commercial packages that have dedicated approaches to try and detect anomolies. Another classic approach is Tsay's time series outlier detection algorithm,  similar to Chen and Liu's approach it detects different types of outliers. I recently also stumbled on this commercial software solution called metafor which might be more suited for your data.
Hope this is helpful
Comment: Thanks, it gives me perspective on similar problems and approaches; special thanks for the links!
Comment: If anyone is looking for Metafor, we got acquired by Splunk. Our TSAD algorithms are included in recent versions of Splunk IT Service Intelligence ("ITSI").
Answer: What other answers didn't seems to mention is that your problem sounds like a changepoint detection. The idea of changapoint detection is that you are seeking for segments in your data that significantly differ in terms properties (e.g. mean, variance). This can be achieved my using maximum likelihood estimation, where for $m$ changepoints the likelihood function is
$$
L(m, \tau_{1:m}, \theta_{1:(m+1)}) = \prod_{i=1}^{m+1} p(y_{(\tau_{i-1} + 1):\tau_i}\mid \theta_i)
$$
where $y_1,\dots,y_n$ is your data, $1 < \tau_1 <\dots<\tau_m<n$ are the boundary points marking the changes, and probability distributions $p$ are parametrized by $\theta_i$ for each $i$-th segment. This can be easily generalized to vide variety of situations. A number of algorithms exist to find the parameters, including finding the unknown $m$. There is also software that is available to estimating such models, e.g. changepoint package for R. If you want to learn more, you can check the following publications and the references they provide:

Rebecca Killick and Idris A. Eckley. (2013) changepoint: An R Package
  for Changepoint Analysis. (online paper)
Eckley, I.A., Fearnhead, P. and Killick, R. (2011) Analysis of
  changepoint models. [in:] Bayesian Time Series Models, eds. D.
  Barber, A.T. Cemgil and S. Chiappa, Cambridge University Press.
Answer: Have you tried using Statistical Process Control rules (e.g. Western Electric http://en.wikipedia.org/wiki/Western_Electric_rules)?
I use them for time series data - often with a touch of intuition about the data - to assess whether the data is going somewhere I don't want it to go.  Like your example, these rules say if the delta / change is consistent over several data points, it flags that there may be an issue.
Also Statistical Process Control (SPC) can be good for working out if you are getting better or worse than before.
One issue with SPC is that much of it relies on a normal distribution which probably doesn't suit your data which can't go below zero.  Others better than I with SPC can suggest options here.  I like using it to flag an issue but, like all models, is best used with a grain of knowledge about the data itself (and source).
Answer: Given that the periodicity of the time series should be well understood a simple, but effective, algorithm based on differencing can be devised.
A simple one-step differencing will detect a sudden drop from a previous value
$$y_t'= y_t - y_{t-1}$$
but if the series has a strong periodic component you'd expect that drop to be considerable at a regular basis. In this case it would be better to compare any value to its counterpart at the same point in the previous cycle, that is, one period ago.
$$y_t'= y_t - y_{t-n} \quad \text{where } n=\text{length of period}$$
In the case of the posted question it would be natural to expect two significant periodic components, one the length of a day, the other the length of a week. But this isn't much of a complication, as the length of the longer period can be neatly divided by the length of the shorter.
If the sampling is done every hour, $n$ in the above equation should be set to $24*7 = 168$
If the drops are more of a proportional character a simple difference will easily fail to detect a sudden drop when activity is low. In such circumstances the algorithm can be modified to calculate ratios instead.
$$y_t^*= \frac{y_t}{y_{t-n}}$$
I did some tests in R using a simulated dataset. In it data is sampled 6 times a day and there are strong daily and weekly periods, along with other noise and fluctuations. Drops were added in at random places and of durations between 1 and 3.
To isolate the drops first ratios was calculated at distance 42, then a threshold set at 0.6, as only negative change of a certain size is of interest. Then a one-step difference was calculated, and a threshold set at -0.5. In the end one false positive appears to have slipped through (the one at the end of week 16). The graphs at the left and right show the same data, just in different ways.
Answer: Would it be more useful to think of the changes in the time series as a beginning of a new trend rather than an anomaly?  Taking the difference between adjacent points would help tell when the slope (derivative) is changing and might signal the beginning of a new trend in the date. Also taking the differences of the difference values (the second derivative ) might be of use. Doing a Google search on "times series beginning of trend) may give good suggestions for methods. In financial data a late of attention is paid to new trends  (do you buy or sell?) so there's papers on this topic.
A good intro to wavelet is "The world according to wavelets" by Hubbard I  believe is the author.
Answer: I was able to get some nice results for multiple-seasonality time series (daily, weekly) using two different algorithms:

Seasonal-trend decomposition using loess (or STL) to establish the midpoint series.
Nonlinear regression to establish thresholds around that midpoint, based on the relationship between the variance and the level.

STL does a time domain decomposition of your time series into a trend component, a single seasonal component, and a remainder. The seasonal component is your high frequency seasonality (e.g., daily), whereas the trend includes both the low frequency seasonality (e.g., weekly) and the trend proper. You can separate the two by simply running STL again on the trend. Anyway once you isolate the remainder series from the other components, you can perform your anomaly detection against that series.
I did a more detailed writeup here:
https://techblog.expedia.com/2016/07/28/applying-data-science-to-monitoring/
Answer: Inspired by David, have you tried to use FFT? It might be able to spot sudden drops because those are indicating your anomalies. The anomalies might appear in a narrow spectrum.  So you can easily capture them.


Gson is crashing with Failed to invoke protected java.text.NumberFormat() with no args error

Question: I want to use gson to store and retrieve Event objects. Gson seems to handle converting to json correctly but it's crashing on the gson.fromJson() method. I am not sure how to proceed.
public static void putValue(Context ctx, Event event){
        //get preferences
        SharedPreferences prefs = PreferenceManager
                .getDefaultSharedPreferences(ctx);
        Gson gson = new Gson();
        String json = gson.toJson(event);
        Log.i(TAG, "saving "+EVENT_KEY +Integer.toString(event.getId()));
        Editor editor = prefs.edit();
        editor.putString(EVENT_KEY + Integer.toString(event.getId()), json);
        editor.commit();
        //testing
        String json2 = prefs.getString(EVENT_KEY + Integer.toString(event.getId()), null);
        Event savedEvent = gson.fromJson(json2, Event.class);
        if(savedEvent !=null){
            Log.i(TAG, "yup");
        }
        else{
            Log.i(TAG, "nope");
        }
    }

Event class:
https://gist.github.com/Janek2004/9950294
Error:
04-03 02:58:54.597: E/AndroidRuntime(18793): FATAL EXCEPTION: main
04-03 02:58:54.597: E/AndroidRuntime(18793): Process: com.itenwired.itenconference, PID: 18793
04-03 02:58:54.597: E/AndroidRuntime(18793): java.lang.RuntimeException: Failed to invoke protected java.text.NumberFormat() with no args
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:107)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:162)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.Gson.fromJson(Gson.java:803)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.Gson.fromJson(Gson.java:768)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.Gson.fromJson(Gson.java:717)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.Gson.fromJson(Gson.java:689)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.itenwired.itenconference.model.MyEventManager.putValue(MyEventManager.java:37)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.itenwired.itenconference.adapter.AgendaAdapter$EventHolder$1.onClick(AgendaAdapter.java:88)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at android.view.View.performClick(View.java:4438)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at android.widget.CompoundButton.performClick(CompoundButton.java:100)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at android.view.View$PerformClick.run(View.java:18422)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at android.os.Handler.handleCallback(Handler.java:733)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at android.os.Handler.dispatchMessage(Handler.java:95)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at android.os.Looper.loop(Looper.java:136)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at android.app.ActivityThread.main(ActivityThread.java:5017)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at java.lang.reflect.Method.invokeNative(Native Method)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at java.lang.reflect.Method.invoke(Method.java:515)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:779)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:595)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at dalvik.system.NativeStart.main(Native Method)
04-03 02:58:54.597: E/AndroidRuntime(18793): Caused by: java.lang.InstantiationException: can't instantiate class java.text.NumberFormat
04-03 02:58:54.597: E/AndroidRuntime(18793):    at java.lang.reflect.Constructor.constructNative(Native Method)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
04-03 02:58:54.597: E/AndroidRuntime(18793):    at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:104)
04-03 02:58:54.597: E/AndroidRuntime(18793):    ... 23 more

Comment: I had the same problem. I solved removing this from the class variable:
  SimpleDateFormat formatter = new SimpleDateFormat("dd/MM/yy EEE", Locale.ITALY);

You should start to remove instance variable to find the one that cause the error.
Comment: Yes. I stopped using Gson.
Comment: Did you solve your problem?
Comment: @JanuszChudzynski What you have used for these
Answer: It's a definitely a problem with SimpleDateFormat, as I had the same problem and followed Accollativo's adivce, and that fixed the bug for me. Very weird.
I realized I didn't actually need the SimpleDateFormat as a variable of the class, I instead just put it in the method where needed, so GSON didn't have to store the SimpleDateFormat object. Worked for me!
Answer: As Accollativo and storm143 said, this is a problem with
SimpleDateFormat. If you want to keep SimpleDateFormat as an instance variable, you can fix this by using an ExclusionStrategy.
public class GsonDeserializeExclusion implements ExclusionStrategy {

    @Override
    public boolean shouldSkipField(FieldAttributes f) {
        return f.getDeclaredClass() == SimpleDateFormat.class;
    }

    @Override
    public boolean shouldSkipClass(Class<?> clazz) {
        return false;
    }

}

You can then create a Gson instance with this exclusion using the GsonBuilder.
Gson gson = new GsonBuilder()
        .addDeserializationExclusionStrategy(new GsonDeserializeExclusion())
        .create();

Comment: I am getting the same error but it was giving on third party library bean and I can't check where it is coming. Please help me how can I detect and skip that field
Answer: Another fix for this is to add the transient keyword to the NumberFormatter so gson doesn't try and convert it into the json string. Which is fine since you don't need to save any values just make a new one when the class gets recreated.
For example
private transient NumberFormat currencyFormatter = NumberFormat.getCurrencyInstance();


Config not visible on Parse Server Dashboard?

Question: So I've got a successful Parse-Server up and running on Heroku, and a functioning Parse-Server-Dashboard as well.
My issue is that through Parse-Server-Dashboard I'm not able to see the Config section. I've been searching endlessly, but I haven't been able to find any resources on this issue.
I've set my Config Var on Heroku PARSE_EXPERIMENTAL_CONFIG_ENABLED = 1, which allows me to get the config client-side just fine, but I still can't see it in dashboard.
Dashboard is V 1.0.15, and Parse-Sever is V 2.1.6
Here's what I'm seeing:

Here's (something similar to) what I want to see:

Cheers,
Comment: When you run it on your local machine you do see the config ?
Comment: How does your server's config look like?
Comment: @RanHassid, nope, it's never showed up for me.
Comment: Hi @AndreyGordeev, are you asking about my index.js file, or the actual global config that I'm trying to fetch/edit.
Comment: Can you try to access your config by adding  /config after you application name in parse-dashboard link
Answer: update parse-dashboard to the latest version as well as parse-server (you're running 2.1.6 and the current version is 2.2.18) it should be here...
At least I can see it.
Comment: Very well spotted! I was having problems updating up Parse-Server via get, so I just manually downloaded the zip of the Parse-Server-Example, and then slapped my package.json, cloud code, index.js, and jsconfig.json into the example folder, and pushed all of that to Heroku. It looks like it's working now. Cheers!
Comment: Awesome! Glad to help
Answer: You can view/edit your config parameters by using Parse RestAPI.
To view your config parameters:
curl -X GET \
  -H "X-Parse-Application-Id: APP_ID_KEY" \
  -H "X-Parse-REST-API-Key: REST_API_KEY" \
  PATH_TO_PARSE_API + "/config"

Example:
curl -X GET \
  -H "X-Parse-Application-Id: zirTLZa3h88Q6BBXj8WkkLxxxxxxxxxxxxxxxxxx" \
  -H "X-Parse-REST-API-Key: ydcXhehk9wkz01bDBgpteVxxxxxxxxxxxxxxxxxx" \
  https://api.parse.com/1/config

To edit your config parameters:
curl -X PUT \
  -H "X-Parse-Application-Id: zirTLZa3h88Q6BBXj8WkkLxxxxxxxxxxxxxxxxxx" \
  -H "X-Parse-Master-Key: OXuUPDbmAWOIcYZdzUXR0prxxxxxxxxxxxxxxxxxx" \
  -d "{\"params\":{\"Test\":\"222\"}}" \
  https://api.parse.com/1/config

NOTE:

To read the config you must to set the REST-API-KEY
To edit you must set your MASTER_KEY

Good Luck


Exchange 2013 - Edit Mailbox in ECP, Page Blank

Question: We have Exchange 2013 on Server 2012, and when logged into the ECP if I double click on a user mailbox the EditMailbox.aspx page pops up, but all the fields are blank and the links on the left hand side don't work - see attached image.
This previously worked and the only things which I know to have changed are Windows Updates applied last week.  We did install a new public certificate, but that was a few months back and I am 90% sure it has worked since then.
I've have checked the back end certificate in IIS as per https://support.microsoft.com/en-us/help/2971270/blank-page-after-login-exchange-eac-owa-ecp (even though it's not exactly the same error!).
I've checked the Application Event log and can't see anything in there.
The popup pages for groups, resources etc all work perfectly.
Any idea where I could look next?  Which log files might shed some light?
MailboxPopupPage
Answer: Do All mailboxes in EAC have this issue? If run "get-mailbox" in EMS, do all properties show correctly?
How many Exchange servers do you have? If many servers, you can test in different servers and check if there is the same issue.
Answer: Cause of Error: SSL Certificate Binding  Issue
Steps to Resolve Issue:

Open IIS manager.
Go to the sites under Exchange Server Environment and Select "Exchange Back End".
Open Bindings from right side "Actions" panel.
Select https under binding settings and click on Edit.
Select "MS Exchange" under SSL certificate and click on OK to apply settings.
Now, you should try to open Exchange EAC or OWA.


Show that a given transcendental function is uniquely solvable.

Question: I'm looking for a theorem to use to show that the following transcendental function is uniquely solvable. I don't care really about the solution.
$$ \rho = \mathrm{e}^{x-\rho t} $$
How might I show this function is uniquely solvable for $\rho$?
Answer: The equation is equivalent to $\rho t e^{\rho t} = te^x$.
Let $y=\rho t$ and $a= te^x$. Then $y e^y = a$.
The equation $y e^y = a$ has a solution iff $a \ge -1/e$. The solution is unique iff $a \ge 0$.
All this follows from plotting a graph. Uniqueness also follows from taking the derivative and seeing that it is positive for $y \ge 0$ when $y e^y \ge 0$.
Answer: Your proposition is true only if $t$ is non-negative. If $t=0$, this is trivial to see. If $t \gt 0$, transform the equation to
$$\rho e^{\rho t} = e^x$$
The right hand side is positive, so the left hand side must be positive as well, hence $\rho$ is positive. Now, the left hand side is a product of 2 positive values $\rho$ and $e^{\rho t}$, both of which are strictly increasing functions of $\rho$ for positive $\rho$ (and $t$). So the left hand side is a strictly increasing function of $\rho$ and thus the equation has at most one solution. Since the left hand side becomes $0$ for $\rho=0$ and goes to infinity for $\rho \to \infty $, it has exactly one solution.
However, the proposition is not true for negative $t$. The function $f(\rho)=\rho e^{\rho t}$ has a maximum at $\rho=-\frac{1}{t}$, so the above equation will have 2 possible solutions for "small enough $x$".


Bootstrap Align Image with text

Question: I am trying to align an image on the left side with text using boostrap, and when page is viewed on mobile devices the images becomes centred on top of the text !

About Me

    </class>

Lots of text here...With the four tiers of grids available you're bound to run into issues where, at certain breakpoints, your columns don't clear quite right as one is taller than the other. To fix that, use a combination of a .clearfix and o

I have also tried .col-md-3 .col-md-pull-9 along with .col-md-9 .col-md-push-3 , however I still was not able to achieve desired results, similar to this design
Comment: What are those ".span 4" and "span 8" classnames doing in there? And there is no tag
Comment: @ravb79, my bad it was old code I tried. Updated
Answer: You have two choices, either correct your markup so that it uses correct elements and utilizes the Bootstrap grid system:

@import url('http://getbootstrap.com/dist/css/bootstrap.css');

About Me

Lots of text here...With the four tiers of grids available you're bound to run into issues where, at certain breakpoints, your columns don't clear quite right as one is taller than the other. To fix that, use a combination of a .clearfix and o

Or, if you wish the text to closely wrap the image, change your markup to:

@import url('http://getbootstrap.com/dist/css/bootstrap.css');

About Me

Lots of text here...With the four tiers of grids available you're bound to run into issues where, at certain breakpoints, your columns don't clear quite right as one is taller than the other. To fix that, use a combination of a .clearfix and o

Answer: use the grid-system of boostrap , more information here
for example

here img

here text

in this way when the page will shrink the second div(the text) will be found under the first(the image)
Answer: Try using .pull-left to left align image along with text.
Ex:

At the time all text elements goes here.At the time all text elements goes here. At the time all text elements goes here. to uncovering the truth .

Answer:

About me

Lots of text here...

Answer: I think this is helpful for you

About Me

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,

Answer: i am a new bee ;p . And i faced the same problem. And the solution is BS Media objects. please see the code..

Beside Image

Answer:

About Me

Lots of text here...With the four tiers of grids available you're bound to run into issues where, at certain breakpoints, your columns don't clear quite right as one is taller than the other. To fix that, use a combination of a .clearfix and o

Bootstrap Align Image with text


anova like object for betaregression

Question: I'm wondering, is there any anova-like object in R for betaregression? I know it is not implemented in betareg package but I don't see why is that.
If there is no such an object, is there any easy was how to perform it?
Any help would be appreciated.
EDIT:
What I actually want is to determine significance of factors. With lmtest package it would be something like that:
library("betareg")
data("ReadingSkills", package = "betareg")

m1 <- betareg(accuracy ~ dyslexia, data = ReadingSkills)
m2 <- betareg(accuracy ~ iq, data = ReadingSkills)
m3 <- betareg(accuracy ~ iq + dyslexia, data = ReadingSkills)
library("lmtest")
lrtest(m2, m3)
lrtest(m1, m3)

But with a large number of covariates in the model this approach seems to be a little bit uncomfortable
Answer: Similar to https://stats.stackexchange.com/questions/297465/deviance-in-hurdle-model/299857#299857 beta regression models are not standard GLMs, especially if they have regressors in the precision submodel. Therefore, an analysis of deviance does not work in the same way as in GLMs. (Consequently, the proposed Anova() solution from @Adela is incorrect.)
Instead one can use the generic waldtest() and lrtest() functions from package lmtest to carry out Wald or likelihood ratio tests for nested model comparisons. This is also illustrated in vignette("betareg", package = "betareg") and a simple example is also included below:
library("betareg")
data("ReadingSkills", package = "betareg")
m0 <- betareg(accuracy ~ 1 | 1, data = ReadingSkills)
m1 <- betareg(accuracy ~ iq | 1, data = ReadingSkills)
m2 <- betareg(accuracy ~ iq + dyslexia | 1, data = ReadingSkills)
m3 <- betareg(accuracy ~ iq + dyslexia | iq + dyslexia, data = ReadingSkills)
library("lmtest")
lrtest(m0, m1, m2, m3)
## Likelihood ratio test
##
## Model 1: accuracy ~ 1 | 1
## Model 2: accuracy ~ iq | 1
## Model 3: accuracy ~ iq + dyslexia | 1
## Model 4: accuracy ~ iq + dyslexia | iq + dyslexia
##   #Df LogLik Df  Chisq Pr(>Chisq)
## 1   2 26.421
## 2   3 34.586  1 16.330  5.322e-05 ***
## 3   4 49.951  1 30.730  2.966e-08 ***
## 4   6 61.257  2 22.612  1.230e-05 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Comment: Thank you for the new point of view on this topic. So I guess there is no way how to "automatically" determine significance of covariates in betaregression models..
Comment: I wouldn't know what exactly you _conceptually_ want to do to automatically determine significance of all covariates. If you do, it shouldn't be hard _computationally_. In other words: I don't think this is a software issue.
Answer: Well, I've found the answer - it is possible with Anova() function from car package.

anova like object for betaregression


Song Cycle Texts, all at the beginning or placed individually before each song?

Question: Following the recommendation that the text for a song be provided separately as part of the front matter of a vocal composition, what is the best way to handle the texts for a song cycle if each text is an individual poem? Should all the texts be placed at the beginning of the song cycle, or should each text be placed before the song in which it is set?
Answer: Place all the poems together in the front matter.
This lets someone who's investigating the text independently of the music
do so for all the poems together, instead of having to leaf back and forth.
Placing each poem before its song would be slightly more convenient for someone who's interested in only one song of the cycle.
Comment: Excellent insight.


Approximation of $\ln(x+1)$ with $\Psi$ function

Question: I found the following approximation for the function
$$f=\ln(x+1)$$
$$f\simeq\Psi\left(x+\dfrac{3}{2}\right)-2+\gamma+\ln(2)$$
where $\Psi(x)$ is the 'Digamma' function:
$\Psi(x)=\dfrac{\dfrac{d}{dx}\Gamma(x)}{\Gamma(x)}$
and $\gamma$ is the Euler - Mascheroni constant.
The following inequality holds:
$$\left|\ln(x+1)-\Psi\left(x+\dfrac{3}{2}\right)-2+\gamma+\ln(2)\right|\lt 0.036$$
Is this a known result? Thanks
Answer: I'm not sure why there is a $(-2+\gamma+\ln 2)$, because $\left|\ln(x+1) - \Psi(x+\frac32)\right| < 0.0365$ itself is already satisfied.
The approximation comes from the series expansion of $\exp\left(\Psi(x+\frac12)\right)$, which states, for $x > 1$,
\begin{align}
\Psi\left(x + \frac12\right) = \ln\left( x + \frac{1}{4!\cdot x} - \frac{37}{8\cdot 6!\cdot x^3} + o\left(\frac1{x^5}\right) \right)
\end{align}
and thus $\Psi(x+\frac32)$ is approximately $\ln(x+1)$. The largest difference happens at $x=0$ which is $\Psi(\frac32) = 2 - \ln 4 - \gamma \approx 0.03648997$.
Answer: I doubt your approximation. First I guess you missed a factor $2$ for $\ln 2$, the approximation should be
$$f(x) =\ln(x+1)-\Psi(x+3/2)-2+\gamma+2\ln 2 \cdot$$
If you omit the $2$ you have
$$f(0)= 0 -(2-\gamma-2\ln2)  -2+\gamma+ \ln 2 = -4 + 2\gamma + 3\ln 2 \approx -0.766$$
while with the factor $2$ you have
$$f(0)= 0 -(2-\gamma-2\ln2)  -2+\gamma+ 2\ln 2 = -4 + 2\gamma + 4\ln 2 \approx -0.0723$$
which is much smaller but still violates your given bound.
PS: A Taylor series at $x=0$ computed with Maple is
$$\ln(x+1)-\Psi(x+\tfrac{3}{2}) = (-2+\gamma+2\ln2)+(5-\tfrac{1}{2}\pi^2)x+(-\tfrac{17}{2}+7\zeta(3))x^2
+O(x^3)$$
Comment: In fact, I omissed to write that the inequality holds for $x \gt 1$
Comment: But in any case you have to add the factor $2$, otherwise you have $f(2)\approx -0.734$! But even with the factor you have $f(2)\approx-0.041$


Intersection of open and compact

Question: In a metric space let $A$ be compact and $B$ - open, dense everywhere. Does it mean that
$$
\overline{A} = \overline{A\cap B}
$$
or there are counterexamples?
Comment: Note that compact sets in Hausdorff spaces are closed, so the left-hand side of your equation can be replaced by A.
Comment: Metric spaces are Hausdorff, so compact sets are closed. Therefore, $\overline{A}=A$. I know what "dense" and what "nowhere dense" mean. What does "dense everywhere" mean?
Comment: I think you might need the interior of $A$ to be non-empty.
Comment: Even with non-empty interior the statement is false - take $A = [0,1] \cup \{2\}$ and $B = \mathbb{R} - \{2\}$.  You could ask that $A$ is equal to the closure of its interior, but then the statement is trivial.
Comment: Thanks for the post; it reminds me to not jump to the conclusion that $ \overline{A\cap B}$=$\overline{A}\cap\overline{B}$, and examples like A$=(0,1)$ and B=$(1,2)$
Answer: What about $A=\{0\}$ and $B=\mathbb R\setminus\{0\}$?
I assume you use everywhere dense as a synonym to dense - like here.
Perhaps this result is worth mentioning in this context: If $A$ is dense in $X$, then for every open $U\subset X$ we have $\overline U=\overline{U\cap A}$.

Intersection of open and compact


how to get a node field value in a html.tpl.php template file?

Question: I have a field in a node type called body_styles.
I need to modify html.tpl.php to get the value of body_styles to add it to the classes of the body tag.
I tried with dpm($node->content['body_styles']['#value']); but I get nothing. I think it is because $node is not available in html.tpl.php.
how do I get a node field value in a html.tpl.php template file?
@svetlio: almost done:
The class is correctly added to $variables['classes_array'] but if I inspect the body (see my capture below the code) tag is not there
    function omega_subtheme_2_preprocess_html(&$variables, $hook) {
    if (arg(0) == 'node' && is_numeric(arg(1))) {
        global $language;
        $node = entity_load_unchanged('node', arg(1));

        dpm($node -> field_body_class['und'][0]['value']);

        if (isset($node -> field_body_class['und'][0]['value']) && $node -> field_body_class['und'][0]['value']){
            $variables['classes_array'][] = $node -> field_body_class['und'][0]['value'];
            dpm($variables['classes_array']);
        }
  }

Comment: Have you tried clearing your cache?
Comment: I clear all cache several times, not working...
Answer: Another way - without edit template and adding code in template.php
function THEME_preprocess_html(&$variables, $hook) {
    if (arg(0) == 'node' && is_numeric(arg(1))) {
      $node = entity_load_unchanged('node', arg(1));

      if (isset($node -> field_FIELD_NAME['und'][0]['value']) && $node -> field_FIELD_NAME['und'][0]['value']) {
        $variables['classes_array'][] = $node -> field_FIELD_NAME['und'][0]['value'];
      }
    }
  }

replace THEME with your theme_name and FIELD_NAME with your field_name, clear the cache
edit: I change "$node = entity_load('node', arg(1));" to $node = "entity_load_unchanged('node', arg(1));"
Comment: I read the doc for preprocess function with omega. You are right, it is different. I can't get it working though, but this out the scope of this question
Comment: I use friendly url and it is a multilingual site then the urls has appended the Path prefix (en, es, fr). then the function get the arguments in a wrong way: arg(1) and arg(0)
Comment: add "global $language;" before "$node = entity_load_unchanged('node', arg(1));" and replace all 'und' with "$language -> language" without the quotes
Comment: almost there. I need to leave 'und' to get it to work. The class is correctly added to $variables['classes_array'] but if I inspect the body tag is not there. Please see the function at the bottom of my question.
Comment: must to clear cache after each change in template.php before refreshing page, maybe twice
Comment: I clear all cache several times, not working...
Comment: omega is more specific theme, I don't know it well, but in her readme.txt in preprocess folder have "Any custom preprocess functionality can (rather than directly in template.php) be placed
in this preprocess folder in a file named as such:

TEMPLATE_preprocess_html() = preprocess-html.inc", so try to move this function following the theme instructions
Comment: A couple of notes of caution: as it stands, this preprocess function will require a node to be loaded twice on every full node view; also I believe it's better to use [`field_get_items()`](http://api.drupal.org/api/drupal/modules!field!field.module/function/field_get_items/7) rather than accessing the field directly.


If $\gcd(a, b) = 1$ then $\gcd(ab, a+b) = 1$?

Question: In a mathematical demonstration, i saw:
If $\gcd(a, b) = 1$ Then $\gcd(ab, a+b) = 1$
I could not found a counter example, but i could not found a way to prove it too either.
Could you help me on this one ?
Answer: Hint:
If a prime $p \mid ab$, then $p$ divides either $a$ or $b$.  WLOG let $p \mid a$.  If $p \mid (a+b)$ also, then $p \mid b$.
Answer: Hint: It is true. Indeed, if $\gcd(ab,a+b) > 1$, then there is a prime $p$ divding it, in particular, there is a prime $p$ dividing both $ab$ and $a+b$, and remember Euclid's Lemma...
Answer: First prove

$\gcd(mn, k)=1$ if and only if $\gcd(m,k)=1$ and $\gcd(n,k)=1$.
If $\gcd(m,k)=1$ then $\gcd(m,m+k)=1$.

The desired result follows from these like so: From $\gcd(a,b)=1$, we have $\gcd(a, a+b)=\gcd(b, a+b)=1$, implying
$\gcd(ab, a+b)=1$.
Answer: Suppose that we have that some prime $p$ divides the gcd of ab and a+b. Then, we know that $p\mid a$ or $p\mid b$. WLOG suppose that p divides a. Then, it follows that a+b is divisible by by p, so b must also be divisible by p. But this means that $a$ and $b$ have a prime factor in common, contridicting the assumption that they were coprime.
Answer: If a prime $p$ divides $ab$, then it divides $a$ or $b$. WLOG it divides $a$. If that prime also divides $a+b$, then since it divides $a$ it must divide $b$ as well, contradicting $gcd(a,b)=1$.
Answer: We can take as the definition of "relatively prime" that there are $s,t$ with $as+bt=1$ (or we can show that this is equivalent to our favorite definition.) SO given that, can we find $S,T$ with $$abS+(a+b)T=1?$$ Well we have $$a(s-t)+(a+b)t=1$$ and $$b(t-s)+(a+b)s=1$$ So multiplying gives $$ab(s-t)(t-s)+(a+b)\left[\ a(s-t)s+tb(t-s)+(a+b)st \  \right]=1 .$$ That does the job, but expanding and simplifying gives the more satisfactory $$ab(s-t)(t-s)+(a+b)[\ as^2+bt^2\ ]=1 $$
Answer: We use the idea of the Euclidean algorithm. I'll arbitrarily decide that "b" looks bigger than "a", so I can reduce $ab$ by subtracting off $a$ times $a+b$:
$$ \gcd(ab, a + b) = \gcd(ab - a(a+b), a+b) = \gcd(a^2, a+b)$$
Hrm. Usually this works out for homework problems. But never fear, there is something related we can still do. We know that the $\gcd$ divides $ab$, $a+b$, and $a^2$. Similarly, we can see that it also has to divide $b^2$.
In particular, it has to divide
$$ \gcd(a^2, b^2) = 1 $$


Am I going to be handicapped for attempting to study theory with a monophonic instrument?

Question: The instrument that I intend to learn is the bass. The bass is what gets me excited to study theory in the first place. I've done a bit of reading on theory and I find it's fun to practice it on the bass.
However, I'm also kind of worried that I might miss out on something since the bass is mostly a monophonic instrument and I'm going to need to learn chords after all. Now I understand I can simply play the intervals on the bass one note at a time(arpeggio?)and still get an idea of what the chord sounds like. But the piano/keyboard allows you to pull those big extended and suspended chords on the fly and I find it's quite frightening. I also have tried to visualize theory on the piano and admire how it simplify theory so I get why it's the standard instrument for music.
Now I have nothing against the piano. I like it a lot as well. It's just that the instrument that have always seduced me is the bass and good keyboards don't come cheap so I gotta pick one or the other.
Hence I'm in a bit of dilemma at the moment. So what do you guys think? Am I really just going to make it harder for myself for attempting to study theory without a piano or will I be doing just fine except I'm gonna have to put more efforts in everything I will be doing?
Thanks in advance.
Comment: The electric bass is not strictly monophonic, you can play chords, though of course piano is much more suitable for this.
Comment: I know we've answered a similar question recently. (Duplication?) Your problem is not one of alternatives. It makes lot of sense to practice both instruments. The bass (your favorite one) and the piano additional.
Comment: watch some Adam Neely youtube videos, preferably all of them, best music youtube, basist and absolutely nuts for theory.
Comment: People who play actual monophonic instruments (woodwind, brass) don't seem to have any trouble understanding harmony.
Comment: Right, but most "serious" woodwind/brass/whatever players also have some rudimentary keyboard skills.

For that purpose a very cheap keyboard would probably be fine.
Answer: You've expressed an accurate sense of why there is a theory advantage with the piano, but since you also very clearly are drawn to the bass, a couple of comments:

Particularly if you play jazz or popular music, you will by necessity learn a huge amount about chords and scales and how they relate, because this is the essence of bass's harmonic role in a band. If anything, bass probably has an advantage over many other monophonic instruments, because its role is so fundamental (pun intended) to the harmony.

For the purpose of learning theory, you don't need to play keyboard beyond a few basics, like forming a chord or playing a scale (but even the one-finger approach will be okay). If you can, just pick up a cheap keyboard so you can play and see chords and scales. The sound, the touch, even the size don't really matter. For your purposes it will just be a visualization/tactile sensation tool.
Comment: I wanted to write my own answer, but your point 1. summarizes it well enough. Especially in the context of modern music, the bass is often a vital component of the harmony. It doesn't just add to the "rhythm guitar" chord our "outline" the progression, it definitely introduces a new quality there. So understanding the relations with _other_ instruments is extremely important. Just looking at a few Polyphia songs would give you an idea how much that synergy can be utilized - e.g. in [this nice analysis video](https://www.youtube.com/watch?v=tuAc8Tihuto&ab_channel=MichaelPalmisano).
Comment: An example of 1., a very common track structure in jazz is "theme, (chorus, theme), solos, theme".  And the bass sheet music will have notes for the theme part, and "chords and rhythm" for the solos. Your job as the bass player is to play notes in the chord, or notes that complement the chord, throughout, so that the solo can shine on top of it.  Doesn't have to be the same ones every time, and you can work with the soloist to make your ground bass "his style", and then "her style" for the next one.  And, of course, sometimes the band leader points to you!
Comment: @MaxD "[Fundamental bass](https://www.oxfordmusiconline.com/grovemusic/view/10.1093/gmo/9781561592630.001.0001/omo-9781561592630-e-0000010388)" is an early term for the idea of a chord root.
Comment: Anyone care to explain the pun in "fundamental"? In my defense, non-native-speaker.
Answer: There is nothing theory-related that you cannot learn on the bass. Also the bass is really far away from being a monophonic instrument!
However, like you have noticed already, piano makes visualizing things easier SOME TIMES (I find it MUCH easier to visualize diminished chords on a guitar instrument than on the piano for example, either on bass or guitar; I find it much easier to count the semitones on the fretboard rather than on the keyboard as another example)
My suggestion is: HAVE FUN. If you enjoy music, playing around with another instrument should only add to the fun, so experiment as much as possible.
Finally, I would recommend to always have some sort of keyboard around so that you can use it to visualize/understand things that you struggle on the bass.
Answer: There's a lot more to music theory than chords, which can (and are) played on bass. And extended chords, available easily on piano, won't be part of your curriculum for quite a while yet. And by that time, you'll be sufficiently knowledgable to find those extra notes, even if in arpeggio style, which won't hurt.
The bigger problem of bass versus piano is that on piano there is only ever one place to play one particular note. On bass (4 string) there are often three or four different strings/frets for an individual note, which can be confusing, especially for a beginner playing the instrument, let alone trying to understand theory! Sometimes, you may find it simpler to just use one string to find chord notes, for instance.
As has been said many times, a piano, or cheap keyboard (car boot sale, a tenner), would be a great help. That's why, in music school, whatever instrument one studies, the piano/keyboard is used alongside. It will also give you an insight into the treble clef - unknown territory for most bass players...
Answer: Theory studies (even advanced harmony, post tonal music and counterpoint) are topics apart from instruments or instrumental proficiency. You may not play any instrument at all, and still excels in music theory. But you'll need to take this serious as you'd take an instrument.
Of course keyboards will help visualizing harmony and sets, but it's not mandatory playing piano to learn well this. But you will need to improve your aural skills and solfege a lot; hear perfectly inside your head what's written it's the goal and ultimate perception level. If you study theory in a clever way - not the way it is usually systematized - you may achieve this faster. Partimento and solfege exercises may help a lot at this.
A example of how good one can be without any instrument: Paris Conservatory had annual contests on solfege, harmony and counterpoint. Participants of counterpoint contest were locked for 16 hours in a room only with sheet paper and a pen and must compose a 4-part fugue with more than a hundred bars, using a given subject (the same for all competitors) and his/her aural skills only - without access to anything else, including instruments.
If people 100-150 years ago could nail perfectly this, we can also.
Answer:
Am I really just going to make it harder for myself for attempting to study theory without a piano or will I be doing just fine except I'm gonna have to put more efforts in everything I will be doing?

Music theory is about the relationship between the notes, not how they're laid out.
For instance, a Major Third, Minor Third, Perfect Fourth, etc. are all defined by the number of semi-tones between them, not where they fall on the instrument. Scales are the same way. Knowing what they are and how they apply to your instrument specifically--whatever it is--will be more important than what that instrument actually is.
The only thing I could see being easier on a Piano is the fact that E and F are right next to each other (as well as C and B)--as opposed to having another note between them like F and G, for instance. On the Piano, because of the way the keys are laid out, it has a nice visual reminder of this. On the other hand, on the Bass it's a little harder. But that's really the only problem I could see a beginner running into.

The one thing I will tell you is that it will be harder to learn an instrument that you aren't motivated to play. Being inspired will help you learn, and even make you excited to learn. There's no use learning an instrument you don't want to play just because you think you'll get some theoretical benefit out of it. At the end of the day, a good teacher and/or a good curriculum can teach you all the theory you need to play the instrument properly. And that goes for both Bass and Piano, or really any instrument.
So, if you like the Bass better than Piano, then go for the Bass. You seem to be really inspired to play Bass, so do that.

And, I know you think you can only choose one. But think of it this way: you only have to choose one right now. There is nothing stopping you from coming back in a year, or five years, or even ten years, and picking up another instrument (such as the Piano). I myself have learned more than one instrument, and know several other people who have done so as well.
And, in the mean time, you can take time and learn and practice the instrument you chose and perfect it. That way, when you finally get a future opportunity to get another instrument, you have already mastered the first one.


What is the interplay between treatments for multiple testing and autocorrelation in hypothesis tests?

Question: I would like to apply a null hypothesis significance test that requires iid data (specifically the Pettitt test for change-point detection) to multiple time series in different locations. These time series typically exhibit significant autocorrelation to some lag. My understanding is that both autocorrelation and multiple testing can lead to inflation of the Type I error rate, and hence an over-rejection of the null hypothesis.
To deal with multiple testing, one can control the false discovery rate (FDR). For autocorrelated data, one might apply prewhitening methods to the time series, or perhaps use block bootstrapping.
My conceptual understanding of these kinds of treatments is that they effectively try to apply a stricter threshold of significance than one would use without accounting for multiple testing or spatial correlation (e.g. simply taking $p<0.05$ as indicating significance). Practically, when I have applied FDR to my data, this results in a much smaller p-value threshold for indicating significance, while block bootstrapping typically results in larger p-values than those generated using non-block bootstrapping.
I have difficulty in understanding whether both multiple testing and autocorrelation always need be accounted for when significance testing, or whether judicious treatment of either one of them would suffice. For example, as FDR controls the number of false rejections, does it implicitly account for the effects of autocorrelation (or any other effect that would lead to an inflated Type I error rate)?
From a practical (i.e. programming) point of view, it would be very beneficial for me to use FDR, but safely ignore autocorrelation.
My question is: Given I am applying hypothesis tests to autocorrelated data for numerous time series in space, should I account for both multiple testing and autocorrelation, and what is the logic behind this?
Note: I have some grasp of statistics, and often struggle more with the concepts than math, but I am really trying to learn more and become more statistically competent. Links to journal articles/books that can help me understand the problem would be much appreciated.
Answer: Short answer is no.
Take it in two steps. First, ensure that the tests are valid, irrespective of any multiple testing correction. In your case, that means accounting for autocorrelation. Then you can apply multiple testing corrections like Bonferroni and standard fdr to the results.
To take it a step further and improve power, there are modifications of Bonferroni and standard fdr that incorporate dependence structures.


Conversão de datas no banco de dados

Question: Como converter uma data que está no formato DDMMYYYY para YYYYMMDD?
Estou tendo problemas para enviar estes dados para diferentes bases SQL, tendo em vista que cada uma foi elaborada de um jeito e já estão alimentadas.
Comment: Poderia especificar o banco de dados?
Answer: Se o seu banco de dados for MySQL você pode usar uma função do mysql ao fazer o insert.
DATE_FORMAT(data, '%Y-%m-%d').

http://www.w3schools.com/sql/func_date_format.asp

Conversão de datas no banco de dados


How to read efficiently from an stdin pipe data that need seeking

Question: I'm looking for the best way to read data from an stdin pipe  in C programming.
Problem : I need to seek on this data, ie I need to read data from the start of the stream after reading some data at the end of this same stream.
Small use case : gunzip -c 4GbDataFile.gz | myprogram
Another one :

On local host : nc -l -p 1234 | myprogram
On remote host : gunzip -c 4GbDataFile.gz | nc -q 0 theotherhost 1234

I know that reading from fifo can be done only once. So, at the moment :

I slurp everything from stdin to memory and work from this allocated memory.

It's ugly, but it works. An evident issue is that if someone sends a huge (or a continuous) stream to my app, I'll end with a big allocated memory chunk or I'll run out of memory. (Think about an 8Gb file)
What I thought next :

I set a size limit (maybe user-defined) of that memory chunk. Once I've read this much data from stdin :

Either I stop here : "Errr. Out of memory, bazinga. Forget it." style.
Either I start dumping what I am reading to a file and work from this file once all data is read.

But then, what is the point? I can not find out the origin of the data that I am reading. If this is a local 8Gb file, I'll be dumping it to another 8Gb file on the same system.
So, my question is :

How do you read efficiently a lot of data from an stdinpipe  when you have to seek back and forth in it?

Thanks in advance for your answers.
Edit :
My program needs to read metadata somewhere (depending of the file format) in the given file, so that maybe at the end of the stream. Then it may read back other data at the start of the stream, then at another place etc. In short : it needs to have access to any bytes of the data.
An example would be to read data of an archive file without knowing the file format before starting to read from stdin: I need to check the archive metadata, find archive files names and offsets etc.
So I'll make a local copy of stdin content and work from it. Thanks everyone for your inputs ;)
Comment: ITYM `myprogram < 4GbDataFile`. Currently, `cat 4GbDataFile > myprogram` overwrites your program binary. And when you replace '>' with a pipe you have a useless use of cat.
Comment: Thanks, fixed the typo, added a more useful use of cat.
Answer: You need to get your requirements clear. If you need to seek() then obviously you can't take input from stdin. If you need to seek() then you should take input file name as argument.
Comment: Except that `stdin` is not a normal **file**, isn't it? I can't seek on or read from it as usual. That's why I'm currently mapping data read from stdin to memory (or to a file, but if the $user just cat a local file to my program, I'll be duplicating the file rather than working on the original one...). That works, what i'm looking for is an elegant way to fallback if the $user sends me **a lot of** data.
Comment: I mentioned that, at the moment, it works by slurping all to memory and doing the work here. I suppose it will works too when using a temporary file. I know that I can't seek on a `fifo`, I'm looking for an elegant (more than my slurping) workaround ;)
Comment: I supposed it will works to when using a temporary file. -- but why would you do that, when you are just duplicating a file content? What your program does?
Comment: My program needs to read metadata somewhere (depending of the file format) in the given file, so that maybe at the end of the stream. Then it may read back other data at the start of the stream, then at another place etc. In short, it needs to have access to any bytes of the data. An example would be to read data of an archive file _without_ knowing the file format : I need to check the archive metadata, find archive files names and offsets etc.
Comment: So, if your program works on __files__ then treat them as files, i.e. seek and read them from any location.
Comment: If you are clear about your requirements and **you need to read() from stdin**, then storing data to temp file is a better option - as you aren't sure about file size.
Comment: @Lenain: `stdin` might or might not be a normal file. The issue is that a *pipe* is not seekable (always true), not that `stdin` is not seekable (which might or might not be true depending on what type of file you inherited as `stdin`).
Comment: Ok, edited my question to reflect that I'm working with an `stdin` pipe. So the only "efficient" way would be to use a temporary file? And warn the user that they should not pipe huge local files to my program somewhere?
Answer: The data structure in your 4GbDataFile just doesn't lend itself to what you want to do. Think outside the box. Don't hammer your program into something it shouldn't even attempt. Try to fix the input format where it is generated so you don't need to seek back 4 GB.
In case you do like hammering: 4GB of in-core memory is pretty expensive. Instead, save the data read from stdin in a file, then open the file (or mmap it) and seek to your heart's content.
Comment: **You can't have your pie and eat it.** Seeking on pipes is impossible. If you insist on it, you'll have to turn the unseekable data into seekable (by saving a copy). When mmapping, don't map the whole data, just mmap the blocks where you want to seek to, calculated from the seek offset.
Comment: I can not fix the input format as it can be anything (I edited my question and added an example of program work at the end). Does not mmaping a 4GB file result in a 4GB in-core memory chunk? Saving the data read into a file and seeking in it maybe more appropriate, but if it is a local file it's just duplicating the original file and work on the copy rather than the original...
Comment: Ok thanks, I'll make a copy and warn the user about not piping huge local files and prefer to give them as fd to avoid duplicating content.
Answer: I think you should read the infamous Useless Use of Cat Award.
TL;DR: change cat 4gbfile | yourprogram to yourprogram < 4gbfile.
If you really insist on having it work with data from a pipe, you'll have to store it in a temporary file at startup then replace file descriptor 0 with a copy of the fd for the temp file, using dup2.
Comment: That is what I want : work with data from a pipe. (Edited my question to remove "Useless Use of Cat", it was just meant to be an example by the way... otherwise I would have used what you and @Jens recommended)


Encrypting db password stored in settings.php

Question: The db password in settings.php is stored as plain text.
Is there any way to encrypt that password?
My client says its a security risk even though the file permissions is as should.
Comment: As a side note, this issue can be mitigated by having Drupal connect as a database user who (1) can only connect to the database catalog for the site, (2) can only connect from localhost or from the webserver when the webserver and database servers are different machines (and preferably via an RFC 1918 address), and (3) does not have `GRANT` permissions on the catalog.
Answer: This is not a Drupal question specifically. Depending on the OWASP security standards you're following keeping the password in plain in settings.php could be considered a risk (in a normal, usual Drupal environment it won't be the case)
This thread on Information Security Stack Exchange discusses the topic, but in my opinion is a little paranoid and most of the options will slow down your site.
Comment: Upvoted for a link to an existing and more thorough discussion.
Answer: Security is always a matter of compromises. The question is always what threats do you want to protect against and what level of hindrance are you taking.
For example, you could create a small script that asks for the DB password, stores it in shared memory (with APC/APCu) and then change Drupal settings.php to read the DB password from there. Of course, any Apache restart now requires providing the DB password. But, you don't have your password in any file. Compromise, see?
If someone manages to run arbitrary PHP code, for sure they can gain the password but then it doesn't matter much anyways, right?
Comment: I thought about putting the password as an apache variable, but as you said, every time the server will restart the password will be lost... And in our servers architecture, its not quite possible also(we have multiple servers with 1 nfs drupal installation).
Answer: It is possible to do this. I have implemented this on my website and also written a post about this here:
http://techrappers.com/post/3508/how-secure-settingsphp-file-drupal
Above articles explain step by step procedure of implementing an encrypted password inside settings.php file with a PHP script and AES-256-CBC encryption method.
Answer: Create a new file somewhere on the webserver (mypsswrdfile.php) to which only web admins should have rwx rights to.
In it have them create a variable that holds the password.
$pass = "dbpass";
Include that file in your settings.php and use the variable for the password key.
"password" => $pass,
Comment: Your solution still doesn't encrypt the password... There is no difference between your solution with the current state because the file has the needed permissions...
Comment: Good luck with that. Please provide us the solution if you find it because in eager to know myself. Thanks


Normal or lognormal? After comparing the fit of those two models by calculating a likelihood ratio (LR), how to pool LRs from several data sets?

Question: Using the method of section 6.7.2 of Burnham and Anderson, Model selection and multimodel inference: a practical information-theoretic approach, 2nd edition, one can compute the likelihood ratio comparing the likelihood that a particular data set (numerical variable; all values are positive) would occur with sampling from a normal distribution compared to sampling from a lognormal distribution. Let's say that the likelihood ratio (LR) is 3.1.
Now repeat with a second data set (a different condition in the same experiment), and that LR is 2.5.
Assume that both data sets are measurements of the same variable in similar conditions, so both come from the same distribution family. Is it appropriate to multiply those two LR values (3.1 * 2.5 = 7.75) and conclude that all the data would be 7.75 times more likely to occur with sampling from a normal distribution than with sampling from a lognormal distribution?
It sure "feels right" to multiply those likelihood ratios, but is it mathematically correct?
...
In a comment, @Sextus Empiricus asked me to clarify the context and so clarify my question.
I don't have particular data at hand, but am asking a general question. You are about to start a series of experiments. The variable you measure is always positive, and you aren't sure if the underlying distribution (population) is normal or lognormal. So you look at existing data of the same variable.
Using normality and lognormality tests doesn't quite answer the question, and would lead to lots of questions (What value for alpha?? Which normality test? How to cope with data sets that pass both normality and lognormality tests or fail both?...). Computing likelihood ratios for each data set seems (to me) to directly answer the question.  For each data set, compute the ratio of the likelihood that those data would occur with sampling from a normal distribution divided by the likelihood that the data would emerge from sampling from a lognormal distribution.
You compute that likelihood ratio for each data set of each prior experiment that measures the variable of interest (in similar conditions). To directly answer the question in the comment: this would mean fitting the mean and SD when fitting the normal distribution, and the Geometric mean and geometric SD when fitting the lognormal distribution. You assume that the underlying distribution (population) is always normal or always lognormal for every data set you analyze. But the mean and SD (and GeoMean and GeoSD) will differ among data sets.
Now you have a set of likelihood ratios. How can one combine them to come up with an overall likelihood ratio for all data sets answering the question: For all the data sets, what is the ratio of the likelihood that all the data sets were sampled from a normal distribution divided by the likelihood that all the data sets were sampled from a lognormal distribution.  Assuming that that overall LR is far from 1.0, it can guide analysis of data to be collected in the future -- assume normality or assume lognormality?
Comment: Do you propose to fit the same models separately on the two datasets? In that case, wouldn't you expect that the MLEs of the model parameters are (at least somewhat) different as well? So the first likelihood ratio would compare models $M_A(\hat{\theta}_1)$ vs $M_B(\hat{\phi}_1)$ and the second likelihood ratio would compare $M_A(\hat{\theta}_2)$ vs $M_B(\hat{\phi}_2)$.
Comment: @dipetkov. You understand right. But if both data sets (or more generally, all data sets) are of the same variable in not-too-different conditions, you'd expect that one model is correct for both (all).

I'm trying to figure out how to decide when to assume sampling from normal vs. lognormal. You'd need to look back at old (or preliminary) data, and see which model fits better, and then use that model for future similar studies.
Comment: Would stock returns be a good example? Eg each data set would have a year of data, and each datum would be the S&P 500 price divided by the price on the previous day.
Comment: @MattF. Might be a good example. But.. A better example would be weight (or glucose concentration in the blood) of a bunch of mice. In your example,  the values follow some distribution, but you collect them sequentially which adds another variable.
Comment: @MattF.  Let's make it simple. Your plan is to compare weights in mice given one of two diets. One measurement per mouse after 2 months on the diet. To plan your analysis you need to know if weight is closer to normal or lognormal. So you look back at some prior data from several prior studies. What calculations do you do to pool the data in order decide (in general) whether mouse weight is closer to normal or lognormal?
Comment: OK, it seems worth putting that example in the main part of the post. (Even then, the best model of starting weights might depend on the supplier of lab mice, so you might choose the model that best fits the current population, regardless of any meta-analysis of prior studies.)
Answer: Case of simple hypotheses
Yes the likelihood for independent events is multiplicative.
$$\mathcal{L}(\theta ; x_1, x_2)= \mathcal{L}(\theta ; x_1) \cdot \mathcal{L}(\theta ; x_2)$$
And as a consequence the likelihood ratio is multiplicative as well.
$$\frac{\mathcal{L}(\theta_1 ; x_1, x_2)}{\mathcal{L}(\theta_2 ; x_1, x_2)}= \frac{\mathcal{L}(\theta_1 ; x_1)}{\mathcal{L}(\theta_2 ; x_1)} \cdot \frac{\mathcal{L}(\theta_1 ; x_2)}{\mathcal{L}(\theta_2 ; x_2)}$$

In the case of log likelihood and log of the likelihood ratio, the multiplicative property changes into an additive property. $\log(ab) = \log(a) + \log(b)$

Be aware that the multiplication is only true for independent events. Multiple measurements are not always independent. For instance, the second measurement can be made as a consequence of the first measurements, and this leads to the stopping problem.
Case of a composite alternative hypothesis 1
It might be that you are using the likelihood ratio test for testing a null hypothesis $H_0: \theta = \theta_0$ against an alternative composite hypothesis $H_a: \theta \neq \theta_0$ and you are using the likelihood ratio
$$\lambda_x = \frac{\mathcal{L}(\theta_0;x)}{\mathcal{L}(\hat\theta_{ML};x)}$$
Here you can again multiply the likelihood ratio values (or sum the log values), but you need to consider that you computed two times parameters and the distribution of the ratio will be different.
Typically the distribution of the logarithm of the likelihood ratio, multiplied with -2, is considered to be (approximately) chi-squared distributed with $\nu$ degrees of freedom (where $\nu$ is the number of free parameters that are used to fit) and now you will have the sum of two such values which will be chi-squared distributed with $2\nu$ degrees of freedom.
Using the product of the likelihood ratios is similar to Fisher's method of combining p-values. (Which involves multiplication of the p-values and comparing with a chi-squared distribution)
Case of a composite alternative hypothesis 2
In your explanation you have a composite hypothesis for both hypotheses. You can still multiply likelihood functions and ratios. You can consider the two different datasets as if it was a single dataset that you fit with twice as many parameters.
A question is however what use your likelihood ratio has. Often you use it for a hypothesis test and you can know some distribution for the likelihood distribution given that null hypothesis is true. In your example, hypotheses of lognormal versus normal, it may not be easy to figure out this distribution. But, this is a problem that is unrelated to combining ratios and already occurs with a single dataset.

and conclude that all the data would be 7.75 times more likely to occur with sampling from a normal distribution than with sampling from a lognormal distribution?

This is not the interpretation of the likelihood ratio when it is computed with composite hypotheses. In the case of composite hypotheses there are nuisance parameters where you fit the cases that maximise the likelihood.
Compare for instance with a Bayesian analysis where those parameters could have a prior distribution. In that case you would have a likelihood ratio that can be interpreted as the relative probabilities for the data to occur with the two different hypotheses.
Case of information interpretation.
In the comments a situation is described that seems like model selection and one might use the likelihood as part of the Akaike information criterion (AIC).
In this case, AIC, it might make sense to add up the average log likelihood, the log likelihood divided by the number of observations. It is the average log likelihood that approaches the cross entropy and it might make more sense to add up the entropy rather than the likelihood (which differ by a constant, the number of observations).
If the number of observations are different for the different datasets then using this way of combining the likelihoods will be of influence.
Comment: I think this question is about multiplying the calculated likelihood ratios after maximizing the likelihoods separately on each dataset. In your answer you have $L(\theta;x_1)$ and $L(\theta;x_2)$, which are functions of $\theta$, while in the question the numbers 3.1 & 2.5 refer to $L(\hat{\theta}_1;x_1)$ and $L(\hat{\theta}_2;x_2)$ where the $\hat{\theta}$s are the dataset-specific MLEs.
Comment: @dipetkov clarified my question correctly. Does that change the answer?
Comment: @HarveyMotulsky that makes that I don't understand your question anymore. What hypothesis are you testing with the likelihood ratio test if you consider two different $\hat\theta$ for the two data sets?
Comment: Is the $-2\log$ x the likelihood ratio (approximately) Chi-squared? The two models are Normal and Log-Normal, so they are not nested? Burnham & Anderson use the likelihood ratio to select between the models (or equivalently, the AIC since the models have the same number of parameters). But they don't test a null hypothesis.
Comment: @dipetkov. I've lengthened my original question substantially to clarify precisely what my question is.
Comment: @dipetkov, that's right, for non-nested distributions this is not gonna work. However, one may still combine the p-values, but it will be different from adding the likelihood distributions.
Answer: Burnham and Anderson use a Monte Carlo simulation to evaluate selecting between the normal (N) and the log-normal (LN) models. I extend this simulation to select between the normal and the log-normal when the data comes in two batches (drawn from the same data generating process — either normal or log-normal). Empirically and under these idealized conditions, the simulation suggests that multiplying likelihood ratios across datasets is a reasonable approach.
Part I. Model selection with one dataset
I start by reproducing Table 6.14 to make sure I've set up the simulation correctly and to introduce the two evaluation measures.
(a) $\pi$ is the fraction of repeated experiments when the normal model, N, is selected over the log-normal model, LN, because it has higher likelihood:
$$
\begin{aligned}
\operatorname{I}\left\{\frac{L_{\text{N}}}{\operatorname{L}_{\text{LN}}} > 1\right\} = \operatorname{I}\left\{\ell_{\text{N}} > \ell_{\text{LN}}\right\}
\end{aligned}
$$
where $L$ is the likelihood and $\ell$ is the log-likelihood.
(b) $\operatorname{E}(w)$ is the average Akaike weight for the normal model.
This concept was new to me. The Akaike weight is defined in terms of the difference in AIC with the best candidate model, ie, the model with the smallest AIC. Here we are comparing two models with the same complexity (number of parameters), so the model with the smaller AIC has the higher likelihood.
$$
\begin{aligned}
w_{\text{N}} = \frac{\exp\left\{\ell_{\text{N}} - \ell_{\text{best}}\right\}}{\exp\left\{\ell_\text{N} - \ell_{\text{best}}\right\} + \exp\left\{\ell_{\text{LN}} - \ell_{\text{best}}\right\}}
= \frac{1}{1 + \exp\left\{\ell_{\text{LN}} - \ell_{\text{N}}\right\}}
\end{aligned}
$$
While $\pi$ indicates whether we select the correct model or not, the Akaike weights scale the difference $\ell_{\text{LN}} - \ell_{\text{N}}$ to a measure between 0 and 1. While it's wrong to interpret $w_{\text{N}}$ as the "probability that the normal model is correct", the weights indicate whether it is easy or difficult to discriminate between the models. Here is example: Say the data is normal and $\ell_{\text{N}} = -11$, $\ell_{\text{LN}} = -11.2$. Since the normal likelihood is higher, we choose the normal model, correctly; its Akaike weight is 0.55. (The Akaike weight of the log-normal model is 045.) Now if we had ~10x more data and the log-likelihoods are $\ell_{\text{N}} = -110$, $\ell_{\text{LN}} = -112$, we still select the correct model by comparing the likelhoods; but now $w_{\text{N}} = 0.88$. We take the same decision but with more "certainty".
And here is my version of Table 6.14; the results are close to those in the book. It's harder to discriminate between the two models when there is less data (we can tackle this by getting more data) and when the mean is larger (the distributions are very similar, so the model choice makes little difference).

n
data
μ=5: π
μ=5: E(w)
μ=10: π
μ=10: E(w)
μ=25: π
μ=25: E(w)
μ=50: π
μ=50: E(w)

10
normal
58
54
54
51
51
50
51
50

50
normal
78
71
65
57
56
51
53
50

100
normal
87
83
72
63
59
53
54
51

500
normal
100
100
91
87
71
62
61
53

10
log-normal
38
46
44
49
48
50
49
50

50
log-normal
21
29
34
43
43
49
47
50

100
log-normal
12
18
28
37
41
47
45
49

500
log-normal
0
1
9
13
29
38
39
47

Part II. Model selection with two datasets
Now that I can run the simulation for one dataset of size $n$, it's easy to extend it to two datasets of size $n_1$ and $n_2$. For comparison with the previous results I choose the sample sizes so that $n_1 + n_2 = n$, for $n = \{25, 50, 100, 500\}$ split into ratios $9:1$, $4:1$, $2:1$ and $1:1$. [I omit $n = 10$ as it's too small to split and compute the MLEs.]
Here is how I compute the pooled measures:
(a) Exactly as suggested, $\pi$ is the fraction of repeated experiments when the normal model, N, is selected over the log-normal model, LN:
$$
\begin{aligned}
\operatorname{I}\left\{\frac{L_{\text{N}}^{(1)}}{\operatorname{L}_{\text{LN}}^{(1)}}\frac{L_{\text{N}}^{(2)}}{\operatorname{L}_{\text{LN}}^{(2)}} > 1\right\} = \operatorname{I}\left\{\ell_{\text{N}}^{(1)} + \ell_{\text{N}}^{(2)} > \ell_{\text{LN}}^{(1)} + \ell_{\text{LN}}^{(2)}\right\}
\end{aligned}
$$
where the superscripts indicate datasets $1$ and $2$.
(b) The average Akaike weight for the normal model across the two datasets is:
$$
\begin{aligned}
w_{\text{N}} = \frac{\sum_{i}\exp\left\{\ell_{\text{N}}^{(i)} - \ell_{\text{best}}^{(i)}\right\}}{\sum_{i}\exp\left\{\ell_\text{N}^{(i)} - \ell_{\text{best}}^{(i)}\right\} + \exp\left\{\ell_{\text{LN}}^{(i)} - \ell_{\text{best}}^{(i)}\right\}}
\end{aligned}
$$
The log-likelihood is normalized separately for each dataset. So we can't compute the pooled Akaike weight from the likelihood ratio alone; we need to have the likelihood of each model separately.
I summarize the results in a small-multiples plot. There is a separate panel for each combination of sample size $n=\{25, 50, 100, 500\}$ and population mean $\mu=\{5, 10, 25, 50\}$. The dotted horizontal lines indicate the model selection performance (in terms of $\pi$ in blue and $\operatorname{E}(w)$ in red) when we analyze one dataset of size $n$. The solid lines indicate the "pooled" performance with $n$ samples split across two datasets according to the ratio given on the x-axis.
Figure 1: Selecting between the normal model (correct) and log-normal model (wrong) by pooling log-likelihood ratios between two datasets.

As Burnham and Anderson explain, $\pi$ reflects sampling variation (how often we select the correct model) and $\operatorname{E}(w)$ reflects inferential uncertainty (how easy it is to discriminate between the two models).
In this simulation at least, we don't lose much accuracy to select the correct model unless $\mu$ $=$ $5$ or the (total) sample size is $n$ $\leq$ $50$: the solid blue line is only slightly below the dotted blue line. But there is more uncertainty (big gap between the solid and dotted red lines) and it's worse to have the samples unevenly split between the two datasets. So we lose some information by splitting the data and the estimation into two; this is not surprising. In this simulation there is no order in how the datasets are generated. But in practice the datasets could be collected & analyzed sequentially. There may be little gain to collect a small followup sample and study it separately. It would be better to put the datasets together and analyze all the data we have, esp. if as stated in the question, the datasets are generated under similar conditions.
Figure 2: Selecting between the normal model (wrong) and log-normal model (correct) by pooling log-likelihood ratios between two datasets.

The R code to reproduce the simulation (mostly in base R) and the figure (with ggplot2).
<code>
R <- 100000

# Find the MLEs of the normal mean and variance; Compute the log-likelihood.
mle.norm <- function(x) {
  xbar <- mean(x)
  s2 <- mean((x - xbar)^2)
  sum(dnorm(x, xbar, sqrt(s2), log = TRUE))
}
# Find the MLEs of the log-normal mean and variance; Compute the log-likelihood.
mle.lnorm <- function(x) {
  lxbar <- mean(log(x))
  ls2 <- mean((log(x) - lxbar)^2)
  sum(dlnorm(x, lxbar, sqrt(ls2), log = TRUE))
}

# Generate normal data; Maximize the normal and log-normal log-likelihoods.
fit.norm <- function(n, mu) {
  sigma <- 1
  x <- rnorm(n, mu, 1)
  c(ll.a = mle.norm(x), ll.b = mle.lnorm(x))
}
# Generate log-normal data; Maximize the normal and log-normal log-likelihoods.
fit.lnorm <- function(n, mu) {
  sigma <- 1
  phi <- sqrt(mu^2 + sigma^2)
  x <- rlnorm(n, log(mu^2 / phi), sqrt(log(phi^2 / mu^2)))
  c(ll.a = mle.norm(x), ll.b = mle.lnorm(x))
}
# ---
# Do model selection using log-likelihood ratio & Akaike weights.
# NB. Since the two candidate models -- normal and log-normal -- have the same
# number of parameters, AIC(M1) - AIC(M2) = LL(M1) - LL(M2) = log{L(M1)/L(M2)}
select.model <- function(lls) {
  lls %>%
    drop_na() %>%
    mutate(
      better = ll.a > ll.b,
      Δ.a = ll.a - pmax(ll.a, ll.b),
      Δ.b = ll.b - pmax(ll.a, ll.b),
      weight = exp(Δ.a) / (exp(Δ.a) + exp(Δ.b))
    ) %>%
    summarise(
      π = mean(better),
      `E(w)` = mean(weight)
    )
}
select.model.2batches <- function(lls) {
  lls %>%
    drop_na() %>%
    mutate(
      better = (ll.a1 + ll.a2) > (ll.b1 + ll.b2),
      Δ.a1 = ll.a1 - pmax(ll.a1, ll.b1),
      Δ.b1 = ll.b1 - pmax(ll.a1, ll.b1),
      Δ.a2 = ll.a2 - pmax(ll.a2, ll.b2),
      Δ.b2 = ll.b2 - pmax(ll.a2, ll.b2),
      weight = (exp(Δ.a1) + exp(Δ.a2)) / (exp(Δ.a1) + exp(Δ.b1) + exp(Δ.a2) + exp(Δ.b2))
    ) %>%
    summarise(
      π = mean(better),
      `E(w)` = mean(weight)
    )
}
# ---
# Boilerplate code to run the simulations
simulate.norm.data <- function(n, mu, reps = R) {
  seq(reps) %>% map_dfr(~ fit.norm(n, mu))
}
simulate.lnorm.data <- function(n, mu, reps = R) {
  seq(reps) %>% map_dfr(~ fit.lnorm(n, mu))
}
simulate.norm.data.2batches <- function(n, mu, p, reps = R) {
  n1 <- round(n * p)
  n2 <- n - n1
  data1 <- simulate.norm.data(n1, mu) %>% rename_with(~ str_c(., "1"))
  data2 <- simulate.norm.data(n2, mu) %>% rename_with(~ str_c(., "2"))
  bind_cols(data1, data2)
}
simulate.lnorm.data.2batches <- function(n, mu, p, reps = R) {
  n1 <- round(n * p)
  n2 <- n - n1
  data1 <- simulate.lnorm.data(n1, mu) %>% rename_with(~ str_c(., "1"))
  data2 <- simulate.lnorm.data(n2, mu) %>% rename_with(~ str_c(., "2"))
  bind_cols(data1, data2)
}
run.simulations.save.results <- function(seed = 1234) {
  set.seed(seed)
  # Table 6.14A
  grid <- expand_grid(
    n = c(10, 25, 50, 100, 500),
    μ = c(5, 10, 25, 50)
  )
  sim.norm <- grid %>%
    mutate(
      data = pmap(list(n, μ), simulate.norm.data),
      table = map(data, select.model)
    )
  sim.lnorm <- grid %>%
    mutate(
      data = pmap(list(n, μ), simulate.lnorm.data),
      table = map(data, select.model)
    )
  bind_rows(
    norm = sim.norm,
    lnorm = sim.lnorm,
    .id = "data"
  ) %>%
    unnest(
      table
    ) %>%
    write_csv(
      "600592-Table-6.14A.csv"
    )
  # Table 6.14B
  grid <- expand_grid(
    n = c(25, 50, 100, 500),
    μ = c(5, 10, 25, 50),
    p = c(1 / 10, 1 / 5, 1 / 3, 1 / 2)
  )
  sim.norm <- grid %>%
    mutate(
      data = pmap(list(n, μ, p), simulate.norm.data.2batches),
      table = map(data, select.model.2batches)
    )
  sim.lnorm <- grid %>%
    mutate(
      data = pmap(list(n, μ, p), simulate.lnorm.data.2batches),
      table = map(data, select.model.2batches)
    )
  bind_rows(
    norm = sim.norm,
    lnorm = sim.lnorm,
    .id = "data"
  ) %>%
    unnest(
      table
    ) %>%
    write_csv(
      "600592-Table-6.14B.csv"
    )
}

# 6.7.2 A Normal Versus Log-Normal Example
library("tidyverse")

plot_selection <- function(data) {
  table.b %>%
    filter(data == {{ data }}) %>%
    ggplot(
      aes(
        p, value,
        group = name,
        color = name
      )
    ) +
    geom_line() +
    geom_hline(
      aes(
        yintercept = value,
        color = name
      ),
      linetype = 2,
      data = table.a %>% filter(data == {{ data }})
    ) +
    facet_wrap(
      ~ n + μ,
      scales = "free_y",
      labeller = "label_both"
    ) +
    scale_x_continuous(
      breaks = c(1 / 10, 1 / 5, 1 / 3, 1 / 2),
      labels = c("1:9", "1:4", "1:2", "1:1")
    ) +
    theme(
      axis.title = element_blank(),
      legend.title = element_blank(),
      panel.spacing.y = unit(0, "line")
    )
}

run.simulations.save.results()

table.a <- read_csv("600592-Table-6.14A.csv")
table.b <- read_csv("600592-Table-6.14B.csv")

table.a %>%
  pivot_wider(
    names_from = μ,
    names_glue = "{μ},{.value}",
    values_from = c(π, `E(w)`),
    values_fn = ~ round(100 * .)
  ) %>%
  select(
    n, data,
    starts_with("5,"),
    starts_with("10,"),
    starts_with("25,"),
    starts_with("50,")
  ) %>%
  knitr::kable()

table.a <- table.a %>%
  filter(n >= 25) %>%
  pivot_longer(c(π, `E(w)`))
table.b <- table.b %>%
  filter(n >= 25) %>%
  pivot_longer(c(π, `E(w)`))

plot_selection("norm")
plot_selection("lnorm")
</code>
Comment: Thanks. Just to clarify. For the normal distributions, you always choose a population standard deviation of 1.0. So the coefficient of variations runs from .2 to 0.02.  For the lognormal distributions, the SD of the logarithms is 1.0, so the Geometric standard deviation is always exp(1) or 2.72.

I understand the blue lines (solid and dotted) in your figures, but can't really understand the red lines. What are they telling us?
Comment: I set up the simulation exactly as in Table 6.14 as the only thing I wanted to vary is the data split. From the text: "...select between the simple normal and log-normal models, variance(y) = 1 for both distributions". Of course, I also put all the code up, so feel free to vary $\sigma^2$ if that's of interest.
Comment: Here is my interpretation of $E(w)$ (in red) vs $\pi$ (in blue). To calculate $\pi$ you only look at the selection decision (was the normal model or the log-normal the one selected?). To calculate $E(w)$ you look at the difference between the normal and log-normal likelihoods, so you get an idea of how "difficult" it was to choose between the two models. If the likelihood of the correct model is much higher than the wrong model, it will also have higher weight (than 0.5). So the red lines are telling us that we do lose information about the model when we split the data. Hardly a surprise.
Comment: PS. By "lose information" I mean the gap between the solid and dotted red lines, vs. the much smaller, or in some cases, non-existent gap between the solid and dotted blue lines.


Frequency of an event at a given time, from a list of times of occurrence

Question: I have data in the following format:
[$t_1, t_2, t_3, \dots, t_n$] - which is a list of times at which an event has occurred during a day
They are in minutes within 24 hours and limited by $t_{\text min} == 0, t_\text{max} == 1440 $.
What I would like to obtain is a (smooth, if possible) function that takes a given time and returns the frequency of that event, sampled at the given time.
For example, if I have the following list (values represented as HH:MM for simplicity):
[05:20, 06:00, 07:40, 08:20, 09:00, 09:10, 09:20, 09:30, 09:40, 09:50, 10:00 (...)]
if I sample my function at t=06:10 I'd like to obtain the frequency of around f(t)=1/40 [1/min]
and if I sample at t= 09:45, f(t)=1/10 [1/min]
and at t= 09:00, f(t)= something in between (?)
I was wondering what functions should I look at to be able to fit to my data  to obtain the desired result.
Comment: I don't think it's a matter of fitting functions unless there is a really simple dependence on time of day that can be modelled as sines and cosines. Rather, density estimation is the name of the game. There's a twist: if you regard times after midnight as following times before midnight, you'll want to ensure that the estimation wraps around. Most programs will require that you work in minutes, not hours and minutes.
Answer: I decided to use a weighted average of the time differences between occurrences. The weight is the distance (in time) from the sampled point.
Comment: Do you mean the weight is the *inverse* of time from the sampled point?
Comment: No, the distance between the point of sampling and a given occurrence. I give detail ed explanation in this paper https://link.springer.com/chapter/10.1007%2F978-3-030-03424-5_18
but it's basically so that the closer a given occurence is (in time), the more relevant it is for the frequency sampled at a given point
Comment: Looks like it's just a a moving average with exponential weight function. Seems reasonable.
Answer: Use a boxcar filter. The problem with survival methods is that they estimate hazards which have to be transformed to risks and, at times, rely on parametric assumptions. Since you have no censoring in these data, the mean is an unbiased estimator of the counting process. The boxcar filter provides a nonparametric moving average of the frequency of events at each time and is a nice descriptive statistic.
An example of my superinefficient approach:
<code>data(cancer)
window <- 10
time <- sort(unique(cancer$time))
freqs <- colSums(abs(outer(time, time, `-`)) < window)
val <- data.frame(n=freqs, t=time, t0=pmax(time - window/2, 0), t1=time+window/2)
val$t1 <- pmin(val$t1, c(val$t[-1], Inf))
plot(val$t1, val$n, type='s', xlab='time', ylab='Number of events in 10 day window')
</code>
Comment: "Box-Car" better as "boxcar" to reduce the risk of someone attributing yet another thing to George E.P. Box and/or inventing a mythical collaborator Car. (According to legend, "B. Chir" has multiple publications in various data bases given the number of surgeons who put their degrees after their names.)


Automatic height of UITableView based on its content

Question: I have a UITableView within a standard ViewController and I'd like its height to stretch along with its contents - exactly like a normal webpage. So for example, if a single row is 50px and two rows exist, then the height would go no further than those two rows (100). However, if there are 100 rows then the tableview should go beyond the bottom of the window.
I'd also like to keep a margin of 10 between the bottom of the table view and the window when the tableview is taller that the main window itself and the user has scrolled to the bottom of the list.
Is there a way in interface builder to do this or is this something that has to be done in code?
Here's an example of what I want it to look like when there are three items in the list;

And when it reaches the bottom of the list, it has a constraint in relation to the bottom of the window;
Answer: I think what you want is easier to get if you just expand the table to the bottom of the view (with your 10px margin) and make the background transparent/grey. Then, configure each row to have a white background. The result will look as your screenshot.
Is there any specific reason to expand the table frame? if so, please explain...
Comment: Thanks for the suggestion, I will give that a go. As I am coming from a web development background, I am naturally expecting the container to expand in unison with the size of its contents, but I can't seem to quite grasp this behaviour in iOS when using constraints.
Answer: I am also doing same thing what you want to after searching on stack overflow found solution.
If you want to set the table view height dynamically, I think the best way is to set the table view height constraint equal to its content size after the table view done loading. I mean something like,
tableViewHeightConstraint.constant = tableView.contentSize.height

Just do this -

Add UITableView in UIScrollview. This will give infinite scroll to tableview.
Disable UITableView scroll.
Add a height constraint for table view. (height = 50px)
Select table height constraint and Set priority to High (750)

Create an outlet of the height constraint(Ctrl+Drag).

Set the height constraint to table view content height.
Add below code in cellForRowAt:

tableViewHeightConstraint.constant = tableview.contentSize.height
wherever you need to reload data of your table.
Hope this would help other.


Failed to allocate memory, NoMemoryError

Question: I am trying to serialize an object and save it to a Postgresql Database using ActiveRecord in ruby (without rails). I am using the following code to do so:
<code>  components.keys.each do |key|
    cphash = {:training_url => components[key][:training_url],
              :klass => key.to_s,
              :component => Marshal::dump(components[key][:component]),
              :network_id => n}
    inline = Inline.new(cphash)
    inline.save!
  end
</code>
However, I keep getting the error
<code>failed to allocate memory (NoMemoryError)
</code>
How do I get around this. I have tried breaking the object into smaller bits (I am saving each bit in the 'each' above), but Im still getting the error. Anyone knows what gives.
Answer: The Inline gem is for running C code within ("inline") ruby.
http://www.zenspider.com/ZSS/Products/RubyInline/
Comment: Umm, inline is not the inline gem, its a class I have written
Comment: Well the provide the source. How can you expect us to help you when you don't provide all of the code? NoMemoryError comes up for very specific reasons.


"Equal Division of objects into groups" formula

Question: My book says the number of ways to distribute to 2n objects equally among two groups where order is considered is $\frac{2n!}{(n!)^2}$ but I have a doubt
Let's take an example
No. Of objects= 4 $\{A, B, C, D\}$
Number of ways to distribute these four object into groups which contain two objects each
G1(_ )               G2( _)
Now let's make all possible groups
<code> **G1**             **G2**
    AB.  |   AC, AD, BC, BD, CD
    AC.  |   AB, AD, BC, BD, CD
    AD.  |   AB, AC, BC, BD, CD
    BC.  |   AB, AC, AD, BD, CD
    BD.  |   AB, AC, AD, BC, CD
    CD.  |   AB, AC, AD, BC, BD
</code>
So, the number of ways to distribute these objects into two groups each having two objects when order of groups is considered is   6×5= 30 ways
As,  $(AB, AC) (AB, AD) (AB, BC) (AB, BD) (AB, CD)$ 5 groups from each row above
**But according to the formula it is **
4!/(2!×2!)= 6 ways
Please explain how this formula is true.
Comment: Use mathjax to render the maths, you have a tuto here https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference
Comment: Why can't G2 contain AB in the first line?
Comment: @JMP bcoz when we place A and B in 1st group they are already taken and we can't put  them in 2nd group
Comment: Then you can't have AC in the second group, can you ?
Comment: @Fabien here i tried to show when I take A and B in 1st group then those in G2 are all other options of groups that can be in group 2
Comment: @NikhilPant but if you put A and B in G1, you can't put one of them in G2. Or do you mean you want to split the pairs of elements in two groups ?
What do you mean by "where order is considered" ? The order in a group ? Between the two groups ? Both ?
Comment: @Fabien thanks man I got it
Comment: In the first line if $G1$ is $AB$ then $G2$ can't have $A$ or $B$.  It can only have $C$ and $D$.  And as the order of groups matter but not the order within the groups do not there is only one possible $G2$.  So there are $6\times 1 = 6$ ways.
Comment: @fleablood thanks i got it
Answer: There are $(2n)!$ ways to line up the objects, because each way to do it is a permutation of these objects. The $n$ first objects in the row form group 1 and the $n$ last objects form group 2. There are $n! n!$ copies of the same distribution in two groups in this because each group can be permutated. Hence the result
\begin{equation}
\frac{(2n)!}{(n!)^2}
\end{equation}
Note that if you don't differentiate the groups (by their number) you need to divide this result by 2.


Enable CKeditor on Webform's textarea

Question: I'm trying to create a contact form supporting HTML formatting with Webform and CKEditor. So I created a new webform with a single field of type Textarea and now I have a simple webform with the basic textarea.
How can I enable rich text formatting on this form? It doesn't work by default and I cannot find any option to enable it neither in Webform nor CKeditor settings.
Comment: May I know have you found any solution for this?
Comment: Not yet. I have installed only Webform and CKEditor and I'm looking for a simple module that will bind them together (without extra dependencies).
Answer: Webform HTML Textarea
Try this module.
Comment: Does it support CKeditor natively or I have to install the [Wysiwyg](https://www.drupal.org/project/wysiwyg) module? I have not found this info in the module description.
Comment: You have to install Wysiwyg module. Once it done, choose html textarea instead of normal textarea. (When you create the field)

Enable CKeditor on Webform's textarea