Time Series Decomposition: Is it necessary (or wise) to remove outliers beforehand?

Question: Do outliers change the outcome of time series decomposition?
As far as I understand it, outliers occur in the residual-component. In the residuals plot they can be visually identified as spikes.
However, if I know the outliers within a time series before applying the decomposition, does it make sense to remove the outliers in order to increase the "quality" of the decomposition?
Comment: You may want outliers to show mostly in residuals, but that hope or expectation may not be satisfied, depending largely on how your decomposition is fitted, and whether the fitting method is robust. Ignoring outliers is usually regarded as a bad idea. In the case of time series there is a real problem that you have to fill gaps for many calculations to make sense at all.
Answer: See section 12.9 of Hyndman and Athanasopoulos's textbook.
https://otexts.com/fpp2/missing-outliers.html
Often in a time series you need to handle outliers -- e.g. if you are predicting airline passengers you will have big outliers around Sept 11, 2001 and currently due to COVID-19. These are special events that have to be handled or they will throw everything off. So, yes, you need to handle these before the decomposition.
Data errors, or likely data errors, have to go.
Otherwise be careful. Be particularly careful in just applying some global filter (e.g. everything outside 3 standard deviations). You may be just suppressing the naturally high variability of the data.
Answer: In the cross sectional literature it is generally seen as a bad idea to remove an outlier not caused by a data error. Instead you should try to understand why it is occurring. Which explains your data better. I have not seen it addressed much in time series.


Convolution of probability mass functions (3 non-parametric distributions)

Question: I am familiar with the convolution of probability mass function when it involves two random variables, but I get a little confused when there's a third one.
I have to find the probability mass function of Z = W + X + Y where W, X and Y are independent random variables and with the given table:

I am not sure about the general formula to solve this. Thank you in advance for your help!
Comment: Hi @boiba, did this answer your question? If so, could you please accept and upvote the answer? Otherwise, what can be clarified?
Answer: You asked for a general formula for convolving several independent variables, so I'll give you that rather than solving the specific case from your homework.
If you can convolve W and X into a third variable (call it Q), then you can convolve Q with Y to get Z, following the same procedure you just used:
$$ f_Z(z) = f_Q(z) * f_Y(z) = f_W(z) * f_X(z) * f_Y(z) $$
More details can be found from this tutorial. Note the claim:

Nevertheless, this quickly becomes computationally difficult. Thus, we often resort to other methods if we can. One method that is often useful is using moment generating functions, as we discuss in the next section.


how to post a Photo or link on facebook wall without using FBSDK in android

Question: Want to post some photos or link on user Facebook wall without using FBSDKShareDialog. I am already post with using the FBSDKSharePhoto And FBSDKShareDialog defult dialog from facebook. For that I do the below code.
Comment: where is the code can you post the code .
Comment: Could the upvoter please explain how this is useful and clear?
Comment: Are you need to share the Data From Your App to Facebook App without FBSDk ?
Answer: If you don't want to use the dialogs (regardless of whether you invoke them using the SDK or your own code), you will have to resort to using Graph API.
In order to create posts using Graph API, you will have to POST to the <code>/me/feed</code> endpoint (docs), but that requires an access token.
In order to obtain an access token from Facebook, you need to implement Facebook login, and their Platform Policy section 7.2 says:

Native iOS and Android apps that implement Facebook Login must use our official SDKs for login.

So in the end, you might as well use their SDK after all, or risk your app getting blocked.

However, assuming you have an access token and you are using the SDK (but not the dialog - since that is what you are actually asking), you can use this docs section to learn about how you can make Graph API calls using the SDK.


icCube with olap4php

Question: We started to build a PHP application to connect to icCube and I want an example for olap4php, because I couldn't find it anywhere in the icCube documentation.
In particular, I need the Provider to use in building connection object to icCube.
<code>// Example for Mondrian
$connection = new Connection(

    new SoapAdaptator('http://localhost:8080/mondrian/xmla'),
    array(
        'DataSourceInfo' => '**Provider**=Mondrian;DataSource=MondrianFoodMart;'
        'CatalogName' => 'FoodMart',
        'schemaName' => 'FoodMart'
        )
);
</code>
Can you help me?
Answer: Olap4Php is using XMLA protocol, the catalog is icCube's schema :
<code>$properties = array(
   XMLAConnection::PROP_CATALOG => 'Your icCube Schema'
);

$connection = new XMLAConnection( 'http://localhost:8282/icCube/xmla', $properties );
</code>

icCube with olap4php


If $f'(a)=\lim\limits_{x\to a}\dfrac{f(x)-f(a)}{x-a}$, can I write $\lim\limits_{x\to a}f(x)=f(a)+\lim\limits_{x\to a}\big[f'(a)\!\cdot\!(x-a)\big]$?

Question: I want to solve for $f(x)$ in the definition of the derivative at a point. Can I write $$f'(a)=\lim_{ x \to a }\frac{f(x)-f(a)}{x-a}\quad\implies\\ \lim_{ x \to a }f(x)=f(a)+\lim_{ x \to a }\big[f'(a)\!\cdot\!(x-a)\big]\;\;?$$ I know it seems trivial since evaluating the limits in the latter equation gives $f(a)=f(a)$, but the motivation is to determine whether a linear approximation of $f(x)$ is exact in the limit as $x$ approaches $a$. If my reasoning is incorrect, is it possible to solve for $f(x)$ in the derivative equation?
Comment: Second summand is $0$, so, what you state is, that differentiable function is continuous.
Comment: Yes, the left implies the right. You're effectively multiplying both sides by the limit of $x-a$ and then taking limits.
Comment: @BobTivnan The latter equality is technically correct, but pretty much useless since it's essentially $f(a)=f(a)$ (as you wrote). The "*motivation*" part suggests that you might misconstrue what the equality actually means. Note that the $f'(a)$ factor is wholly irrelevant there, and the similar equality $\lim_{ x \to a }f(x)=f(a)+\lim_{ x \to a }\big[C\!\cdot\!(x-a)\big]$ holds true for *any* constant $C\,$, not just for $C=f'(a)\,$.
Comment: It is not possible to "solve for $x$".  Consider that $\lim_{x\to 0}x=0$ cannot be "solved for $x$".
Comment: But is the multiplication property valid when the $x-a$ is inside a limit? It seems we should take the quotient of the limits first and then multiply both sides by $\lim_{x \to a}(x-a)$. However, this limit is zero which makes me wonder whether the quotient property of limits can be applied.
Answer: The rigorous derivation would rewrite the first equation as the equivalent
$$
\lim_{x\to a} f'(a)=\lim_{ x \to a }\frac{f(x)-f(a)}{x-a},
$$
then multiply everything by $\lim_{x\to a} (x-a)$ to obtain
$$
\lim_{x\to a} (x-a) \cdot \lim_{x\to a} f'(a)= \lim_{x\to a} (x-a) \cdot\lim_{ x \to a }\frac{f(x)-f(a)}{x-a},
$$
and finally use the property that the product of two convergent limits equals the limit of the product (on both sides):
$$
\lim_{x\to a} \big( (x-a) f'(a) \big) = \lim_{x\to a} \bigg( (x-a) \frac{f(x)-f(a)}{x-a} \bigg) = \lim_{x\to a} \big( f(x)-f(a) \big).
$$
The last statement in the OP now follows from $$\lim_{x\to a} \big( f(x)-f(a) \big) = \lim_{x\to a} f(x) - \lim_{x\to a} f(a) = \big( \lim_{x\to a} f(x) \big) - f(a)$$ and rearranging.
(It's a very good exercise to identify the precise facts/rules/theorems that are being used in every step.)
Comment: Careful not to misread: it's $\lim_{x\to a} f'(a)$, not $\lim_{x\to a} f'(x)$.
Comment: This seems totally legitimate, but I wonder, why are we allowed to multiply by $0=\displaystyle\lim_{x\to a}(x-a)$?
Comment: You can multiply both sides on an equation by any quantity and the result is again a valid equation. This is not to be confused with cancelling the same quantity on both sides of an equation, where the nonzeroness is important. (Indeed, that fallacy would lead to a "proof" that every continuous function is differentiable here, or something like that.)
Comment: So by multiplying both sides by $\lim_{x \to a}(x-a)$ and rewriting the products inside a single limit on each side, you are basically working around the quotient property of limits which would be problematic since we would get division by zero?
Comment: @GregMartin: So that means the setup is not reversible? And from what I understand, any arbitrary equation (even false) maybe multiplied by $0$ to give a true equation, which must be either an implication of the initial (if it's true) or just a vacuous truth (if it's false).
Comment: @GregMartin I disagree with the first equation, as the derivative is not necessarily continuous. No need for the limit in the LHS


Converting Date Format to numeric Date

Question: I have a Date in this format:
Tue Mar 10 00:00:00 UTC 1987

It is stored in a Object Date.
Object tmp = solrDoc.getFieldValue("date_from")

I would like to convert it to a strictly numeric format, without the hours, timezone etc., like
10.03.1987

This is what I tried so far:
DateFormat date = new SimpleDateFormat("dd.MM.yyyy");
date.format(tmp);

It returns:
 "java.text.SimpleDateFormat@7147a660"

Comment: Convert the `Object` to `Date` and then pass it to the converter.
Comment: You need `String formattedDate = date.format(tmp);`.
Comment: If you have date in such format, then `dd.MM.yyyy` will not work at all.
Comment: @OldCurmudgeon this solved my problem.. I kinda always struggle with date datatypes..
Comment: I recommend you avoid the `SimpleDateFormat` class. It is not only long outdated, it is also notoriously troublesome. Today we have so much better in [`java.time`, the modern Java date and time API](https://docs.oracle.com/javase/tutorial/datetime/).
Comment: Please search before asking. Formatting of dates has been covered literally hundreds of times already. So you'll find a good answer much faster that way.
Answer: You're attempting to use the format method on an Object, but according to the documentation, you need to pass this method a Date. So what you actually need to do is parse that original String into a Date, then format that.
For example, you could do it like this:
String tempString = String.valueOf(solrDoc.getFieldValue("date_from"));
DateFormat formatToRead = new SimpleDateFormat("EEE MMM dd HH:mm:ss z yyyy");
DateFormat formatToWrite = new SimpleDateFormat("dd.MM.yyyy");
formatToWrite.setTimeZone(TimeZone.getTimeZone("UTC"));
Date tempDate = null;
String result = null;
try {
    tempDate = formatToRead.parse(tempString);
} catch(ParseException e){
    e.printStackTrace();
}
if(tempDate != null){
    result = formatToWrite.format(tempDate);
    System.out.println(result);
}

Note that I had to set the TimeZone on the formateToWrite in order to keep it in UTC.
If you want more information on the SimpleDateFormat that I used to parse your original String, please refer to this SO answer.
Comment: I was close, the thing I didn't do is to store the formated value in a String object.. after I did that, it worked just fine.. Thank you for your answere

Converting Date Format to numeric Date


How to get preferred AutoLayout height given a width for a collection view cell with multiline labels?

Question: I'm building a list view with self-sizing in my app using UICollectionView. Getting UICollectionViewFlowLayout to do a vertical list layout is a pain, so I'm writing my own UICollectionViewLayout subclass to do it.
My cells look a lot like regular table view cells - an image view on the left, a couple of labels vertically stacked in the center, and an accessory view on the right. The labels can wrap to a few lines and grow in font size according to the system font size setting, which is why they need to be self-sizing. Constraints are sensible - left to image view, image view to label, label to accessory, accessory to right, label to top and bottom.
To size my cells via preferredLayoutAttributesFittingAttributes, I need to get the desired height of the cell given the width of the collection view. I'd expect to use systemLayoutSizeFittingSize:horizontalPriority:verticalPriority: for this, passing a size like {desiredWidth, crazyLargeHeight}, UILayoutPriorityRequired for horizontal priority, and 1 for vertical priority. But the size I get back from systemLayoutFittingSize isn't sensible. The width is some previous width value from the cell (which doesn't necessarily match self.bounds.size.width or the passed-in layoutAttributes' size.width), and a height that works with that width. I also have tried passing a small height instead of a large one, and it still doesn't return the size it actually needs.
So how do I get the preferred height for the cell given a width value?
Sample code:
-(UICollectionViewLayoutAttributes *)preferredLayoutAttributesFittingAttributes:(UICollectionViewLayoutAttributes *)layoutAttributes
{
  UICollectionViewLayoutAttributes *result = [super preferredLayoutAttributesFittingAttributes:layoutAttributes];
  CGSize exampleSize = CGSizeMake(layoutAttributes.size.width, 20);
  CGSize size = [self systemLayoutSizeFittingSize:exampleSize withHorizontalFittingPriority:UILayoutPriorityRequired verticalFittingPriority:1];
  result.size = size;
  return result;
}

Comment: You need your `self` here to be _self-sizing_ in the sense that it has sufficient _internal constraints_ so that when `systemLayoutSizeFittingSize`, the internal constraints provide a _solution_ to the size question. Would it help you to see an example that works?
Comment: " thought I might be misunderstanding systemLayoutSizeFittingSize." Well, it looks like you _are_ misunderstanding it. But given the right conditions and the right call, it _will_ do what you are hoping for.
Comment: @matt exactly. Which means I need to tinker.
Comment: @matt and...turns out if you ask `self.contentView` for `systemLayoutSizeFittingSize`, it works fantastic. Posting an answer in case someone else has the same headache someday.
Answer: Turns out if you ask the cell's content view for a size instead of the cell itself, it works. And you have to give a sample height that is smaller than will be needed.
-(UICollectionViewLayoutAttributes *)preferredLayoutAttributesFittingAttributes:(UICollectionViewLayoutAttributes *)layoutAttributes
{
  UICollectionViewLayoutAttributes *result = [super preferredLayoutAttributesFittingAttributes:layoutAttributes];
  CGSize exampleSize = CGSizeMake(layoutAttributes.size.width, 20); //magic number for example purposes...my cell will definitely be taller
  CGSize size = [self.contentView systemLayoutSizeFittingSize:exampleSize withHorizontalFittingPriority:UILayoutPriorityRequired verticalFittingPriority:1];
  result.size = size;
  return result;
}

I'm guessing that the content view isn't pinned to the cell's bounds the way I expected. And this exact implementation will likely fall down if the content view is inset for some reason. But the key to solving this was the fact that having the cell contents pinned to the cell's content view does not mean that the cell itself will compute a size as you expect.


How do I connect to multiple servers at the same time using gRPC in C++?

Question: I wanted to submit the same data to multiple servers at the same time using gRPC.
I looked at the greeterAsyn2 c++ example:
https://github.com/grpc/grpc/blob/v1.8.x/examples/cpp/helloworld/greeter_async_client2.cc
From the example: In order to create 1 channel you can just do this:
GreeterClient greeter(grpc::CreateChannel(
            "localhost:50051", grpc::InsecureChannelCredentials()));

As this would create the stub for the channel:
class GreeterClient {
  public:
    explicit GreeterClient(std::shared_ptr<Channel> channel)
            : stub_(Greeter::NewStub(channel)) {}
}

And I would be able to submit data using
greeter.SayHello("hello world");

But what if I want to submit data to 2 different servers using 2 different channels?
If I would just add another GreeterClient object called greeter2:
GreeterClient greeter2(grpc::CreateChannel(
            "10.0.0.3:9008", grpc::InsecureChannelCredentials()));

I get a segmentation fault when trying to submit the data to the second server:
greeter2.SayHello("hello world");

Comment: I'm facing the same problem now. Have you solved the seg fault on creating multiple connections to servers? It seems like a restriction on the client stub but not really clear to me.
Answer: What you are doing seems okay, I will need to see the stack trace to see what is going wrong.
Consider the channel as an abstract data pipe to a service and if you want to talk to multiple servers, just create multiple channels and send rpcs on them separately.


Security Review - Untrusted users are allowed to input dangerous HTML tags

Question: I am getting the following error in my security review:
Untrusted users are allowed to input dangerous HTML tags.
Is this because in my text formats I've got the following:
Plain text  All roles may use this format
Should I disable  anonymous users from being able to access plain text?
Also, it is saying:
It is recommended you remove the following tags from roles accessible by untrusted users.
<code>img
</code>
Should I remove the img tag from authenticated users???? Does this mean that authenticated users will not be able to uploaded images to their site at all?
Comment: What security review tool are you using?
Comment: http://drupal.org/project/security_review @MPD
Answer: Plain text is just that, no html tags, so this shouldn't be a security issue.
Perhaps the img tag is considered dangerous because it is possible to use a specially crafted png or jpg to launch an attack against a users browser.
Comment: Thanks for that @MarkMuhleder  . Any tips as to how to get rid of this error message "Untrusted users are allowed to input dangerous HTML tags."
Comment: Does the module let you specify some roles as "trusted users"?
Comment: I see that it does http://drupalcode.org/project/security_review.git/blob/refs/heads/7.x-1.x:/README.txt "Navigate to
  39 Administer >> Reports >> Security Review >> Settings to mark which roles are
  40 untrusted."
Comment: Thank you!!!
Having only the anonymous user set as untrusted user fixed the problem.
Comment: It may "fix the problem" but is it the right thing to do? See http://drupalscout.com/knowledge-base/importance-user-roles-and-permissions-site-security
Answer: img tags are vulnerable to XSS attacks
Since my site required roles higher than authenticated to submit content, I simply unchecked "authenticated user" from admin/config/content/formats/filtered_html and that solved my audit report.


basic node website returns "Cannot GET /" when hosted with forever.js

Question: I'm trying to get my first production node website up (just a basic hello world on my production web server).
The below is what I'm using (basic http proxy to pass off apache websites to port :9000 and node websites to port :8000). I know this part works because apache vhosts are forwarded as I expect. What does not work however is the node part -instead I get the error below
"Cannot GET /"

This is running node 0.8.1 on Ubuntu 12.04
I'm hosting this with forever.js (forever start foo.js). when I echo the NODE_ENV -it shows "production"
It might also be noted that I don't have node_modules on the path (as you will see in my require statements) **not sure if this has anything to do with my issue
var httpProxy = require('/usr/local/lib/node_modules/http-proxy/lib/node-http-proxy');
var express = require('/usr/local/lib/node_modules/express/lib/express');

httpProxy.createServer(function (req, res, proxy) {

    var nodeVhosts = ['www.mysite.com'];
    var host = req.headers['host'];
    var port = nodeVhosts.indexOf(host) > -1
        ? 8000
        : 9000;

    proxy.proxyRequest(req, res, {host: 'localhost', port: port});
}).listen(80);

var one = express.createServer();
one.get('/', function(req, res){
  res.send('Hello from app one!')
});

var app = express.createServer();
app.use(express.vhost('localhost', one));
app.listen(8000);

Answer: Since you are running Ubuntu, you might take a look at upstart.  In case you don't know, upstart replaces the old-school-unix init-scripts approach to starting and stopping services.  (Those were dark, scary days!)  If you want your app to start automatically when your box boots/reboots and restart automatically after it (your app) crashes, then you want upstart.  Learning the basics of upstart is easy, and then you have a tool that you can use over and over again, whether it's node, apache, nginx, postfix, mongodb, mysql, whatever!
I mean no disrespect to the good folks who work on the forever module.  Arguably, it does have a solid use case, but too often it is used to imperfectly duplicate the bedrock which is already on your system -- upstart.  Also, you might Google for some of the comments made by experienced users and some node.js committers about the forkability of node.js and the pitfalls, which are very relevant to forever.
I'd like to post links, but I don't have enough rep yet.  Hopefully what I wrote is enough to google on.
Good luck!
Comment: interesting -I actually wrote an upstart script to run it using forever (should I just use node instead (from the upstart script that is)) ? I thought forever would be a safer runtime for the web proxy?
Comment: Actually, I and many other salty old unix dogs run node apps, each one as a service managed by upstart.  Fewer moving parts is usually safer.  I use upstart to exec the node executable as www-data.  Life is good.
Comment: It sounds like we have similar setups.  I am running nginx as my reverse proxy to (1) a node "virtual reverse proxy" handing of to four different node webapps; (2) a bugzilla instance; and (3) a landing page for lost puppies.  I really enjoy how easy it is to set up vhosts with node.
Comment: Would you mind sending me an email w/ your upstart script? Mine is currently running as root (but I need to switch this before I push the final vhost script to production clearly)
Comment: Sure Toran.  First thing in the morning when I get to the office.
Answer: http-proxy module doesn't change the host header of the request, and that's what connect/express vhost  uses to distinguish virtualhosts.
In this line:
proxy.proxyRequest(req, res, {host: 'localhost', port: port});

you tell the proxy server to proxy the unchanged request to localhost:port.
So what you need to do is to change:
var app = express.createServer();
app.use(express.vhost('localhost', one));
app.listen(8000);

to:
var app = express.createServer();
app.use(express.vhost('www.mysite.com', one));
app.listen(8000);

and it should work.
Alternatively you can simply set the req.headers.host to localhost before proxying.
Comment: Awesome => this has me moving in the right direction now! Thanks for your time and detailed reply!


How to Assess the Fit of Thousands of Distributions?

Question: I have thousands of subjects, for each of whom I have a fitted gamma distribution, with the parameters estimated from each subject's data.  It is easy to look at the distribution for one subject (say qq-plots, etc.) to get an idea of how good is the fit. But how can I do this on a large scale, for all subjects?
Comment: When you say 'test' do you mean 'assess' (as in 'how far from gamma does this seem to be?') or do you intend some hypothesis test (which I'd advise against ... and doubly so if you're doing a bunch of them).
Comment: Are you talking about a single measure over all subjects or one per subject?
Comment: @ChrisC Chi-square testing is the wrong way to go here, not because it cannot be applied, but because it entails binning and does not use all the information in the data (notably neglect of even the order of the bins). It's an embarrassment that there are still texts recommending this approach for examining the fit of continuous distributions.
Comment: @Glen_b, Yes, I mean assess. Thanks for pointing it out.
Comment: Ideally, one per subject, but I can live with a single measure.
Comment: Could you not use a $\chi^2$ goodness of fit test per subject?
Comment: @ChrisC Isn't $\chi^2$ for discrete random variables? Here we have a continuous distribution.
Comment: $\chi^2$ is used for continuous variables too by using binning. You can apply KS-statistics or similar measures, even AIC (assuming you fit distributions with MLE).
Comment: You can't really use KS here, because the parameters for the NULL distribution need to pre-specified in KS. You can generate the NULL distribution using Monte-Carlo, but that's something that I am trying to avoid.
Comment: @NickCox This is very useful information, thank you for letting me know. I had unfortunately read one of those books!
Answer: I'll suggest using representative plots. Pull 16 or 20 subjects, and show their QQ-plots in 4x4 or 4x5 chart. Sometimes, you can plot several subjects in the same plot. This doesn't substitute other ways of representing the fits, but on the other hand I don't think you can avoid this step either. It's used a lot in panel (longitudinal) data analysis. You really need to see the representative plots.
See, Fig.12-1.3 in this book. It's not the distributions, but the same idea: show the sample plots for subjects.
You can get fancy and draw 3d plots, of course, or contour plots, where x-axis is subject, but these are sometimes hard to analyze visually. They may reveal important patterns though.
UPDATE
You can also show the histogram of Kolmogorov-Smirnov statistics. It's true that the critical values are expensive to compute, but the statistics itself is easy to compute. So, you can obtain KS-stat for each subject, and show the histogram of obtained values. This will give you a great visual cue as to how the gamma distribution fits in general. It's almost like bootstrapping.
Comment: Agreed with the idea of sampling: what could be more statistical? You could also (1) look at the distributions of parameter estimates (two each?) and check that any extremes make sense in terms of the original data (2) automate corr(observed quantiles, expected quantiles) and check very poor fits (and extremely good ones!)
Comment: A variant of this idea is to make confidence bands around the empirical distribution function, based on the KS statistic (ecdf.ksCI in package sfsmisc), and then plot the fitted gamma distributions to see if they generally falls within the bands.
Comment: The most practical, and really the best answer, for me is the first comment by @NickCox, but since he didn't put his comment as an answer, and he approves of this answer, I am accepting it as the correct answer.
Comment: @user765195 Thanks for the remark, and that's fine by me.
Comment: @NickCox, I updated the answer with histogram of KS-stats.
Comment: I agree with Rupert Miller on K-S statistics. See p.9 in http://www.stata.com/manuals13/rdiagnosticplots.pdf for a quotation. His context was normality testing, but I think the point carries over here. On K-S testing, regardless of that view, newcomers need the emphasis that the standard test needs modification when parameters are estimated from the data.
Answer: I hope that understood your situation and the question correctly. Considering your data set's number of distributions, visual exploratory approaches (such as QQ plots, which you mentioned) are not feasible in this case. Therefore, you have to resort to analytical approaches, such as goodness-of-fit (GoF) tests, as some have already mentioned in the comments above.
Since you have informed that distribution parameters are estimated from data, I assume that you have used or plan to use one of distribution fitting approaches. One of the most popular fitting approaches (along with least squares, to a lesser degree) is maximum likelihood estimation (MLE), which is generally easy to perform, for example, using function fitdistr() from R package MASS. However, depending on your particular data, fitting via fitdistr() might not be so trivial. Some people prefer R package fitdistrplus, as they consider it more advanced or useful.
After this straightforward step, you need to validate the estimation results, using one or more of the following GoF tests for continuous data (considering their pros and cons): chi-square (via binning), Kolmogorov-Smirnov (via corrected tables for critical values or Monte Carlo simulation, which I'm listing here just for completeness, as you are trying to avoid this), Anderson-Darling, Lilliefors, Cramér–von Mises and Watson. In terms of performance, the problem gets reduced to performing a relatively large number of non-parametric GoF tests, which IMHO is achievable either via doing it on a more powerful hardware (i.e., renting Amazon EC2 virtual instance), or via parallelizing code.
Returning to the essence of your question, my idea of possible approaches is to aggregate results either via bootstrapping (similarly to the one presented in this excellent answer), or some kind of averaging approach, similar to ensemble methods (for example, take a look at this research paper).
Comment: I disagree partly with the emphasis in the first paragraph. What's impractical is looking at **every** QQ plot; that doesn't rule out looking at **some** and automating a search through QQ plot results. @Aksakal's answer illustrates well. Thousands of significance tests too raise problems of multiplicity and of checking wild or puzzling results. You will be aware of that, but it needs emphasis.
Comment: As before, this boils down to emphasis, not disagreement. I think that any automated analysis would, or should, need to be **followed** also by looking at the poorest fits to see what is happening.
Comment: @NickCox: I have nothing against your suggestion of looking at some QQ plots for _initial_ **exploratory** assessment (trends, outliers, etc.), however I don't think that such visual EDA methods can be automated by definition. That is, unless one will use AI/ML to process plots, which is likely much more difficult and resource-intensive that analytical approaches. I agree with you that one needs to be careful with _scaling_ analytical methods, but I think that it still is easier and more accurate to come up with good **strategies** for _analytical solutions_.
Comment: @NickCox: I agree. I like Aksakal's idea of QQ plots sampling and your thoughts on some manual analysis (and maybe intervention), but they just represent exploratory approach. My answer, as you noticed, emphasizes analytical one. Certainly, the optimal (and correct) way of doing things is combining both approaches so that they would "inform" each other and, ultimately, inform decision-maker.


On a particular matrix homomorphism

Question: I recently noticed that, for $2\times 2$ matrices, dividing one off-diagonal element by a constant while multiplying the other off-diagonal element by the same constant produces a matrix homomorphism.
i.e., if we call this operation $g$, then it's defined as
$$g\left(\begin{bmatrix}a & b\\c & d\end{bmatrix}\right) =\begin{bmatrix}a & (b\cdot\lambda)\\(c/\lambda) & d\end{bmatrix} $$
and then the homomorphism properties are:
$$ g(A) g(B) = g(AB) $$
(where juxtaposition indicates the usual matrix multiplication)
and
$$ g(A)+g(B) = g(A + B).$$
My question(s): Does this homomorphism have a name, any useful applications, and does it hint at any kind of deeper structure?  Are there others like it?
Answer: You have
$$
g \left(
\begin{bmatrix}a & b\\c & d\end{bmatrix}
 \right)
=
\begin{bmatrix}\lambda & 0\\0 & 1\end{bmatrix}
\begin{bmatrix}a & b\\c & d\end{bmatrix}
\begin{bmatrix}\lambda & 0\\0 & 1\end{bmatrix}^{-1}
=
\begin{bmatrix}\lambda & 0\\0 & 1\end{bmatrix}
\begin{bmatrix}a & b\\c & d\end{bmatrix}
\begin{bmatrix}\lambda^{-1} & 0\\0 & 1\end{bmatrix}.
$$
In fact, all automorphism of a matrix algebra over a field are inner (that is, of the form $x \mapsto t x t^{-1}$), see Skolem-Noether. See also this article for a more general situation.
Answer: Your homomorphism is an example of a similarity transform. A similarity transform is a transformation on matrices of the form $X\mapsto AXA^{-1}$ where $A$ is invertible. In your case $A = \begin{pmatrix}\lambda^{1/2} & 0 \\ 0 & \lambda^{-1/2}\end{pmatrix}$.

On a particular matrix homomorphism


Solving polynomial to get all coefficients

Question: Given an array of N integers where N can go upto 10^4 and each element can be upto 10^5.
Now i need to find the coefficients of polynomial p that is given as :
<code>(1+a[1]*x)(1+a[2]*x).............(1+a[n]*x)
</code>
Note : as the coefficents can be very big so we need to find them modulo 100003.
Example : Let N=3 and array is [1 2 3].
Then as polynomial is (1+x)(1+2x)(1+3x)
So i need to put all its coefficeints in an array like here 6 11 6 1 as polynomial is 6x^3 + 11x^2 + 6x^2 + 1.
How to do it efficenlty?Please help
Comment: Why modulo $100003$? do you mean $10000^3$?
Comment: @nbubis No its 100003.I had taken this modulo so that it becomes possible to calculate and result dont overflow
Comment: Why? where does that number come from?
Comment: @n Its a prime number.No other reason.
Answer: Exponential solution
Let $b_{n}x^n+...b_{1}x+b_{0}$ be the polynomial in discussion. You first find $b_{n}$(what is it?), then using Vieta's formulas you can easily derive all the coefficients. You can use it since you get all the zeroes to the polynomial(make sure to make the additions modulo $100003$).
Hint(for solution in $O(n^2)$):
Think recursively.
LE:
Solution$O(n^2)$
Start with the polynomial : $1+a_{1}x$, multiply by $1+a_{2}x$, then apply mod 100003 to all of the coefficients of the new polynomial. Continue in this way to get the solution... For the proof I would need to write a lot...
Comment: a[n] is nth array element and can you describe a bit how to use Vieta's formulas?
Comment: @ shooting-squirrel I want to ask how to do with computer not manually.
Comment: I mean i want an algorithm sort of thing
Comment: Moreover i am not having roots of this polynomial
Comment: I think you didnt get the question properly
Comment: But how can i compute product of twos at a time then three and so on.It would be very complex as N increases
Comment: Ok... that's true... it should be easy to get a formula. Think recursively.
Comment: Do you think recursive solution will help as N can be upto 10^4 ?
Comment: Also what should be recursion according to you?
Comment: How is it a hint to think of O(N^2) recursive solution
Comment: Using Vieta's formula would require a minimum of $2^N$ summations.
Comment: @shooting-squirrel Also do you think O(N^2) is most efficient approach here as N is 10^4.So 10^8 solution
Comment: Yes. What number does this project euler problem have?
Answer: I've implemented an $O\left( n^2\right)$ algorithm at
http://jsfiddle.net/xcrqoxk5/
I believe it's the recursive solution Shooting Squirrel is talking about.
Check to see if you understand how it works.

Suppose you want to multiply a polynomial $b_4 x^4 + b_3 x^3 + b_2 x^2 + b_1 x + b_0$ with the polynomial $ax + 1$. By a standard multiplication, we have $$\begin{array}{*{20}{c}}
  {}&{}&{{b_4}{x^4}}&{{b_3}{x^3}}&{{b_2}{x^2}}&{{b_1}x}&{{b_0}} \\
   \times &{}&{}&{}&{}&{ax}&1 \\
\hline
  {}&{}&{{b_4}{x^4}}&{{b_3}{x^3}}&{{b_2}{x^2}}&{{b_1}x}&{{b_0}} \\
   + &{a{b_4}{x^5}}&{a{b_3}{x^4}}&{a{b_2}{x^3}}&{a{b_1}{x^2}}&{a{b_0}x}&{} \\
\hline
  {}&{a{b_4}{x^5}}&{\left( {a{b_3} + {b_4}} \right){x^4}}&{\left( {a{b_2} + {b_3}} \right){x^3}}&{\left( {a{b_1} + {b_2}} \right){x^2}}&{\left( {a{b_0} + {b_1}} \right)x}&{{b_0}}
\end{array}$$
Hence if the coefficients of the input polynomial are stored as $$B =\left[ {\begin{array}{*{20}{c}}
  {{b_0}}&{{b_1}}&{{b_2}}&{{b_3}}&{{b_4}}
\end{array}} \right]$$
then we see the way to get the coefficients of the output polynomial $$C = \left[ {\begin{array}{*{20}{c}}
  {{c_0}}&{{c_1}}&{{c_2}}&{{c_3}}&{{c_4}}&{{c_5}}
\end{array}} \right]$$ is to set $c_0 = b_0$, $c_5 = ab_4$, and $c_i = b_i + ab_{i-1}$ for all other $i$.
To compute the full array of coefficients, all we have to do is recursively apply the above procedure to each element in our set of $a$ coefficients, $X$.
Hence start with a coefficient array $C_0 = \left[1\right]$ (representing the polynomial $1$), and multiply it by the polynomial $1+ax$ using our routine, with $a = X\left[0\right]$, to get our first output coefficient array $C_1$. Then multiply the polynomial represented by this array by $1 + ax$ with $a = X\left[1\right]$ using the routine to get our output coefficient array $C_2$. Do this for every element of $X$ and we'll have computed the coefficients for the complete polynomial.
I've called the routine that does the multiplication "convolveWithOnePlusAx", and the function that computes the complete set of coefficients from $X$ "polyCoefsFromArray".
The last line in the script simply prints them out in reverse order. You can ignore it.
Comment: Can you please prvoide a pseudocode here as i dont work on javascript/jquery
Comment: @user3923257 - I've added a summary of the algorithm above.


Fingering for some arpeggios on piano (Final Fantasy prelude)

Question: So I came across this video and decided to learn this arrangement to improve my technique (of course not as fast as in the video).
I'm not too sure what the best fingering is (for the left hand) despite the video, simply because I can't deduce how exactly he's playing it.
Now, the first part is arpeggios with both hands:

The right hand fingering, if I'm correct, is:

ascending: 2 3 4 1 per quarter note (so thumb on G)
descending: 2 1 4 3 per quarter note (so thumb on G still)

The left hand fingering is unclear, does anyone have a suggestion?
My own idea was:

ascending: 4 2 1 3 per quarter note (???)
descending: 2 3 4 1 per quarter note (so thumb on G)

Also, for the second part: is the left hand always playing with a fingering of 5/2 (even for the first sixteenth, which is an octave) alternated with 4/1?
Answer: I would use:
First passage, right hand, going up: 2341 2341 etc.
First passage, right hand, going down: 2143 2143 etc.
First passage, left hand, going up: 5213 5213 etc.
First passage, left hand, going down: 2312 3412 3412 etc.
Second passage, going up: 15/25/13/24 13/25/13/24 13/25/13/24 etc.
Second passage, going down: same in reverse.
(In contrast to Brian Chandler's answer, I have the upper note first in my notation for the second passage.)
The second passage is obviously more difficult than the first, and cases can be made for a variety of fingerings.  (I think the performer uses Brian's fingering.)  While it may seem a bit counterintuitive to take the larger intervals with closer together fingers (13 on the EC sixth instead of 14 as Brian has it), it is easier (for me) to jump to 24 on the FB fourth from 13 on the EC third than it is to jump to 25, as would be necessary if I were to take the EC sixth with 14.
To a large extent the fingering you use depends on the size and configuration of your hand.  Take into account not just whether you can comfortably take an interval with a particular fingering, but whether you can easily go to the next note with that fingering as well. For example, I have to stretch to hit an octave with 25, so I'm limited as to where I can go next if I do it.  As such, the fingering for the second passage that you ask about is awkward to me (for other reasons as well).  I wouldn't use it.
Answer: The key to playing passages like this fast is to unlearn the "thumb under" technique that you probably started out with. Pay a group of notes, "flip your wrist" to get you hand in position for the next group, repeat till done.
For the first passage, I would play each beat as a group:
RH 1 2 3 4 1 2 3 4 .... 4/5 3 2 1 4/5 3 2 1 (4 or 5 depending how big your hands are)
LH 5 3 2 1 5 3 2 1 .... 1 2 3 4 1 2 3 4 (yes, that really does mean two consecutive notes with the thumb at the top - move your hand between the beats)
For the second passage, the first octave is on its own, and then take pairs of chords
5-2 3-1
5-2 3/4-1
etc
and similarly back down.
But first, watch this:

Note:, I wrote this before looking at the OP's video - and on the basis that "fast" meant about "fast as in the links below". But the same technique works just as well at Allegretto as at Presto.
Answer: No-one can say which is the best fingering for you, but here's what I would do.
First part arpeggios (no apostrophe!)
RH up: 1 2 3 4 : thumb on the 1st note of the beat helps with emphasis; any break as your hand moves comes in a good place. (Your fingering looks very awkward)
RH down: 5/1 4 3 2 - just the reverse of up
LH up: 5 2 1 4 5 2 1 4... last one 1 2 1 2
LH down: 1 2 3 4
Also, for the second part: is the left hand always playing with a fingering of 5/2 (even for the first sixteenth, which is an octave) alternated with 4/1?
I would do
51 52 31 52 41 52 31 ...
but this is really quite hard, and you have to find the smoothest way that fits your fingers.
Comment: Can you clarify "LH up"? Why would you start with the thumb on the lowest note? Also, have you checked the video - not saying that's the only solution, but that RH 2341 doesn't seem very awkward. Apostrophe removed, that's the spelling in my native language.
Comment: Sorry: that was a mistake - start with 5, obviously. I didn't read the question very carefully, and did not realise that I was supposed to be reconstructing someone else's fingering. The one cited is perhaps not too awkward, as you say, but it means four thumb-unders, whereas mine has only three. And I think shaping each group of four into a single hand movement makes it easier to aim for a smooth result. But in the end, as I said at the beginning, everyone has to find the fingering that is best for themselves.


Script exits on drupal_bootstrap_full when run from system cron

Question: I have a script that imports data into custom entities created in a drupal 7 database. It needs to run daily so i've added it to my system crontab (running CentOS 6.7). The problem i am having is that when it's run from cron (the script is executable and there's a symlink to it added in /usr/sbin) it runs but fails when the drupal bootstrap is run.  It works flawlessly when run from my user id, the service account user id, another persons id, and on so i know that the script is written correctly. There is no error reported to the console (grabbing std_out and std_err output in my cron command) and there are no php errors logged.  I am at a loss of where to look for the answers i need to find out why it's failing.  I assume it's something to do with the environment when run from cron but i've confirmed the shell is bash as expected and i'm not using any environment variables that aren't set explicitly within the script.  Where can i look for more answers?
Here's My crontab:
55 16 * * * svc_load cs-import >> /var/log/import-log-`date "+\%Y.\%b.\%d"`.log 2>&1

Here's the relevant lines to load drupal from my script.  I see all the echo's print out except the last "drupal loaded". I even tried an or die() after the bootstrap function but it didn't do anything, there must be a crash of some sort that aborts the script before php can process the die() function.
// Drupal Connectivity
echo "Start Loading Drupal\r\n";
define('DRUPAL_ROOT', dirname($parent_dir)."/httpdocs");
echo "Set Drupal Root\r\n";
require_once (DRUPAL_ROOT.'/includes/bootstrap.inc');
echo "Included Bootstrap.inc\r\n";
drupal_override_server_variables();
echo "Overrided Server Variables\r\n";
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
echo "Drupal Loaded\r\n";

I have PHP logging on, here's the relevant lines from php.ini but the file size never increments when I run this script (it does on other errors).
log_errors = On
log_errors_max_len = 1024
error_log = /var/www/vhosts/dev/logs/php_errors.log

Thank you in advance for any help you can provide!
~Lindsay
Comment: What Drupal version?
Comment: I guess you aren't using Drupal 8, are you?
Comment: i'm using Drupal 7
Comment: Your script works perfectly for me on some sites but not others. It works better on sites that have fewer contrib modules installed, but when it fails there's always an exception for me. I see problems specifically with multiupload_filefield_widget. The only ways you could be getting to the end of this script without seeing "Drupal Loaded" are 1. Something (module/theme/whatever) is causing an explicit `exit()` or `die()` for whatever reason, or 2. It _is_ failing with an exception, but you don't have logging switched on.
Comment: @Clive I updated my question with relevant lines from php.ini, How can i track down which module it would be?
Answer: I found the issue is i have multiple PHP versions installed on my system because my hosting company is running plesk. PHP 5.2 is a base version and it looks like drupal loads something that makes php5.2 fail.  After upgrading my script to point to the proper 5.5 version of php everything is running as expected.  Thanks to all who replied.


What mail storage should I choose for our web application; IMAP, key-valud store, rdbms,

Question: I have to store e-mail messages for use with our application. I have "metadata" for all messages inside a relational database, but I don't feel comfortable keeping message content (gigabytes and terabytes of email data) inside a database. I'm currently using IMAP as a storage, but I have my doubts if I choose correctly. First of all there is a problem of uidvalidity and how to keep a permanent reference to message inside IMAP. Second, I'm not sure if this is the most robust solution in terms of backup/restore strategies, corruption of store, replication ... Positive side is that I can query IMAP using the headers because the data is mostly indexed.
I don't know if key-value stores are a better approach (Casandra, Tokyo cabinet, redis). How they handle storing 1KB and 50MB of data. How they prevent corruption and when corruption or device failure happens how can I repair the store.
Answer: What about using the filesystem? Your metadata in the database could point to the pathname of the actual message (one message per file).
If your message volume is high enough that you're worried about performance based on # of files per directory, you can split things up into a 2-(or more) level directory hierarchy (e.g., by hashing on the messageid).
Using a standard filesystem means you can use all the existing technology for journaling, replication, backups, etc.


TYPO3 - How to customize extension properly

Question: I took one Project which is based on TYPO3 6.2.0
I needed to customize some extensions (templates) and I made this changes directly in the original template files.
I am just wondering what is standard and proper (update safe) way for customizing TYPO3 extensions?
Thanks in advance!
Comment: You also might want to check this: http://stackoverflow.com/documentation/typo3/2196/fluid-templating-in-typo3#t=2017012606054559697
Answer: You can look in the typoscript template of the extension (often setup.ts or setup.txt).
In newer extensions you will find something like
<code>    plugin.tx_bla {
       view {
          templateRootPaths.0 = EXT:bla/Resources/Private/Templates/
          templateRootPaths.1 = {$plugin.tx_bla.view.templateRootPath}
          partialRootPaths.0 = EXT:bla/Resources/Private/Partials/
          partialRootPaths.1 = {$plugin.tx_bla.view.partialRootPath}
          layoutRootPaths.0 = EXT:bla/Resources/Private/Layouts/
          layoutRootPaths.1 = {$plugin.tx_bla.view.layoutRootPath}
</code>
Then copy the templatefiles to your config extension or the fileadmin and adjust the path either in the constants (for the .1) or directly in the setup. After that clear system cache.
In older extension this might look likewise:
<code>    plugin.tx_bla {
       view {
          templateRootPath =
          partialRootPath =
          layoutRootPath =
</code>
Comment: In the first example you have only to copy the template files which you want to change.
Comment: And I forgot, you should enter the typoscript in your own typoscript template, which shoud load after the typoscript template of the extension. Best is never to edit the extension code.
Comment: @Jack70 you can edit your answer if you want to add something. And please don't advice people to put anything layout-related in the fileadmin. It works - nut it's bad practice.
Comment: @Jack70 thanks for the help. I tried to do this on my project but somehow typo3 use always default template.
Here is everything in detail --> http://stackoverflow.com/questions/41874809/typo3-properly-edit-extension-template

Thanks in advance!


Different mysql join query based on user level

Question: I'm seeking an idea or best practice to perform the below:
I have the tables:
profile contains basic info (id, email, pass, lvl)
profile_user contains (pid, name, age, etc..)
profile_mod contains (pid, company_name, money, etc..)
as you can see that I will do left join on two tables to get the whole information, but the join will be all based on the attribute lvl. If lvl=1 it will leftjoin on profile_user and lvl=2 will perform join on profile_mod.
My question is, I don't want to perform a check on every query to get the lvl and then do the leftjoin on the other two tables. I was wondering if there is a best practice for such case.
Answer: You can do:
SELECT    *
FROM      profile a
LEFT JOIN profile_user b ON a.id = b.pid AND a.lvl = 1
LEFT JOIN profile_mod  c ON a.id = c.pid AND a.lvl = 2

If lvl is 1, values for fields in profile_user will populate, and values in profile_mod will be NULL.
Likewise, if lvl is 2, values for fields in profile_user will be NULL, but values in profile_mod will be populated.
Comment: how i can automatically omit the selected attributes rather than having them as NULL values?


Classify the points at 0 and $\infty$ of $x^7~\frac{d^4y}{dx^4}=y'$

Question: $$\displaystyle x^7~\frac{d^4y}{dx^4}=y'$$
I know that 0 is an irregular singular point. At $\infty$, I'm using the change of variable $x = \frac{1}{t}$ and I don't understand how to differentiate and do the substitution.
Comment: If $x=\frac1t$, then $y(x)=y\left(\frac1t\right)$, and by the chain rule $\frac{\mathrm dy}{\mathrm dt}=\frac{\mathrm dy}{\mathrm dx}\frac{\mathrm dx}{\mathrm dt}$, or $\frac{\mathrm dy}{\mathrm dx}=-t^2\frac{\mathrm dy}{\mathrm dt}$. Do this a few more times to find a similar relation between the fourth-order derivatives.
Answer: Use the chain rule
$$y'_x=\frac {dy}{dx}=\frac {dy}{dt}\frac  {dt}{dx}$$
We have that $x=\frac 1t \implies t=\frac 1x \implies \frac {dt}{dx}=-\frac 1 {x^2}=-t^2$
$$\implies y'_x=y'_t\frac {dt}{dx}=-\frac 1 {x^2}y'_t$$
Substitute $\frac 1 {x^2}=t^2$
$$y'_x=-t^2y'_t$$

$$y''_x=\frac d{dt}(-t^2y'_t)\frac {dt}{dx}$$
Since $\frac {dt}{dx}=-t^2$
$$y''_x=-t^2\frac d{dt}(-t^2y'_t)$$
$$y''_x=t^4y''_t+2t^3y'_t$$
Do the same for $y''', y''''$

For y''' I got this
$$y'''=\frac d{dt}(t^4y''_t+2t^3y'_t
)\frac {dt}{dx}$$
Since $\frac {dt}{dx}=-t^2$
$$y_x'''=-t^2\frac d{dt}(t^4y_t''+2t^3y'_t
)$$
$$y_x'''=-t^2(t^4y_t'''+4t^3y''_t+2t^3y''_t+6t^2y_t'
)$$
Finally
$$y'''_x=-(t^6y_t'''+6t^5y_t''+6t^4y_t'
)$$
Comment: Can I ask you to verify y'''? I got $y'''_x = t^6 y'''_t - 6t^5 y''_t - 12t^4 y'_t$
Comment: Be carefull because all the terms should be negative since you multiply by $-t^2$ @p3ngu1n


Server 2008 Likes to restart itself

Question: I  have a weird issue here.
I notice about once a week the web server restarts itself. This would be only a minor issue if we were not planning on implementing an IP failover.
I have checked the event logs. I don't see anything that indicates a reason for the restart.
I need some help diagnosing the reason the server restarts. It happened last night at 5:00AM
Last even in the log was 1 hour before the unexpected shutdown.
Here is the Log for the shutdown event. Any help is much appreciated. I know there isn't much to go on yet.
<code>    Log Name:      System

Source:        EventLog

Date:          5/5/2010 5:01:12 AM

Event ID:      6008

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      SERVERNAME

Description:

The previous system shutdown at 4:56:41 AM on 5/5/2010 was unexpected.

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="EventLog" />

    <EventID Qualifiers="32768">6008</EventID>

    <Level>2</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2010-05-05T09:01:12.000Z" />

    <EventRecordID>346094</EventRecordID>

    <Channel>System</Channel>

    <Computer>SERVERNAME</Computer>

    <Security />

  </System>

  <EventData>

    <Data>4:56:41 AM</Data>

    <Data>5/5/2010</Data>

    <Data>

    </Data>

    <Data>

    </Data>

    <Data>39594</Data>

    <Data>

    </Data>

    <Data>

    </Data>
    <Binary>DA070500030005000400380029008E03DA070500030005000800380029008E033C0000003C000000000
000000000000000000000000000000100000000000000</Binary>

  </EventData>

</Event>
</code>
Comment: Most crashes are driver problems. Are all your drivers the latest available from the manufacturer?
Comment: Interesting becuase I do get some failed to start errors for HP hardware. Like fans and other stuff. Did not think they were that important.
I can post them If you think it may help.

We have an HP DL380 G6
Comment: it could be a driver issue it could be an ASR- either way the message is telling  you that whatever happend, nobody told windows it was going to happen and no memory dump was created
Comment: Huh....I also have an HP DL380 G6 running ESXi that seems to go through random shutdowns....Haven't been able to trace what the problem is.
Comment: @Holocryptic How long has that been happening for? What applications do you run on it?
Comment: @Campo It's been sporadic for a couple months now.  It's not a regular occurrence, and couldn't rule out power failure (although it's hooked up to a UPS and is in the same data closet as my other server which doesn't have a problem).  It wasn't really a big deal before because we didn't have critical systems on there until recently.  We're running ESXi 4 Update 1.
Answer: Just found this.  Seems it could be hardware/firmware related.
Comment: Excellent! I do not have event ID 57 so I am going to say it is a bad board? I can't seem to find the site that says which batch have issues.
Comment: @Campo We performed the firmware update, and it still had the problem.  HP came out to replace the MB.  So we'll see what happens next.  Interestingly enough, when we called HP they the techsupport guys weren't aware of the problem.  We had to point them to their own KB.  Funny stuff.
Comment: WEIRD! That sux. We applied and its all fixed! What was your board revision. We were a couple letters past the bad board but the serial number was still too old for the proper firmware.
Comment: I upgraded on Saturday all looks good so far! Thanks! How did it go for you?
Answer: I FOUND THIS! AHHHH!
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01955503
Comment: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=3884083&prodTypeId=15351&prodSeriesId=3884082&swLang=13&taskId=135&swEnvOID=4040
Comment: Nice.  Well, off to update I go....
Comment: @Holocryptic When did you buy the machine? Which updates are you going to do? Just ILO and the Logic Device? Or are there others? Are you going to do it online or offline? I am going tonight to check the revision of our board. You should probably do the same if you don't know it offhand.
Comment: It was bought late last year (not sure exactly).  I'm downloading the offline updates (online makes me nervous for some reason).  I'm getting the package from the link you provided, plus the Smart Update Firmware 9 DVD.  I figure I'll make a clean go of it.  I know for fact that my firmware is a little behind, but I will be verifying the revision as well.
Comment: Awesome! Yah just check em all and update the ones needed. Did you do the upgrade yet? I am going tonight to check the server.
Answer: I spoke to HP:
They said to update the firmware and it will resolve the issue. I do not use ILO so I will be running it offline.
The file and readme are located here.
I was told if this does not fix the problem they will be sending me a new Mother Board.
I really hope this fixes it!
But as of now this is the official solution from HP.

Server 2008 Likes to restart itself


Trying to integrate Rate-Widget (Rate-Module) into Status-Block (Statuses/Facebook Style Status)

Question: With Drupal 7, I am trying to combine the Rate-module and the statuses-module (formerly Facebook Style Status), for my voting widget to show up in status boxes.
At https://drupal.org/node/811742 it explains to implement the hook hook_facebook_status_render_components.
<?php
/**
 * Implementation of hook_facebook_status_render_components().
 */
function MYMODULE_facebook_status_render_components($status) {
  //Only show on individual status pages.
  if (arg(0) == 'statuses' && is_numeric(arg(1))) {
    //Here, the theme function MYMODULE_voting_widget is assumed to take a "type" and "id" parameter, but it may vary.
    return array(
      'MYMODULE' => array(
        'value' => theme('MYMODULE_voting_widget', 'facebook_status', $status->sid),
      ),
    );
  }
}
?>

The README from the rate-module however describes to use the following code:
<?php
if (arg(0) == 'node' && is_numeric(arg(1)) && ($node = node_load(arg(1)))) {
  print rate_embed($node, 'NAME');
}
?>

So I used this latter code and implemented the hook in the rate-module:
<?php
function rate_facebook_status_render_components($status) {
  //Only show on individual status pages.
  if (arg(0) == 'statuses' && is_numeric(arg(1)) && ($node = node_load(arg(1))) ) {
    //Here, the theme function MYMODULE_voting_widget is assumed to take a "type" and "id" parameter, but it may vary.
    print rate_embed($node, 'ratingwidget');
  }
}
?>

I deactivated the module and re-activated it, but nothing shows up.
What did I do wrong? Do I need to include any files from the Statuses-module in my rate.info-file, in order to implement hooks from one module in the other? Did I do something totally wrong and just don't understand the concepts?
Thanks very much in advance.
Answer: A couple problems here.
I assume you are using Drupal 7, and therefor this module:
https://drupal.org/project/statuses
Which is the successor of facebook_status module for D6.
Your hook is for the D6 module, and does nothing for the D7 module.
I couldn't find the replacement hook in the new D7 version:
http://drupalcode.org/project/statuses.git/blob/bc7e520:/api.php
The other problem, is that 'statuses' are their own 'entity' - for lack of a better word.  They are not actually even 'entities' in the Drupal terminology.  Just some entries in it's own DB table.
The reason this is a problem, is your code is looking to load a node from path when on a path like (/statuses/5), but a 'status' is not a node.
As far as a solution, you'll have to decide a new road to go down, but at least this helps you understand the situation.


Why is regularization interpreted as a gaussian prior on my weights?

Question: Going through this online class, the notes specify the following. (In the highlights):

I understand how we get a maximum likelihood interpretation. What I do not get is:
Why does using the Frobenious norm of the weight matrix $R(W)$ as a regularizer, lead to the interpretation of the weight matrix $W$ having a prior as being sampled from a gaussian? How is this arrived at? And what if the regularization term was something other than the frobenius norm?
(And to that last point, does this mean that every element in the matrix $W$ is sampled from a gaussian?)
Thank you.
Answer: Since we're using MAP we are trying to maximize the probability of the parameters given the data.
$$
   P(W|x,y) = \frac {P(x,y|W) P(W)} {P(x,y)}
$$
$P(x,y)$ can be ignored since it's fixed for our data. So we are trying to maximize $log(P(x,y|W)) + log(P(W))$.  Let's look at $P(W)$.
If each element of $W$ is drawn independently from a unit Gaussian, the probability of the matrix is
$$
P(W) = \frac {1} {\sqrt{2 \pi}} \prod_{ij} exp\Big( -\frac {w_{ij}^2} 2 \Big)
$$
so $log(P(W))$ is
$$
-\frac 1 2 \sum_{ij}{w_{ij}}^2
$$
plus some constant terms. The log-likelihood is then
$$
  log(P(x,y|W)) - \frac 1 2 \sum_{ij} {{w_{ij}}^2}
$$
Since we are thinking in terms of losses we negate the log-likelihood and minimize. Now the first term is cross-entropy and the second term is $\frac 1 2 R(W)$.
If you have a (reasonable in some sense that I don't know how to define) more or less arbitrary penalty function you can obtain a density for it by integrating over its domain and then normalizing to obtain a density. In this case it's just easy to see that it's a Gaussian.
This does not mean that the elements of $W$ actually are sampled from a Gaussian. What it means is that you believe that's what $W$ looks like before you have any evidence to the contrary. In other words the prior on the elements of $W$ is a Gaussian.
Comment: Thanks testuser, however I think you lost me on the last part(s). I guess what I am trying to understand is, if we think that a matrix $W$ "looks" like a gaussian... what does that mean? Doesn't that mean that its elements are all iid sampled from a gaussian? Secondly, I understand the formulas you put up, but how can we tie them directly to the cost function in this particular case? Expansion on this would be appreciated. Thanks!
Comment: If you ask the person running an experiment, before they've seen any data, "What's your best guess for $W$"? they'll say: "I think each element was sampled independently from a Gaussian with mean zero and variance 1".
Comment: I think it's a little clearer - Ill digest it tonight and let you know if I have any remaining questions. Thanks again.
Comment: I think I get it testuser. One question though is - why WOULD we ever assume that the weights come from a gaussian, (in this machine learning context)? Why is that a thing to start off with? By what right do we make this assumption? Thanks.
Comment: In this case I don't think that anyone really thinks that the weights are generated this way. It's just something that people know restricts the set of fitted models and has been observed to lead to less overfitting.
Answer: Consider a simple linear regression model for this explanation. The goal is to predict a target variable $y$ using a set of features $x$ and a set of weights $\textbf{w}$.
Without Regularization, the objective in standard linear regression is to minimize the Mean Squared Error (MSE):
$$MSE = \frac{1}{N} \sum_{i=1}^{N} (y_i − \textbf{w}^{T} \textbf{x}_{i})^{2}$$
Here, $y_i$ is the observed target value, $x_i$ is the feature vector for the $i$-th sample, $\textbf{w}$ is the weight vector, and $N$ is the number of samples.
When we add L2 regularization, we modify the objective to include a penalty on the size of the weights. The loss function becomes:
$$ L = MSE + \lambda \sum_{j=1}^{M} w_j^2 $$

where $\lambda$ is the regularization strength and $M$ is the number of weights.
In a Bayesian setting, we start with a prior belief about the distribution of the parameters (weights) and update this belief based on the observed data.
A Gaussian prior on the weights implies that we believe, before seeing any data, that the weights are likely to be small. Mathematically, this is expressed as:
$$ \textbf{w} ∼ N(0,σ)$$
This means each weight $w_j$ is drawn from a Gaussian distribution with mean 0 and variance $\sigma^2$.
The logarithm of the Gaussian prior probability for the weights is:
$$ logP(w)= −\frac{1}{2σ^2} \sum_{j=1}^M w_j^2 + constant $$
Notice that this is a sum of the squares of the weights, similar to the L2 penalty term.
In Bayesian inference, MAP estimation involves finding the weights that maximize the posterior probability, given the observed data. The posterior is proportional to the likelihood times the prior:
$$ P(w∣Data)∝P(Data∣w)⋅P(w) $$
Taking the logarithm (which preserves the location of the maximum):
$$ logP(w∣Data)=logP(Data∣w)+logP(w) $$
For our linear regression model, maximizing $logP(w∣Data)$ involves minimizing the negative of this expression, which includes the MSE (as the negative log-likelihood of the data given the weights under Gaussian noise assumption) and the log of the Gaussian prior.
Finally, the L2 regularization term
$ λ \sum_{j=1}^{M} w_j^2 $ in the loss function is equivalent to the negative log of the Gaussian prior $− \frac{1}{2σ^2} \sum_{j=1}^{M} w_j^2$ in the MAP formulation, up to a constant factor.
If we set the regularization parameter $λ$ equal to $\frac{1}{2σ^2}$, then minimizing the L2 regularized loss function is the same as maximizing the posterior probability under a Gaussian prior.


Create filtered and interactive boxplot in R Shiny

Question: This is the code I have so far
library(shiny)
NJData <- data.frame("Year" = c(2000, 2000, 2001, 2001, 2002, 2002, 2003, 2003),
"WUCode" = c("IR", "PS", "IR", "PS","IR", "PS","IR", "PS"), Annual.Value" = c(12, 14,
19, 7, 11, 13, 20, 17))

ui <- fluidPage(
  titlePanel("Subsetting Dataset"),
  sidebarLayout(
    sidebarPanel(
      selectInput("codeInput1", label = "Choose Year", choices = unique(NJData$Year)),
      selectInput("codeInput2", label = "Choose WU Type", choices = unique(NJData$WUCode))
     ),
    mainPanel(
      tableOutput("view")
    )
  )
)

server <- function(input, output) {

  dataset <- reactive({
    return(subset(NJData, (Year == input$codeInput1 & Water.Use.Subtype.Code ==
input$codeInput2)))
  })

   #output$view <- renderTable(dataset())

 output$plot <- renderPlot({
   p<-ggplot(dataset(),
          aes_string(
            x       = dataset()$Year,
            y       = dataset()$Annual.Value,
            fill    = dataset()$Year # let type determine plotting
          )
    )
  })
}

shinyApp(ui = ui, server = server)

I want to allow the user to choose what year and what WU type to use and then generate a boxplot of the Annual Value for that year and WU type, but I can't get the boxplot to generate.
Any help is greatly appreciated!
Answer: You have to add the type of chart you would like to use in ggplot2, in your case I guess that would be:
ggplot(...) + geom_boxplot()

Here is a refined version of your code:
library(shiny)
library(ggplot2)

NJData <- data.frame(
  "Year" = c(2000, 2000, 2001, 2001, 2002, 2002, 2003, 2003),
  "WUCode" = c("IR", "PS", "IR", "PS","IR", "PS","IR", "PS"),
  "Annual.Value" = c(12, 14, 19, 7, 11, 13, 20, 17)
  )

ui <- fluidPage(
  titlePanel("Subsetting Dataset"),
  sidebarLayout(
    sidebarPanel(
      selectInput("codeInput1", label = "Choose Year", choices = unique(NJData$Year)),
      selectInput("codeInput2", label = "Choose WU Type", choices = unique(NJData$WUCode))
     ),
    mainPanel(
      plotOutput("plot")
    )
  )
)

server <- function(input, output) {

  dataset <- reactive({
    subset(
      NJData,
      (Year == input$codeInput1 & WUCode == input$codeInput2)
    )
  })

  output$plot <- renderPlot({
    ggplot(
      dataset(),
      aes_string(
        x = "Year",
        y = "Annual.Value",
        fill = "Year"
      )
    ) + geom_boxplot()
  })

}

shinyApp(ui = ui, server = server)


Python Recursion - Swap words in a sentence via dictionary index, get all permutations

Question: I am trying to create a recursive function which takes a sentence, iterates through each word, compares the words to a dictionary, and replaces the word if they match within the dictionary. In the end, it should return every permutation possible.
For the sentence example: "For now it is never to late" should return the following:
{"For now it is never to late", "4 now it is never to late", "4 now it is never 2 late", "4 now it is never 2 l8", "4 now it is never to l8", "for now it is never 2 late", "for now it is never 2 l8", "for now it is never to l8"}
I currently have the following snippet of code, which works for the first loop through and returns ['4 now it is never to late', '4 now it is never 2 late', '4 now it is never 2 l8'], but I cannot seem to figure out a method for getting the remaining permutations.
wordChangeDict = {'for':'4',
                    'to': '2',
                    'late': 'l8'}

def phrase2Slang(phrase, resultStrings, index):
    if index > len(phrase)-1:
        return resultStrings
    else:
        word = phrase[index]
        if word in wordChangeDict:
            phrase[index] = wordChangeDict[word]
            resultStrings.append(' '.join(phrase))
        return phrase2Slang(phrase, resultStrings, index+1)

testPhrase = "for now it is never to late"
phraseArray = testPhrase.split()
for idx, word in enumerate(phraseArray):
    modifiedPhrases = []
    modifiedPhrases = phrase2Slang(phraseArray, modifiedPhrases, idx)
    print modifiedPhrases

Update (Answer)
Below, Arief pointed out that my initial list was mutable, causing the blank results. After fixing this, I was able to use one of my original solutions of a multiple recursion loop (dropping the for loop which led into the recursive calls as well):
import copy
wordChangeDict = {'for':'4',
                  'to': '2',
                  'late': 'l8'}

def phrase2Slang(phrase, resultStrings, index):
    phrase = copy.deepcopy(phrase)
    if index == -1:
        return resultStrings
    else:
        word = phrase[index]
        if word in wordChangeDict:
            phrase2 = copy.deepcopy(phrase)
            phrase2[index] = wordChangeDict[word]
            resultStrings.append(' '.join(phrase2))
            phrase2Slang(phrase2, resultStrings, index-1)
        return phrase2Slang(phrase, resultStrings, index-1)

testPhrase = "for now it is never to late"
phraseArray = testPhrase.split()
modifiedPhrases = []
modifiedPhrases = phrase2Slang(phraseArray, modifiedPhrases, len(phraseArray)-1)
print modifiedPhrases

Answer: Quit doing this recursively: Python's itertools package has already implemented this for you.  You have a product to generate with a sequence of lists.  The choice points have two elements each; the fixed parts of the sentence have only one.  Your items are:
[
    ["for", "4"],
    ["now it is never"],
    ["to", "2"],
    ["late", "l8"]
]

Can you take it from there?
P.S. You misspelled "too".
Comment: The purpose of this function was to take unknown strings and map them against a much larger dictionary of possibilities. Specifically, I will be inputting lists of common English idioms and converting them with 'slang' to obtain all possible combinations. Using Python's itertools package, would the build out method be to take an input phrase, parse it out word by word, dynamically build a mapping (similar to what you built above), and then run it through itertools?
Answer: acelives, I may contribute a bit.
Your code will return some of the permutations at first iteration, but empty lists for the next iterations. This is because your phraseArray is a list, and list is mutable, hence it is updated since the first iteration.

To avoid this, use copy.deepcopy :
wordChangeDict = {'for':'4',
                    'to': '2',
                    'late': 'l8'}

import copy

def phrase2Slang(phrase, resultStrings, index):
    phrase = copy.deepcopy(phrase)
    if index > len(phrase)-1:
        return resultStrings
    else:
        word = phrase[index]
        if word in wordChangeDict:
            phrase[index] = wordChangeDict[word]
            resultStrings.append(' '.join(phrase))
        return phrase2Slang(phrase, resultStrings, index+1)

testPhrase = "for now it is never to late"
phraseArray = testPhrase.split()
for idx, word in enumerate(phraseArray):
    modifiedPhrases = []
    modifiedPhrases = phrase2Slang(phraseArray, modifiedPhrases, idx)
    print modifiedPhrases

But notice that it is still does not give all the permutations you wanted.

To fix this, i can create a permutations of the Dict stored in a list, like this way :
wordChangeDict = {'for':'4',
                    'to': '2',
                    'late': 'l8'}
n = len(wordChangeDict)

import itertools
import copy

permDict_inList = []

for i in range(1,n+1):
    for j in itertools.combinations(wordChangeDict, i):
        permDict_inList.append({k : wordChangeDict[k] for k in j})

Then use that list as a filter, iterate through that list instead.

Result :
['for now it is never 2 l8', '4 now it is never to late', '4 now it is never to l8', '4 now it is never 2 l8', '4 now it is never 2 late', 'for now it is never 2 late', 'for now it is never to l8']

Full code :
wordChangeDict = {'for':'4',
                    'to': '2',
                    'late': 'l8'}
n = len(wordChangeDict)

import itertools
import copy

permDict_inList = []

for i in range(1,n+1):
    for j in itertools.combinations(wordChangeDict, i):
        permDict_inList.append({k : wordChangeDict[k] for k in j})

def phrase2Slang(phrase, resultStrings, index, FILTER):
    phrase = copy.deepcopy(phrase)
    if index > len(phrase)-1:
        return resultStrings
    else:
        word = phrase[index]
        if word in FILTER:
            phrase[index] = wordChangeDict[word]
            resultStrings.append(' '.join(phrase))
        return phrase2Slang(phrase, resultStrings, index+1, FILTER)

testPhrase = "for now it is never to late"
phraseArray = testPhrase.split()

modifiedPhrases = []
for PERM in permDict_inList:
    idx = 0
    modifiedPhrases = phrase2Slang(phraseArray, modifiedPhrases, idx, PERM)
    print list(set(modifiedPhrases))

result = list(set(modifiedPhrases))

Comment: Arief, thank you! I was originally working on a multi-recursive loop which didn't use the original for-loop, but I kept having issues, and it was because like you said - the list was mutable. I went back to that solution and was able to get everything completed inside by standard recursive loop. Your solution works as well, so thanks!


Prove that $S(x) = \sum\limits_{n=1}^\infty \frac{\sin nx}{n\sqrt n}$ is convergent and can be differentiated on $x\in(0, 2\pi)$

Question: my task is to prove that $$S(x) = \sum\limits_{n=1}^\infty \frac{\sin nx}{n\sqrt n}$$ is convergent and is continuous on $(0, 2\pi)$.
I did the following:
$$S(x) = \int\limits_{n=1}^\infty \frac{\sin nx}{n\sqrt n}\mathrm{d}x =\frac{\cos(nx)}{\sqrt{n}}
$$
Applying Dirichlet Theorem, observe two series:
$$a_n = \cos(nx), \quad b_n=\frac{1}{\sqrt{n}}$$ $a_n$ is bounded, $b_n$ approaches zero, therefore the initial series is convergent.
Is it the right way to prove the convergence? And how shall I prove that it is continuous and differentiated on the given interval?
Comment: Couldn't you just note that $\lvert \sin x \rvert \le 1$ and then apply the p test to the resulting expression to show convergence? I would have thought continuity comes from the fact that $\sin nx/(n \sqrt{n})$ is the quotient of continuous functions for $n \in \mathbb{N}$ and $x \in \mathbb{R}$ and $S$ is then the sum of continuous functions.
Comment: @mattos ok thx, I'll think about it. But still, is my reasoning towards convergency correct?
Answer: It's unclear what you mean by your line "$S(x)=\int_{n=1}^\infty\ldots$".
Note that
$$\left|{\sin(nx)\over n \sqrt{n}}\right|\leq{1\over n^{3/2}}\ .$$
As ${3\over2}>1$ the given series is a uniformly convergent series of continuous functions. Therefore $x\mapsto S(x)$ is a continuous $2\pi$-periodic function.
With differentiation it is different.The derived series
$$\sum_{n=1}^\infty{\cos(nx)\over\sqrt{n}}$$
is divergent at $x=0$ (since ${1\over2}<1$), but is convergent at all $x\in\ ]0,2\pi[\ $. To show the latter you need Abel's theorem. Looking at a proof of this theorem you can verify that the derived series is uniformly convergent on every interval $[\delta,\>2\pi-\delta]$, $\delta>0$. This does   guarantee that the derived series represents $S'(x)$ for all $x\in\ ]0,2\pi[\ $. The following is a plot of $S(x)$; it corroborates that $S(x)$ is differentiable at all $x\in\ ]0,2\pi[\ $.
Comment: The thing is, my professor asks for an analytical proof in addition to a graph, which I also have. ![pic](https://sun9-36.userapi.com/c858016/v858016777/f235b/PVbZnEm1PD0.jpg)
Answer: $S(x)$ is not differentiable at the origin: let $x=\frac{1}{N}$ for some large $N$. We have
$$ \sum_{n=1}^{2N}\frac{\sin(nx)}{n\sqrt{n}}\geq \frac{4}{5N}\sum_{n=1}^{2N}\frac{1}{\sqrt{n}}\geq \sqrt{\frac2N} $$
while the partial sums of $\sin(n x)$ are bounded by $\frac{1}{2}\cot\frac{x}{4}\leq 2N$, so by summation by parts
$$ \left|\sum_{n>2N}\frac{\sin(nx)}{n\sqrt{n}}\right|\leq 2N\sum_{n>2N}\left(\frac{1}{n\sqrt{n}}-\frac{1}{(n+1)\sqrt{n+1}}\right)\leq \sqrt{\frac1N} $$
and $\frac{S(x)}{x}$ is unbounded as $x\to 0^+$. This implies that $S$ is not differentiable at the origin.


Drupal\Component\Plugin\Exception\PluginNotFoundException: The "commerce_order_item" entity type does not exist

Question: I installed the Commerce core module installed without using Composer.
I tried to update from Drupal 8.4.0 to Drupal 8.4.1 using Drush, but I am getting the following error.

Drupal\Component\Plugin\Exception\PluginNotFoundException: The "commerce_order_item" entity type does not exist. in Drupal\Core\Entity\EntityTypeManager->getDefinition() (line 133 of core/lib/Drupal/Core/Entity/EntityTypeManager.php).

I cannot access update.php, run drush cr or drush updb.
How can I update Drupal without getting that exception?
Comment: Do you have a backup?  Just revert to the backup and troubleshoot the upgrade in a local dev environment.  If you don't have a backup, you have to ask yourself why you are doing updates on production without a backup.
Comment: This is a generic Drupal error message that is being reported by multiple people for multiple entity types recently. Some people have told me that truncating their cache tables fixed the problem. Worth researching, in any case.
Answer: Hm, as I can see you try to update site directly on production version?
In next steps i can describe process that i use while perform site and plugins update, so after all they may be helpful for you:

First rule, never perform updates directly on production site.
Second, start practicing virtualization for projects (docker, vagrant) for local development.
Third, start practicing Drush configuration management for synchronization between local, beta, production sites...
And most important, always use GIT.

Because your site is already broken, I can suggest you to make local version with previously working configuration and database, so after this you can perform updates on your local site and debug problems if they still exists.
For example, on local version you can make clean install of Drupal 8.40, and import untouched database from state where site work and all site data with modules (same versions used on site before they going down). This will provide you site before update to Drupal 8.41, but for future updates I need to suggest you to start using composer, because lot of dependencies for site lying in the vendor directory.
Answer: Your error shows that you updated Commerce without updating Entity API.
Did you run "composer update drupal/commerce --with-dependencies"?
Install drupal commerce using composer with all its dependencies.
You can use below command for that -
composer require "drupal/commerce"
and then run composer update .
If still any packages are missing in your setup, then you can search for the respective package at - https://packagist.org/ and then install that using composer.


How to parse json in android

Question: https://api.mojilala.com/v1/stickers/trending?api_key=dc6zaTOxFJmzC
data class RootApi(val data : List<AllData>)
data class AllData(val images : ImageSticker)
data class ImageSticker(val fixed_height : ActualImage)
data class ActualImage(val url : String)

is this the correct POJO class or not according to given api
Comment: Don't expose key publically it can be misused.
Comment: it's Public API :)
Comment: API is public but are you sure about key?
Comment: https://github.com/mojilala/sticker-api   yes its public
Answer: Firstly, hide your API key in the question. Coming to generating data classes of JSON. I would suggest you use the following Plugin
https://plugins.jetbrains.com/plugin/9960-json-to-kotlin-class-jsontokotlinclass-
And no that is not the correct data class. Infact it is very big. I can't even paste it all in the answer. Use the plugin. Paste the JSON response and generate yourself.
Comment: it's a public API and free also :)
Answer: Following the answer from @che10 you should end up with the below:
data class Data(
    // Assuming you are using Moshi https://github.com/square/moshi
    // Use @SerializedName("data") if you're using Gson
    @Json(name="data")
    val stickers: List<Sticker>,
    val meta: Meta,
    val pagination: Pagination
)

data class Sticker(
    val designer: String,
    val id: String,
    val images: Images,
    val is_animated: Boolean,
    val rating: String,
    val tags: List<String>,
    val url: String
)

data class Pagination(
    val offset: String,
    val total: Int,
    val total_count: Int
)

data class Meta(
    val msg: String,
    val status: Int
)

// Different fields but of two types so we can reuse the classes
// Difference between AnimatedImage and StillImage is the two webp fields
data class Images(
    // Animated images
    val fixed_height: AnimatedImage,
    val fixed_height_downsampled: AnimatedImage,
    val fixed_height_medium: AnimatedImage,
    val fixed_height_medium_downsampled: AnimatedImage,
    val fixed_height_small: AnimatedImage,
    val fixed_width: AnimatedImage,
    val fixed_width_downsampled: AnimatedImage,
    val fixed_width_medium: AnimatedImage,
    val fixed_width_medium_downsampled: AnimatedImage,
    val fixed_width_small: AnimatedImage,
    // Still images
    val fixed_height_still: StillImage,
    val fixed_height_medium_still: StillImage,
    val fixed_height_small_still: StillImage,
    val fixed_width_still: StillImage,
    val fixed_width_medium_still: StillImage,
    val fixed_width_small_still: StillImage,
)

data class StillImage(
    val url: String,
    val size: Int,
    val height: Int,
    val width: Int,
)

data class AnimatedImage(
    val url: String,
    val size: Int,
    val height: Int,
    val width: Int,
    val webp: String,
    val webp_size: Int
)

How to parse json in android


How to set event start and end time dynamically when allDay is set to false with fullCalendar?

Question: I am using a fullcalendar application where upon clicking on an event it opens a     modal with event information. In this modal, it is possible to change event data too, think of event.start, event.end, and event.allDay.
When the allDay checkbox is clicked, the event is automatically updated to represent the new state, however, when the event has been a non-allDay event before. The event data contains two extra properties: oldStart and oldEnd. These two properties I want to be used when the event.allDay is set to false; the event.start should then be set to oldStart and event.end should be set to oldEnd.
The problem is that with this method, when updating the start and end time of an event after setting event.allDay to false, the event when updated disappears from the calendar view and is being updated in the back-end to represent the start time as 00:00 and end time as the event start + defaultEventDuration (which in my case is 02:00:00).
My code looks like this:
function changeAllDay(id) { // Function to change allDay property of event that was clicked on

    // Find corresponding event
    var event = findEvent(id);

    event.allDay = !event.allDay; // Change allDay value

    // Set start and end time to midnight if event is allDay
    if (event.allDay) {

        event.oldStart = event.start;
        event.oldEnd = event.end;

        event.start.set('hours', 00);
        event.start.set('minutes', 00);
        event.end.set('hours', 00);
        event.end.set('minutes', 00);

    } else {

        event.start = event.oldStart;
        event.end = event.oldEnd;
    };

    updateEvent(event);
}

The updateEvent() function does an API call to update the event in the back-end, and when succesful the event is updated with
$('#calendar').fullCalendar('updateEvent', event);

Can anybody tell me what I am missing here?
Comment: can you give us an example please with some real event data for an event where this problem occurs? Also what does "findEvent" do? Does it get the current event from fullCalendar itself (e.g. using the "clientEvents" method)? And have you watched your browser's network tools and/or set JS breakpoints to examine the values of the variables at the point the request is submitted to the server, to see what the start/end values are at that moment? i.e. to prove it's not the back-end's fault?
Answer: Make a copy of your dates when stashing them.  You are calling set right after, and that wipes out your original times.
    event.oldStart = event.start.clone();
    event.oldEnd = event.end.clone();

When you switch back from allDay, your events have no duration.
Comment: Thank you! Creating a copy solved this issue for me :)


Filtro de cadastros por data

Question: Meu código não está funcionando o que pode estar errado?
No banco de dados a data está com Data/Horário como datetime, segue o código:
<?php
    $query = $mysql->query("SELECT * FROM c_clientes WHERE cliente_data_cadastro
               BETWEEN '2018-04-07 12:53:14' AND '2018-05-07 12:53:14'");

    echo "". mysqli_num_rows($query). " Registros";

    while($row = $query->fetch_array())
    {
        echo $row['cliente_data_cadastro']." - "
             .$row['cliente_nome']." "
             .$row['cliente_sobrenome']."
";

    }
?>

Comment: Mas o que não esta funcionando? E qual é o objetivo do código?
Comment: Filtrar os cadastros dentro de 1 mês! Não carrega nada
Comment: Minha pergunta é óbvia, mas você já conferiu se realmente há registros existentes dentro desta faixa de datas?
Comment: Sim tenho cerca de 300 registros por dia! Fora que fiz testes com DB apenas com datas para teste!
Comment: Experimente `SELECT * FROM c_clientes WHERE ( cliente_data_cadastro
               BETWEEN '2018-04-07 12:53:14' AND '2018-05-07 12:53:14')`
Comment: Se o tipo de coluna é DATETIME, que armazena valores de datas e horário no formato YYYY-MM-DD hh:mm:ss, porque retirando as horas funcionou? Eu rodei seu código em uma tabela do meu banco e rodou direitinho. A real solução não me parece passar pela retirada das horas! Algum equivoco da sua parte no banco com relação a coluna das datas. Veja esse resultado  https://i.stack.imgur.com/zTUfU.png
Answer: Se não tem necessidade de comparar as horas, use a função TRUNC:
<?php
    $query = $mysql->query("SELECT * FROM c_clientes WHERE trunc(cliente_data_cadastro)
               BETWEEN '2018-04-07' AND '2018-05-07'");

    echo "". mysqli_num_rows($query). " Registros";

    while($row = $query->fetch_array())
    {
        echo $row['cliente_data_cadastro']." - "
             .$row['cliente_nome']." "
             .$row['cliente_sobrenome']."
";

    }
?>

Comment: Não funcionou, continua não aparecendo nada!
Comment: como está o formato da data no banco? ANO-MES-DIA ?
Comment: Banco de dados: cliente_data_cadastro datetime. http://prntscr.com/jeulpj
Comment: Alterei a resposta, sem a hora. Tenta agora.
Comment: Funcionou! Mas tirei o trunc!
Comment: Ok! Se tua intenção não é saber a hora exata, passa só a data. Mas se for receber uma variavel para essa consulta deixa o trunc, é mais garantido que vá passar a data sem horas.
Comment: Veja a edição que fiz, passei o trunc para o campo do where.
Comment: Se o tipo de coluna é DATETIME, que armazena valores de datas e horário no formato `YYYY-MM-DD hh:mm:ss`,  porque retirar as horas?
Comment: Veja esse resultado https://i.stack.imgur.com/zTUfU.png

Filtro de cadastros por data


Recovering deleted messages on Web host's mail server?

Question: I apologize if this isn't the right place to ask, but due to an automatic list import malfunction, some of my client's email address settings were misconfigured in his Web host's control panel, and an option to delete all messages on the server was checked while no forwarding addresses were specified. The Web host tells me because this option instantly deletes all messages, there is no trace on the server and no way to recover any of the messages. This is very problematic as you might imagine. Could there possibly be a solution the Web host isn't aware of?
Comment: Then you are out of luck :(
Comment: Possibly, but without a *mass* more details (which I would expect you're probably not in a position to provide) nobody's going to be able to help you.
Comment: Which particular details?
Comment: Who it is, what they're running, source code to whatever it was that deleted the messages... that sort of thing.
Comment: Do you have access to the server? Are you willing to spend a few hundred/thousands dollars on this? What OS is it?
Comment: Yeah, you're right, not the kind of info I'm privy to.
Answer: Maybe with some good old digital forensics/file recovery you would be able to recover the files, but then we are talking shutting down their server and manually recovering it to an external drive (providing none of that diskspace got overwritten with a new file).
I highly doubt they are willing to do that, it's very costly.
(I would say recover from backups, but the chance is they do not have any (complete) backup  because of the short timespan)


Javascript: Assignment of the return of the function to the variable

Question: I work on the project that I need to take the data input in the search box and pass it to API. I have built the function
function searchRef(){
  const inputValue = searchIp.value;
  return inputValue
}

searchBtn.addEventListener('click', searchRef)

and I want to assign inputValue to the variable, i.e let input= searchRef()
Regretfully, it does not work. What am I doing wrong?
thank you
Answer: You make use of the arrow function to achieve that:
searchBtn.addEventListener('click', () =>{ let input = searchRef() });

You can assign that input value to anything else you want or create a separate function all together.


Save contact form data in the database instead of transferring it to an email account

Question: So far I had to set up email DNS records and an email server or, alternatively, an email proxy by an email account to transfer emails from a CMS to my email account.

Email DNS records and email server
Contact form inquiry → PHP mail() function → Postfix → SMTP → My email account

Email proxy
Contact form inquiry → PHP mail() function → Mail Transfer Agent (like sSMTP) → SMTP → Email proxy account → My email account

All I want to is to get leads (contact form inquiries) directly into Drupal in a secured way and to manage them from Drupal (in a rather simple way of keep or delete).
The advantages (mainly for freelance, single-administrator applications) are

Easier server and credentials management (No need to worry about securing credentials for an email proxy account.)
Fast email reply (by clicking on the email address of the inquirer from browser or from smartphone) and sending an email directly from your preferred email application in regular PC or smartphone

How could I "instruct" the core Contact module to save leads in Drupal itself and not transferring them to any email account? (I might need a third-party module for that, if it's not possible without customizing its PHP code.)
Comment: IMO Contact module has never been that useful. Use Webform.
Comment: [Webform is available for Drupal 8](https://www.drupal.org/project/webform/releases/8.x-5.1), and it has been for a while already. We have plenty of Drupal 8 sites using it.
Comment: Webform has been around a while....
Answer: What you are asking for is the purpose of the Contact Storage module.
If you need further form elements, consider using the Webform module. It surely adds a whole lot of overhead, but gives you a wide variety of form elements and stores it's results in the database as well.


How to create RDNS (Reverse DNS)

Question: How to create a RDNS (Reverse DNS) in centos or Ubuntu [PTR record]
I searched a lot in Internet and didn't find any solutions. I got a idea it can be done by BIND or BIND9.
In some forum i seen some peoples wrote "RDNS cannot created by user, it can only created by Server provider or server manager" - Is it True.
I tried my best all possible way but failed, Can anyone provide me a procedure or idea to create RDNS.
Example
Example.com - xxx.xxx.xxx.xx0
Example.xyz - xxx.xxx.xxx.xx1
xyz.example.com - xxx.xxx.xxx.xx2

Thanks you.
Comment: RDNS can be created only by your ISP. That's true.
Comment: That's not quite true.  I have, for example, one network in which I own the whole block of IPs, so I am officially the PTR record holder.  You have to have an arrangement with your upstream provider.  Basically they have to be willing to "forward" your DNS server PTR records.  Although most nowadays have an external facility/API to manage this so that you don't accidentally claim IPs you're not entitled to, particularly if you share the IP block with others.
Answer: Well... not so sure if this is the answer but basically, you only need a PTR record on your domain.reverse.conf file that would look like this:
5   IN  PTR    www.example.com.

and on named.conf:
zone "0.168.192.in-addr.arpa" IN {
    type master;
    file "domain.reverse.conf";
};

Basically, that would asign a PTR record for 192.168.0.5 to www.example.com
As @tudor mentioned, in a real scenario, you would need to talk firstly with your service provider, as you are not the owner of a block of IPs (in most cases) but on a local environment, it will work.

How to create RDNS (Reverse DNS)


VirtualHost with Multiple Domains Not Serving Correct DocumentRoot

Question: Running CentOS 6, Apache.
I have the following in my httpd.conf:
<VirtualHost *:80>
    DocumentRoot /var/www/html
    ServerName domain1.example
    ServerAlias www.domain1.example
</VirtualHost>

<VirtualHost *:80>
    DocumentRoot /var/www/html/domain2
    ServerName domain2.example
    ServerAlias www.domain2.example
</VirtualHost>

Whenever I visit domain2.example (or www.domain2.example), I am shown domain1.example.
EDIT: More info from httpd.conf
Listen 80
#NameVirtualHost *:80

Things I have checked:

Apache has been restarted
DNS for domain1.example and domain2.example point to same IP

What am I missing?
SOLUTION:
As Patrick points out, in Apache 2.2, NameVirtualHost is required when using multiple VirtualHosts. The solution here was to uncomment the line in httpd.conf:
#NameVirtualHost *:80

to:
NameVirtualHost *:80

Comment: Is this apache 2.2 or 2.4?  You might consider moving domain2.com DocumentRoot into it's own directory that is not in /var/www/html
Comment: Apache 2.2. Thanks for the suggestion!
Comment: You are not showing everything. What is your `NameVirtualHost` and `Listen` directives? Does the DNS resolution works correctly from the server and the client querying it?
Comment: @PatrickMevzek added more info. `NameVirtualHost` is commented out. Is this necessary in this situation? Every example I've seen on the web doesn't specify this being on.
Comment: Have a look at http://httpd.apache.org/docs/2.4/mod/core.html#namevirtualhost : "Prior to 2.3.11, NameVirtualHost was required to instruct the server that a particular IP address and port combination was usable as a name-based virtual host. In 2.3.11 and later, any time an IP address and port combination is used in multiple virtual hosts, name-based virtual hosting is automatically enabled for that address."  You are on 2.2 so `NameVirtualHost` is mandatory in your case...
Comment: @PatrickMevzek you are correct, sir. Thanks for that. If you can make your solution an answer I'll accept it.
Answer: As you are using only Apache 2.2 you absolutely need to use the NameVirtualHost directive in order to let Apache know that you have multiple websites under different names but with the same IP.
The directive became deprecated in Apache 2.3, as written in this documentation: httpd.apache.org/docs/2.4/mod/core.html#namevirtualhost

Prior to 2.3.11, NameVirtualHost was required to instruct the server that a particular IP address and port combination was usable as a name-based virtual host. In 2.3.11 and later, any time an IP address and port combination is used in multiple virtual hosts, name-based virtual hosting is automatically enabled for that address.

For your version, the guide on configuration name based virtual host is here:
http://httpd.apache.org/docs/2.2/vhosts/name-based.html#using
Answer: Looks like it's matching to the first vhost.
Try something like this:
<VirtualHost *:80>
    DocumentRoot /var/www/html/domain1
    ServerName domain1.com
    ServerAlias www.domain1.com
</VirtualHost>

<VirtualHost *:80>
    DocumentRoot /var/www/html/domain2
    ServerName domain2.com
    ServerAlias www.domain2.com
</VirtualHost>

Comment: Thanks, Ben. Is there an alternative to moving the entire docroot? `domain1` is a live prod server. Is there any explanation for why the first vhost is matching when a different domain is requested?
Comment: @ChrisTanner : have a look at http://httpd.apache.org/docs/2.2/vhosts/name-based.html#using


asymptotic of a product

Question: So the question that I'm working on is the following.
Show that $\Pi_{p\leq z}(1-\dfrac{1}{p})=\dfrac{C(1+\mathcal{o}(1))}{\log z}$.
First off I take logs and just work with the sum and thisis what I get.
$$\sum_{p\leq z}\log (1-\dfrac{1}{p})=-\sum_{p\leq z}\sum_{n=1}^{\infty}\dfrac{1}{np^{n}} $$
now so now I want to show that this sum is $-\log\log z +C' +\mathcal{o}(\log x^{-1})$.
But I'm not sure on who to do this, anyone got any ideas?
Comment: I show what you seek in [this answer](http://math.stackexchange.com/a/643947/5531) by appealing to a strengthened version of Mertens' formula, but I'm not sure if that's the road you'd like to take.
Comment: Actually, since you only want to show that $\prod_{p \leq z} (1-1/p) \sim C/\log z$ then you don't need the strengthened formula with the little o.  Using the elementary version of Mertens' formula with error $O(1/\log z)$ will still yield the desired answer.
Answer: $$\ln(n)=\sum_{d\mid n}\Lambda(d)$$
$$\ln(\lfloor x \rfloor!)=\sum_{n\leq x}\Lambda(n)\lfloor\frac{x}{n}\rfloor$$
$$x\ln(x)+O(x)=x\sum_{p\leq x}\frac{\ln(p)}{p}+O(x)$$
$$\sum_{p\leq x}\frac{\ln(p)}{p}=\ln(x)+O(1)$$
$$\text{ Now conclude your result using summation by parts}$$
Comment: And use summation by parts on what sum?  also I don't see how you can get the little oh out of this?
Comment: Yea my bad, you wont be able to get the little O term out of this, I thought you were asking for something weaker. You could look at Tao's http://terrytao.wordpress.com/2013/12/11/mertens-theorems/, if your comfortable with his Fourier analytic stuff.

asymptotic of a product