How to specify multiple submit handlers for multiple forms

Question: I am implementing hook_forms in my custom module and I don't know how to have multiple form submit handlers.
Here is my custom form function:
function data_retrieval_main_form()
{
    $form ['field2'] = array(
        '#type' => 'textfield',
        '#title' => t('Name'),
        '#default_value' => $name,
        '#maxlength' => 127,
    );

    $form ['field3'] = array(
        '#type' => 'textfield',
        '#title' => t('Age'),
        '#default_value' => $age,
        '#maxlength' => 127,
    );

    $form['fileupload'] = array(
        '#title' => t('Choose Image'),
        '#type' => 'file',
    );

    $form ['submit_field'] = array(
        '#type' => 'submit',
        '#default_value' => 'Submit',
    );

    return $form;
}

Here I am implementing hook_forms:
 function hook_forms($form_id, $args) {

      $forms ['mymodule_first_form'] = array(
        'callback' => 'data_retrieval_main_form',
      );

      $forms ['mymodule_second_form'] = array(
        'callback' => 'mymodule_main_form',
        'callback arguments' => array('some parameter'),
      );

      return $forms;
    }

Now I previously had a single form using hook_form and the handler was simply like this:
function data_retrieval_form_submit($form_id, &$form_state)
{
// validation and submission of form
}

But now that I have multiple forms, how do I have multiple submit handlers for the forms, is there a way in Drupal that checks which form is being submitted?
Comment: Might this can help : http://drupal.stackexchange.com/questions/80136/drupal-7-multiple-submit-handlers
Answer: You can specify a submit handler for each of your forms. Examples:
Override default handler with custom handler (new array):
$form['#submit'] = array('data_retrieval_form_submit');

Run custom handler after default (append to array):
$form['#submit'][] = 'data_retrieval_form_submit';

Run custom handler before default (prepend array):
array_unshift($form['#submit'], 'data_retrieval_form_submit');

You can add it to the $form array in a custom form builder function, or alter an existing form via hook_form_alter.
Comment: `$form['#submit']` needs to be an array...
Comment: Which function does this go into?
Comment: I've expanded on my answer.
Comment: It is not very clear what you mean, I have my form fields in my `hook_form` function, that is where I specify the submit button:  `$form ['submit_field'] = array(
        '#type' => 'submit',
        '#default_value' => 'Submit',
    );` so where and how can I specify which form links to which submit handler. You also mention `$forms` array but the code shows `$form` array?. Showing which function to put these in will help
Comment: It is `$form` (typo). The '#submit' key is not the same as 'submit' (or 'submit_field'), but both do belong to the `$form` array. Where? Unless you are using a hook (which I have shown), the function is one _you_ declare. In your case one of them appears to be `data_retrieval_main_form()`.
Comment: Just a note that your `data_retrieval_main_form()` function is not actually a hook, but a callback (although a hook [exists](https://api.drupal.org/api/drupal/modules!node!node.api.php/function/hook_form/7)). And you have renamed the `hook_forms()` function to `whatevermymodulenameis_forms()`, correct?
Comment: Yes, I have renamed `hook_forms` to my `modulename_forms` and I think I know what you mean now but my custom form builder function won't render a submit button unless I have an explicit submit field like my previous comment, it needs an input type submit
Comment: Sorry, I don't see what any of this has to do with your submit button?
Comment: How does a form get submitted? magic? no through a submit button. So if each form has its own submit handler then it would need its own submit button no?
Comment: Of course. I don't see how that is relevant. I'm talking about a key named '#submit' that is added to the `$form` array. Not 'submit' or 'submit_field' or anything having to do with your button. (Also: I don't believe sarcasm will do you any favours here.)
Comment: So how will the form submission get triggered off?  (Sorry for the sarcasm I'm getting quite frustrated here with Drupal)
Comment: Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/28866/discussion-between-othermachines-and-user3574492).


Optimization when Cost Function Slow to Evaluate

Question: Gradient descent and many other methods are useful for finding local minima in cost functions. They can be efficient when the cost function can be evaluated quickly at each point, whether numerically or analytically.
I have what appears to me to be an unusual situation. Each evaluation of my cost function is expensive. I am attempting to find a set of parameters that minimize a 3D surface against ground truth surfaces. Whenever I change a parameter, I need to run the algorithm against the entire sample cohort to measure its effect.  In order to calculate a gradient, I need to change all 15 parameters independently, meaning I have to regenerate all the surfaces and compare against the sample cohort way too many times per gradient, and definitely way too many times over the course of optimization.
I have developed a method to circumvent this problem and am currently evaluating it, but I am surprised that I have not found much in the literature regarding expensive cost function evaluations.  This makes me wonder if I am making the problem harder than it is and that there might be a better way already available.
So my questions are basically this: Does anyone know of methods for optimizing cost functions, convex or not, when evaluation is slow?  Or, am I doing something silly in the first place by rerunning the algorithm and comparing against the sample cohort so many times?
Comment: Have you heard of stochastic gradient descent? For deep neural networks applied to large Training sets you have a similar issue (but can Eval gradient analytically) and the standard solution is to make gradient descent based only on a single sample(stochastic) vs the whole cohort (batch)
Comment: "what appears to me to be an unusual situation. Each evaluation of my cost function is expensive.". This is not at all an unusual situation, in general. It shows up all over the place, for example when ever your cost function comes from running a simulation (Eg in this paper: http://white.ucc.asn.au/publications/White2015PsoTransistorSizing.pdf we simulate a circuit in SPICE taking 10s).
More boardly, in experimental science, evaluations can take ages. One of my friends Masters project is basically optimizing 5 parameters to find the best way of inserting DNA. Each evaluation takes 24 hours.
Comment: I'm only vaguely familiar with the area, so hence this being a comment rather than an answer. But what you are discussing  sounds **a lot** like the topic of Uncertainty Quantification, often faced by engineers, where a single evaluation of the target function made take weeks to evaluate (at least in the problems faced by my engineering coworkers). My very limited understanding of how this is handled is by making a surrogate approximation that is much easier to evaluate based on past evaluations and simpler engineering models and then use these surrogate models to chose the next evaluation...
Comment: ... of the more expensive target function. I hate to say it, but I don't know anymore about the topic at the moment; I've only been told about it briefly while discussing research topics with said engineers. Interestingly, it seems like a very challenging research area: I believe good models require both a good understanding of the physics *and* statistics.
Comment: I assume you're the first author of the linked paper. I am curious, did you compare it with state-of-the-art Bayesian optimization methods? The paper is very scant in terms of benchmarking (and references). I suppose that RBFs are much faster than GPs and scale better to larger number of training points, but they have a big disadvantage in that they are not probabilistic. So, it's going to be an interesting comparison; the two methods may work well in different regimes.
Comment: @seanv507 Yes, thanks, but I avoided it for a similar reason.  It takes about 30 seconds to a minute to run one sample.  If I have 15 parameters, I am looking at nearly 8 minutes per gradient calculation even if I only use one sample.  If the space is large, it may take a very long time.  Please correct me if you had other ideas in mind.
Answer: TL;DR
I recommend using LIPO. It is provably correct and provably better than pure random search (PRS). It is also extremely simple to implement, and has no hyperparameters. I have not conducted an analysis that compares LIPO to BO, but my expectation is that the simplicity and efficiency of LIPO imply that it will out-perform BO.
(See also: What are some of the disavantage of bayesian hyper parameter optimization?)
LIPO and its Variants
This is an exciting arrival which, if it is not new, is certainly new to me. It proceeds by alternating between placing informed bounds on the function, and sampling from the best bound, and using quadratic approximations. I'm still working through all the details, but I think this is very promising. This is a nice blog write-up, and the paper is Cédric Malherbe and Nicolas Vayatis "Global optimization of Lipschitz functions."
LIPO is most useful when the number of hyper-parameters that you are searching over is small.
LIPO has no tuning parameters and no choices about the surrogate function. By contrast, BO requires choosing a kernel function, kernel hyper-parameters, and an acquisition function and acquisition function hyper-parameters. This can mean that BO has a lot of "knobs" to adjust, and the experimental cost of adjusting those knobs can make BO expensive to apply.
Bayesian Optimization
Bayesian Optimization-type methods build Gaussian process surrogate models to explore the parameter space. The main idea is that parameter tuples that are closer together will have similar function values, so the assumption of a co-variance structure among points allows the algorithm to make educated guesses about what best parameter tuple is most worthwhile to try next. This strategy helps to reduce the number of function evaluations; in fact, the motivation of BO methods is to keep the number of function evaluations as low as possible while "using the whole buffalo" to make good guesses about what point to test next. There are different figures of merit (expected improvement, expected quantile improvement, probability of improvement...) which are used to compare points to visit next.
Contrast this to something like a grid search, which will never use any information from its previous function evaluations to inform where to go next.
Incidentally, this is also a powerful global optimization technique, and as such makes no assumptions about the convexity of the surface. Additionally, if the function is stochastic (say, evaluations have some inherent random noise), this can be directly accounted for in the GP model.
On the other hand, you'll have to fit at least one GP at every iteration (or several, picking the "best", or averaging over alternatives, or fully Bayesian methods). Then, the model is used to make (probably thousands) of predictions, usually in the form of multistart local optimization, with the observation that it's much cheaper to evaluate the GP prediction function than the function under optimization. But even with this computational overhead, it tends to be the case that even nonconvex functions can be optimized with a relatively small number of function calls.
A downside to GP is that the number of iterations to get a good result tends to grow with the number of hyper-parameters to search over.
A widely-cited paper on the topic is Jones et al (1998), "Efficient Global Optimization of Expensive Black-Box Functions." But there are many variations on this idea.
Random Search
Even when the cost function is expensive to evaluate, random search can still be useful. Random search is dirt-simple to implement. The only choice for a researcher to make is setting the the probability $p$ that you want your results to lie in some quantile $q$; the rest proceeds automatically using results from basic probability.
Suppose your quantile is $q = 0.95$ and you want a $p=0.95$ probability that the model results are in top $100\times (1-q)=5$ percent of all hyperparameter tuples. The probability that all $n$ attempted tuples are not in that window is $q^n = 0.95^n$ (because they are chosen independently at random from the same distribution), so the probability that at least one tuple is in that region is $1 - 0.95^n$. Putting it all together, we have
$$
1 - q^n \ge p \implies n \ge \frac{\log(1 - p)}{\log(q)}
$$
which in our specific case yields $n \ge 59$.
This result is why most people recommend $n=60$ attempted tuples for random search. It's worth noting that $n=60$ is comparable to the number of experiments required to get good results with Gaussian Process-based methods when there are a moderate number of parameters.
Unlike Gaussian Processes, for random search, the number of queried tuples does not grow with the number of hyper-parameters to search over. Indee,d the dimension of the problem does not appear in the expression that recommends attempting $n=60$ random values.  However, this does not mean that random search is "immune" to curse of dimensionality. Increasing the dimension of the hyperparameter search space can mean that the average result drawn from among the "best 5% of values" is still very poor. More information: The "Amazing Hidden Power" of Random Search? The intuition is that if we increase the volume of the search space, then we are naturally also increasing the volume of 5% of the search space.
Since you have a probabilistic characterization of how good the results are, this  result can be a persuasive tool to convince your boss that running additional experiments will yield diminishing marginal returns.
Comment: This seems like a modern variant of response surface methods!
Comment: Actually, random search can work remarkably well: http://www.argmin.net/2016/06/20/hypertuning/
Comment: @Tim Yes, your point is well-taken. I didn't want to "decide" the issue of which is better in this post, since there are essentially endless permutations on BO, each claiming to be "the best" black-box optimizer -- making it impossible to be definitive. I agree that random search can work quite well, but I would actually recommend LIPO over PRS. LIPO is provably correct and strongly out-performs PRS (on average) in all of my experiments. LIPO also has minimal estimation cost: if you can minimize a QP, then you can use LIPO, and LIPO has zero hyperparameters (in contrast to BO).
Comment: @DavidSilverberg It doesn't; dimension is not a number that appears in the equations that recommend $n=60$, but see caveats here: https://stats.stackexchange.com/questions/561164/the-amazing-hidden-power-of-random-search Essentially, you can make the search space so large that the "best 5% of hyperparameters" is still too poor quality *on average* to be useful.
Comment: Does the recommendation n=60 depend on the number of dimensions of space to be searched? I'm dealing with a hyperparameter space with 5 dimensions and 60 sounds small to me.
Answer: The literature on evaluation of expensive black-box function is quite vast and it is usually based on surrogate-model methods, as other people pointed out. Black-box here means that little is known about the underlying function, the only thing you can do is evaluate $f(x)$ at a chosen point $x$ (gradients are usually not available).
I would say that the current gold standard for evaluation of (very) costly black-box function is (global) Bayesian optimization (BO). Sycorax already described some features of BO, so I am just adding some information that might be useful.
As a starting point, you might want to read this overview paper 1.
There is also a more recent one [2].
Bayesian optimization has been growing steadily as a field in the recent years, with a series of dedicated workshops (e.g., BayesOpt, and check out these videos from the Sheffield workshop on BO), since it has very practical applications in machine learning, such as for optimizing hyper-parameters of ML algorithms -- see e.g. this paper [3] and related toolbox, SpearMint. There are many other packages in various languages that implement various kinds of Bayesian optimization algorithms.
As I mentioned, the underlying requirement is that each function evaluation is very costly, so that the BO-related computations add a negligible overhead. To give a ballpark, BO can be definitely helpful if your function evaluates in a time of the order of minutes or more. You can also apply it for quicker computations (e.g. tens of seconds), but depending on which algorithm you use you may have to adopt various approximations. If your function evaluates in the time scale of seconds, I think you're hitting the boundaries of current research and perhaps other methods might become more useful.
Also, I have to say, BO is rarely truly black-box and you often have to tweak the algorithms, sometimes a lot, to make it work at full potential with a specific real-world problem.
BO aside, for a review of general derivative-free optimization methods you can have a look at this review [4] and check for algorithms that have good properties of quick convergence. For example, Multi-level Coordinate Search (MCS) usually converges very quickly to a neighbourhood of a minimum (not always the global minimum, of course). MCS is thought for global optimization, but you can make it local by setting appropriate bound constraints.
Finally, you are interested in BO for target functions that are both costly and noisy, see my answer to this question.

References:
1 Brochu et al., "A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical
Reinforcement Learning" (2010).
[2] Shahriari et al., "Taking the Human Out of the Loop: A Review of Bayesian Optimization" (2015).
[3] Snoek et al., "Practical Bayesian Optimization of Machine Learning Algorithms", NIPS (2012).
[4] Rios and Sahinidis, "Derivative-free optimization: a review of algorithms and comparison of software implementations", Journal of Global Optimization (2013).
Comment: @ lacerbi : I really like your answer! Can you please explain what you mean by costly and noisy? My guess is that "costly" means when you have a loss function with many model parameters (i.e. the weight connections in the neural network) and repeatedly evaluating the derivatives of such a loss function is very "costly". On the other hand, does "noisy" refer to loss functions that involve non-deterministic (i.e. random) variables? Why do non-gradient based optimization algorithms present advantages for "noisy" functions? can you please explain this further? thank you so much!
Comment: @stats555: Your guess is inexact in that you are describing a  very specific case: that's not what "costly" generally means. First, here we are talking about black-box optimization, so there are no derivatives involved. Costly means costly: there is a cost involved in the evaluation of the objective function, be it time, energy, money, other resources, etc. and so you cannot afford to perform an arbitrary large number of function evaluations. Functions can be costly without the need to have many parameters (e.g., complex simulators).
Comment: @stats555: Noisy means stochastic, i.e. evaluating the target at the same input values twice may yield different output values for the target functions. That may happen when computation of the target involves randomness of some kind, broadly speaking.
Answer: I don't know the algorithms myself, but I believe the kind of optimization algorithm that you are looking for is derivative-free optimization, which is used when the objective is costly or noisy.
For example, take a look at this paper (Björkman, M. & Holmström, K. "Global Optimization of Costly Nonconvex Functions Using Radial Basis Functions." Optimization and Engineering (2000) 1: 373. doi:10.1023/A:1011584207202) whose abstract seems to indicate this is exactly what you want:

The paper considers global optimization of costly objective functions, i.e. the problem of finding the global minimum when there are several local minima and each function value takes considerable CPU time to compute. Such problems often arise in industrial and financial applications, where a function value could be a result of a time-consuming computer simulation or optimization. Derivatives are most often hard to obtain, and the algorithms presented make no use of such information.
Comment: @stats555: "Costly" just means "too expensive to compute". Like imagine optimizing the shape of an airplane wing, or the folding of a protein, or some other process that might take hours to simulate (...or possibly require you to run a physical experiment!) and/or potentially a ton of money. Simply taking *one* gradient is going to be incredibly expensive at that point, never mind N of them.
Comment: Can you please explain what you mean by costly and noisy? My guess is that "costly" means when you have a loss function with many model parameters (i.e. the weight connections in the neural network) and repeatedly evaluating the derivatives of such a loss function is very "costly". On the other hand, does "noisy" refer to loss functions that involve non-deterministic (i.e. random) variables? Why do non-gradient based optimization algorithms present advantages for "noisy" functions? can you please explain this further? thank you so much!
Comment: Thank you so much! Can you please describe "noisy" as well?
Comment: @stats555: For "noisy", I guess you can think of it as random variables, but I think it's more about the idea of not being able to compute the exact *gradient* (forcing you to approximate the gradient, which would add "noise"). Like think of stochastic gradient descent. You might not have enough resources to compute the full gradient, so you'd have to sample the dataset and compute a noisy approximation to it. Which might be fine—maybe that approach will work for your problem—but you can also imagine problems where a different approach that doesn't use gradients would be superior to SGD.
Comment: @stats555: As a trivial example, if your problem happens to be monotonic across the variables of interest, you might be able to just binary search (or interpolation search) instead of trying to approximate the gradient.
Answer: You are not alone.
Expensive-to-evaluate systems are very common in engineering, such as finite element method (FEM) models and computational fluid dynamics (CFD) models. Optimization of these computatationaly expensive models is very needed and challenge because evoluationary algorithms often needs tens of thouands of evaluations of the problem which is not an option for expensive-to-evaluate problems. Fortunately, there are lots of methods (algorithms) avaiable to solve this problem. As far as I know, most of them are based on surrogate models (metamodels). Some are listed below.

Efficient Global Optimization (EGO, also known as Bayesian optimization) [1] . The EGO algorithm has been mentioned above and may be the most famous surrogate-based optimization algorithm. It is based on the Kriging model and an infill criterion called expected improvement function (EI). R packages including the EGO algorithm are DiceOptim and DiceKriging.
Mode-pursuing sampling (MPS) method [2]. The MPS algorithm is built on the RBF model and an adptive sampling strategy is used to pick up candidate points. The MATLAB code is publised by the authors at http://www.sfu.ca/~gwa5/software.html. The MPS algorithm may need more evaluations to get the optimum, but can handle more complicated probelms than the EGO algorithm from my personal experience.
Ensemble surrogates models by Juliane Müller [3]. She used multiple surrogates to enhance the searching ability. The MATLAB toolbox MATSuMoTo is avaiable at https://github.com/Piiloblondie/MATSuMoTo.

In summery, these surrogate-based optimization algorithms try to find the global optimum of the problem using as few evaluations as possible. This is achieved by making the full use of the informations that the surrogate (surrogates) provides. Reviews on optimization of compuationally expensive problems are in [4-6].

Reference:

D. R. Jones, M. Schonlau, and W. J. Welch, "Efficient global optimization of expensive black-box functions," Journal of Global Optimization, vol. 13, pp. 455-492, 1998.
L. Wang, S. Shan, and G. G. Wang, "Mode-pursuing sampling method for global optimization on expensive black-box functions," Engineering Optimization, vol. 36, pp. 419-438, 2004.
J. Müller, "Surrogate Model Algorithms for Computationally Expensive Black-Box Global Optimization Problems," Tampere University of Technology, 2012.
G. G. Wang and S. Shan, "Review of metamodeling techniques in support of engineering design optimization," Journal of Mechanical Design, vol. 129, pp. 370-380, 2007.
A. I. Forrester and A. J. Keane, "Recent advances in surrogate-based optimization," Progress in Aerospace Sciences, vol. 45, pp. 50-79, 2009.
F. A. C. Viana, T. W. Simpson, V. Balabanov, and V. Toropov, "Metamodeling in Multidisciplinary Design Optimization: How Far Have We Really Come?," AIAA Journal, vol. 52, pp. 670-690, 2014/04/01 2014.
Comment: @Hans Yes. The mentioned EGO and the Bayesian optimization are in fact the same algorithm. It is called EGO in the engineering design optimization field and is called Bayesian optimization in the statistics and machine learning fields. Other mentioned algorithms have very similar framework as the Bayesian optimization with different surrogate models and infill criteria.
Comment: @Hans I am afraid not because I have no knowledge about LIPO.
Comment: Could you compare the methods you mentioned to the Bayesian optimization method mentioned in other answers?
Comment: Thank you. Would you care to also compare them to LIPO?
Answer: The two simple strategies that I have successfully used in the past are:

If possible, try to find a simpler surrogate function approximating your full cost function evaluation -- typical an analytical model replacing a simulation.  Optimize this simpler function.  Then validate and fine tune the resulting solution with your exact cost function.
If possible, try to find a way to evaluate an exact "delta-cost" function -- exact as opposed to be an approximation from using the gradient.  That is, from an initial 15-dimensional point for which you have the full cost evaluated, find a way to derive how the cost would change by making a small change to one (or several) of the 15 components of your current point.  You would need to exploit localization properties of a small perturbation if any in your particular case and you would likely need to define, cache, and update an internal state variable along the way.

Those strategies are very case specific, I don't know whether they can be applicable in your case or not, sorry if they are not.  Both could be applicable (as it was in my use cases): apply the "delta-cost" strategy to a simpler analytical model -- performance may improve by several orders of magnitudes.
Another strategy would be to use a second order method that typically tends to reduce the number of iterations (but each iteration is more complex) -- e.g., Levenberg–Marquardt algorithm.  But considering you don't seem to have a way to directly and efficiently evaluate the gradient, this is probably not a viable option in this case.
Answer: There are many tricks used in stochastic gradient descent that can be also applied to objective function evaluation. The overall idea is trying to approximate the objective function using a subset of data.
My answers in these two posts discuss why stochastic gradient descent works: the intuition behind it is to approximate the gradient using a subset of data.
How could stochastic gradient descent save time comparing to standard gradient descent?
How to run linear regression in a parallel/distributed way for big data setting?
The same trick applies to the objective function.
Let's still use linear regression as an example: suppose the objective function is $\|Ax-b\|^2$. If $A$ is huge, say a trillion rows, evaluating it once will take a very long time. We can always use a subset of $A$ and $b$ to approximate the objective function, which is the squared loss on the subset of the data.


Show that $\int_0^1 (\sin{x}-4/9)^3dx<0$ without a calculator.

Question: In this age of electronic devices, sometimes I like to challenge myself to find clever ways to find solutions to math problems that seem to require a calculator, without a calculator. (Here is a favorite example.)
I recently came up with this:

Without a calculator, show that $$\int_0^1 \left(\sin{x}-\frac{4}{9}\right)^3dx<0$$

It's a pretty close shave: my computer says the LHS $\approx −0.0000050...$.
The exact value of the LHS is:
$$\frac{1}{729}\left(972(2+\cos{1})\sin^4{0.5}-243(2-\sin{2})+432(1-\cos{1})-64\right)$$
I tried to use Maclaurin series, but that seems to be futile.
I also tried to find some kind of useful symmetry of the graph of $y=\sin{x}-\frac{4}{9}$, but to no avail.
Any clever way to do this? Just curious.
Comment: Perhaps another method would be to use approximation methods of an integral to show that the value is strictly negative. (considering that the value for sin(1) and sin(1/2) is known).
I tried before to calculate using an approximation method, so I could see that the value is strictly negative.
Comment: By substitution,$$\int_0^1 \left(\sin(x) - \frac49\right)^3 \, dx = \int_{-\frac49}^{\sin(1)-\frac49} \frac{y^3}{\sqrt{1 - \left(y + \frac49\right)^2}} \, dy$$and you can apply the [binomial series](https://en.wikipedia.org/wiki/Binomial_series) to the denominator.
Comment: @user170231. This is spectacular (look at my second answer)
Answer: For a nicer looking expression
$$I=\int_0^1 \left(\sin(x)-\frac{4}{9}\right)^3\,dx=\frac{1472+972 \sin (2)-3915 \cos (1)+243 \cos (3)}{2916}$$
Now, using the series expansion of the sine and the binomial theorem
$$\left(\sin(x)-\frac{4}{9}\right)^3=-\frac{64}{729}+\frac{1}{216}\sum_{n=1}^\infty \frac{144\times 2^{n} \cos \left(\frac{\pi  n}{2}\right)-2
   \left(27\times 3^{n}-145\right) \sin \left(\frac{\pi  n}{2}\right)}{n!} x^n$$ Now, integrating termwise,  the partial sums
$$S_p=-\frac{64}{729}+\frac{1}{108}\sum_{n=1}^p \frac{72\times 2^{n} \cos \left(\frac{\pi  n}{2}\right)-
   \left(27\times 3^{n}-145\right) \sin \left(\frac{\pi  n}{2}\right)}{(n+1)!}$$ Up to $S_3$, the values are easy to compute by hand. For $p>3$, I cheated and used my fifty plus years old non-programmable pocket calculator (four operations and a single memory) to generate the sequence
$$\left\{\frac{152}{729},-\frac{172}{729},-\frac{31}{2916},\frac{1141
   }{14580},-\frac{31}{7290},-\frac{649}{51030},\frac{1321}{1632960
   },\frac{2089}{1632960},-\frac{1871}{24494400}\right\}$$ For $p \geq 9$, all $S_p$ are negative (checked with a calculator).
Comment: You wish to introduce $p$ on the RHS of the equation defining $S_p$.
Comment: @Gary. Thank you for pointing it.
Answer: Using @user170231's suggestion in comments; letting $a=\frac 49$ and $s=\sin(1)$
$$I=\int_{-a}^{s-a} \frac{y^3}{\sqrt{1 - \left(y + a\right)^2}} \, dy=\frac{s}{2 \sqrt{\pi }}\sum_{n=0}^\infty u_n$$
$$u_n= \frac{\Gamma \left(n+\frac{1}{2}\right)}{\Gamma (n+1)} \left(-\frac{2 a^3}{2 n+1}+\frac{3 a^2 s}{n+1}-\frac{6 a
   s^2}{2 n+3}+\frac{s^3}{n+2}\right)s^{2 n}$$ $u_0$ is the only negative term. All other terms are positive, almost in geometric progression
$$\frac {u_{n+1}}{u_n}=s^2\,\left( 1-\frac{3}{2 n}+O\left(\frac{1}{n^2}\right)\right)$$
and their infinite sum does not exceed $u_0$.
Edit
Pushed by @Dan's  comment to do it, assume the above ratio and write
$$S_p==\frac{s}{2 \sqrt{\pi }}\Bigg[  \sum_{n=0}^p u_n+u_{p+1}\, \, _2F_1\left(1,p-\frac{1}{2};p+1;s^2\right)\Bigg]$$ we have (I used a calculator)
$$\left(
\begin{array}{cc}
p & 10^6\,S_p \\
 0 & -2595.89 \\
 1 & -730.922 \\
 2 & -270.549 \\
 3 & -115.935 \\
 4 & -55.3259 \\
 5 & -29.1880 \\
 6 & -17.1416 \\
 7 & -11.3071 \\
 8 & -8.36943 \\
 9 & -6.84325 \\
 10 & -6.02958 \\
 20 & -5.01422 \\
 30 & -5.00872 \\
 40 & -5.00867 \\
\end{array}
\right)$$
Comment: OK. So the idea is that it is possible (without a calculator) to find an upper bound on the sum of the positive $a$'s, that does not exceed the magnitude of $a_0$?
Comment: Maybe I'm missing something, but letting $n=1$, we get $\dfrac{a_2}{a_1}<0$ so $a_1$ and $a_2$ cannot both be positive.
Comment: @Dan. This is the asymptotics only
Comment: @Dan. This is **the** good question and solution ! What do you I have been doing for the last 3 hours ? Cheers :-)
Comment: Shouldn't it be asymptotic to the value of $\int_0^1 \left(\sin{x}-\frac{4}{9}\right)^3dx$ ?
Comment: @Dan. I type too fast ! Thanks for pointing and for having pushed me.
Comment: Thank you for your answer! I appreciate your contributions to this community.
Comment: @Dan. Thanks for the nice words. I fixed a mistake in my edit. Cheers :-)
Answer: Using Maclaurin series, we find that for all positive real $x$, the Maclaurin series of $\sin x$ alternates between being larger and smaller than $\sin x$. This is true because: 1) the Maclaurin series is always tangent to $\sin x$ as it shares the same first and second derivative (and so the error behaves almost like $\pm x^3$ with even smaller higher-order terms) (2) look at the behaviour of the function to infinity, as given by the sign of the highest-order term. For example, with $n = 3$ the limit to infinity of the Taylor series is $-\infty$, and so $\sin x > x - x^3/3!$. Choosing $x = \pi/4$ for example, we can see this is true given the first 3 decimal places, if we know $1/\sqrt{2} \approx 0.707$ and overestimate $\pi$ as $22/7$.
Given $f(x) = (\sin x - 4/9)^3$ and $T_n (x)$ being the Taylor series of $\sin x - 4/9$ to $n$ terms, $|f(x)^3 - (T_n (x))^3|$ $< f(x) - (T_n (x))$ when both $f(x), T_n (x)$ are small and if $T_n(x) > f(x)$. The right hand side of this inequality is bounded upwards by the Lagrange error bound of $\sin x$. This is given by $\frac{M}{(n + 1)!} |x - c|^{n+1}$. With $M = 1$, $x = 1$ and $c = 0$ (to find the maximum error for all $x$ between $0$ and $1$), we get that $\frac{1}{(n + 1)!} < 5 \cdot 10^{-6}$.
Now to bound this integral upwards, the negative area must be smaller and the positive area must be larger. In other words, we need to find an upper bound of $f(x)$, which gives candidates $x = 1, 5, 9 \cdots$ (and $f(x) < T_n(x)$ also holds for these values of $n$). $n = 5$ does not satisfy the inequality and so the next candidate is $n = 9$.
The value of this integral is $-5.0079 \cdot 10^{-6}$ which is a remarkably good approximation.
So we need to show that:
$$\int_0^1 \left(x-\frac{x^{3}}{3!}+\frac{x^{5}}{5!}-\frac{x^{7}}{7!}+\frac{x^{9}}{9!}-\frac{4}{9}\right)^{3} \ dx < 0$$
which would be extremely tedious to do completely by hand, but you only need a pocket calculator to show this.
Comment: I got the same by Wolphram Alpha, free version.
Comment: How did you get the value $5 \cdot 10^{-6}$ ?
Comment: @Dan This is going to be circular, but use a lot of terms for the Maclaurin series to approximate sin x, use a pocket calculator to find the value of the approximate integral, then we find the minimum number of terms such that we know it will be negative even with the maximum error. Otherwise we have to try $n = 5$ first before we settle on $n = 9$, which you might prefer.


How long should I wait before accepting an answer?

Question: I recently asked two questions for which I would like to receive multiple answers:

What are some results in Economics that are both a consensus and far from common sense?
What are some important mathematics results that were first developed in Economics?

How long can I wait to accept one? If someone gives a good answer, but I would like to get other answers anyway, is it disrespectful or against the rules to delay the acceptance for a while?
Answer: I'm not aware of any rule about this.  My view is that it's good practice to wait a few days after posting a question before accepting an answer.  The advantages of this are:

It allows time for more answers to be posted, and for those answers to be compared, and the best can then be accepted.
It allows time for comments and voting on the answers, which may help in identifying which is best.
While it is possible to withdraw an acceptance and accept another answer, an early acceptance could have the effect of discouraging further answers (which may be illogical but my impression is that is does sometimes have that effect).
Answer: Standard procedure is to accept an answer when there is one that you feel has addressed your question. If a better answer comes along later, you can always switch to accepting the new one instead.


Where should I look at in Eclipse source code to figure out how Eclipse generate JUnit testing result?

Question: I'm analyzing Apache ant source code for my research. When a test suite runs, test cases are executed and Eclipse shows the test result as the image below. My goal is to get the executed test case names as Eclipse does. If I can see source code where Eclipse handles this, I think I can get the name list. Therefore, I'd like to know the source code location where Eclipse handle this or an easier way to achieve the goal... I tried using JUnit task in an ant build script to generate the test report so that I can get the test list by parsing the text/xml report. I could get the report with some warnings stating that duplicate classes are detected because I'm testing ant with ant. However, the report showed the test method name without its full class name...
Answer: Well, there is a short workaround for what you want to achieve (to have a specific, deterministic test run order), it is the @FixMethodOrder annotation on your test class. That will make your tests run in a specified order in every environment (or at least it should :-)).
If you want to do it in the hard way (analyze the source code of the Eclipse JUnit runner), I would advise installing the PDT tools and sources of the plugins first, and also set the Include all plug-ins from target in Java search under the Plug-in Development options (so you can find the plug-in types easily with Ctrl+Shift+T/R).
Then you have the Plug-in Spy which is a rock of a feature: press Alt+Shift+F2 and click on any element on the UI. Then you get all the info where the given component in defined (alternatively, you can press Alt+Shift+F1 zo display data of the selected component). An example is below:


Linear regression is a estimation of conditional expectation?

Question: I am studying the topic of regression for the first time and some questions arise. First, linear regression is a estimation of conditional expectation? And also the conditional expectation estimate is the so-called $y ̂$ estimate? This is:
$$y=E(Y|X)+e$$ $$y=y ̂+e$$ $$y ̂=E(Y|X)$$ $$(|)=+$$
Second, the linearity of the parameters is an assumption of the linear regression to estimate the conditional expectation? $$ $$ Third, Hansen's book on econometrics says about this problem: "the linear CEF model is empirically unlikely to be accurate unless $x$ is discrete and low-dimensional so all interactions are included. Consequently in most cases it is more realistic to view the linear specification as an approximation". What interpretation can be given to this phrase?
Comment: There are some subtleties with calling it a conditional expectation that you'll learn as you study regression for years, but yes, there is an entire distribution for every input $x$, and the fitted value (prediction) is the expected value of that distribution.
Answer: The linear regression provides the minimum mean squared error linear-in-parameters approximation to the CEF. If you can approximate a function with a Taylor series expansion with enough terms, you could do this pretty well, even if the actual CEF is nonlinear, by using lots of interactions and polynomial terms as long as you have enough data and have not left anything important out of your model.
If your world is truly is low dimensional and discrete, by calculating the mean in each cell (like average wage for college educated Asian women who live in the Midwest and enjoy musical theatre), your approximation of the CEF could be very good. This is what it means to include all interactions. With continuous covariates this is harder, since you have to either bin your data or smooth it to interpolate the unobserved data, and the approximation can be quite poor.
Here's toy example where we approximate a fairly non-linear Poisson CEF $$E[Y \vert X,Z] = \exp(a + b \cdot X +c \cdot Z + d \cdot X \cdot Z)$$ with means and with regression with all interactions. Here X takes on 5 values and Z takes on 2, so we have 10 cells in total if we use dummy variables:
<code>. set obs 5
number of observations (_N) was 0, now 5

. gen x = _n

. expand 100
(495 observations created)

. gen z = mod(_n,2)

. gen y = rpoisson(x+2*z)

. table x z, c(mean y)

----------------------
          |     z
        x |    0     1
----------+-----------
        1 | 1.06  2.76
        2 | 2.04  4.16
        3 | 2.96  4.96
        4 | 4.26  6.58
        5 | 5.18  6.76
----------------------

. quietly reg y i.x#i.z

. margins x#z

Adjusted predictions                            Number of obs     =        500
Model VCE    : OLS

Expression   : Linear prediction, predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         x#z |
        1 0  |       1.06   .2874746     3.69   0.000      .495165    1.624835
        1 1  |       2.76   .2874746     9.60   0.000     2.195165    3.324835
        2 0  |       2.04   .2874746     7.10   0.000     1.475165    2.604835
        2 1  |       4.16   .2874746    14.47   0.000     3.595165    4.724835
        3 0  |       2.96   .2874746    10.30   0.000     2.395165    3.524835
        3 1  |       4.96   .2874746    17.25   0.000     4.395165    5.524835
        4 0  |       4.26   .2874746    14.82   0.000     3.695165    4.824835
        4 1  |       6.58   .2874746    22.89   0.000     6.015165    7.144835
        5 0  |       5.18   .2874746    18.02   0.000     4.615165    5.744835
        5 1  |       6.76   .2874746    23.52   0.000     6.195165    7.324835
------------------------------------------------------------------------------

. quietly poisson y i.x#i.z

. margins x#z

Adjusted predictions                            Number of obs     =        500
Model VCE    : OIM

Expression   : Predicted number of events, predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         x#z |
        1 0  |       1.06   .1456022     7.28   0.000     .7746249    1.345375
        1 1  |       2.76   .2349468    11.75   0.000     2.299513    3.220487
        2 0  |       2.04   .2019901    10.10   0.000     1.644107    2.435893
        2 1  |       4.16   .2884441    14.42   0.000      3.59466     4.72534
        3 0  |       2.96   .2433105    12.17   0.000      2.48312     3.43688
        3 1  |       4.96   .3149603    15.75   0.000     4.342689    5.577311
        4 0  |       4.26   .2918904    14.59   0.000     3.687905    4.832095
        4 1  |       6.58   .3627671    18.14   0.000     5.868989    7.291011
        5 0  |       5.18   .3218695    16.09   0.000     4.549147    5.810853
        5 1  |       6.76   .3676955    18.38   0.000      6.03933     7.48067
------------------------------------------------------------------------------
</code>
If you omit the interaction between X and Z, you get something slightly worse:
<code>. quietly reg y i.x i.z

. margins x#z

Adjusted predictions                            Number of obs     =        500
Model VCE    : OLS

Expression   : Linear prediction, predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         x#z |
        1 0  |      1.024   .2111675     4.85   0.000     .6091028    1.438897
        1 1  |      2.936   .2111675    13.90   0.000     2.521103    3.350897
        2 0  |      1.914   .2111675     9.06   0.000     1.499103    2.328897
        2 1  |      3.826   .2111675    18.12   0.000     3.411103    4.240897
        3 0  |      3.324   .2111675    15.74   0.000     2.909103    3.738897
        3 1  |      5.236   .2111675    24.80   0.000     4.821103    5.650897
        4 0  |      3.854   .2111675    18.25   0.000     3.439103    4.268897
        4 1  |      5.766   .2111675    27.31   0.000     5.351103    6.180897
        5 0  |      5.084   .2111675    24.08   0.000     4.669103    5.498897
        5 1  |      6.996   .2111675    33.13   0.000     6.581103    7.410897
------------------------------------------------------------------------------
</code>
This is an example of misspecification.
Comment: thanks for your answer. Another question: when I estimate an OLS regression, it is not necessary to know the distribution of any of the variables, right?
Comment: Strictly speaking, no. It's the unobserved error distribution that matters.
Answer:
Yes and yes. There is a subtle technical point here, though I hesitate to mention it until you've gotten used to the idea of regression predicting an expected value instead of just a number that "should" be the right answer.

(Don't read this parenthetical part for a few months or years until you're much more comfortable with regression. The subtle point is that we often don't see the predictors as random variables, so there isn't a multivariate distribution where we condition on many variables to examine $Y$. We think of $Y\vert X$ as a family of univariate distributions that are parameterized by the predictor variables. This is technically correct in many cases but not especially useful, particularly not to a beginner.)

Right again!

For the first two, I think it makes sense when you start simulating regressions. I'll let you think about how to do that and can come back and edit this answer with some R code. But I do think it's a good exercise to think through it for a while.

This gets into a George Box quote: "All models are wrong, but some are useful." No, we probably don't have real phenomena following perfectly linear patterns, much like real data don't follow perfectly normal distributions. However, a linear model might provide a good enough model for us to do something useful.


Display field collection items together with nodes in view sorted by date

Question: I would like to create a view, which would list all the events of an organization (please look at the mockup). The problem is, that some of the are part of longer series, so I had to group them somehow. So now I have 2 content types (Event and Series of Events), the latter contains Field Collection field with multiple instances, which is used for sub-events. (Please see the scheme)
I use common date field for "Event" nodes and "Sub-event" field collection items, but I can't sort by it, because for "Sub-event" I would need to use relationship, which disables sorting of events.
So my question is: How would you create something like this?
I am using Drupal 7, Wiews 7.3 and latest version of Field Collection.
For better understanding of my problem, please look at following images (sorry, I can't post images because I'm new here):
Scheme
Mockup of view
Answer: I would strongly suggest to move away from field collection implementation. Because, Sub - Events are actually events and should be nodes. Right now they are becoming values of a Single Event by using field collection.
Try considering Entity Reference. Add a reference field called Parent Event to the event content type and make it reference to same content type. Make the field optional so that you can create root Events.
In future, above architecture will provide you lot of advantages. Hope you like it!
Comment: Thanks for your suggestion, seems to be way to go, especially considering the Inline Entity Form module.


Dynamic Select com query

Question: Quero fazer um formulario dinamico igual a este:
https://www.w3.org/WAI/WCAG20/Techniques/working-examples/SCR19/dynselect.html
Estou a ter dificuldades em interligar mysql, php e javascript na seguinte medida:
-O primeiro select as options values são geradas por uma query (Select * from Continent)
-Conforme o Continent selecionado gera automaticamente as opções do Country select fazendo a consulta (Select * from Country where id_continent="id")
Answer: Pode ser utilizado um .getJSON() no seu javascript e retonar uma lista com as opções dos paises ...
<code>$.getJSON("SelecionarPaises.php", data : idcontinente).done(function (paises) {
// vai receber uma lista de paises aqui, ai pode adicionar onde quiser
});
</code>

Dynamic Select com query


Python 2.7 - I'm getting the wrong order of values after converting CSV to JSON

Question: I want to convert CSV to JSON file by taking data from stdin and outputting it to stdout. The input data's format has a format (each line) like this:
    111117_055958,b8:27:eb:da:ef:6d,0,b8:27:eb:8f:ba:38,192.168.1.10,171110_163200,19,0,0,562,18.7,50.1,ON,OFF,44.0,6593,3697,-2.700629,X
My Python code is:
#!/usr/bin/env python
import csv, json , sys

fieldnames = ("Time","MAC_ETH","IP_ETH","MAC_WiFi","IP_WiFi","Up_time","Winsen","ADC_2","ADC_3","VOC","Temp","Humidity","Internet","Lampa_UV","CPU_temp","Amph_P1","Amph_P2","PT100","Brak",)
reader = csv.DictReader(sys.stdin, fieldnames)

for row in reader:
    print(json.dumps(row, sys.stdout))
    sys.stdout.write('\n')

And my output looks like this:
{"ADC_2": "0", "ADC_3": "0", "Up_time": "171110_163200", "MAC_WiFi": "b8:27:eb:8f:ba:38", "CPU_temp": "44.0", "Temp": "18.7", "VOC": "562", "PT100": "-2.700629", "MAC_ETH": "b8:27:eb:da:ef:6d", "Internet": "ON", "Winsen": "19", "Humidity": "50.1", "Amph_P2": "3697", "Amph_P1": "6593", "IP_ETH": "0", "Time": "111117_055958", "Brak": "X", "Lampa_UV": "OFF", "IP_WiFi": "192.168.1.10"}

So names and values of parameters are OK. But the order of parameters is different compared to the input string and order in the fieldnames value. Can you tell me what I am doing wrong?
I use Python 2.7.12 on Synology.
Comment: And it would be in combination with @TalSkverer's dupe, since you're using `DictReader`.
Comment: Actually, [this](https://stackoverflow.com/questions/10844064/items-in-json-object-are-out-of-order-using-json-dumps) may be more appropriate since you're `dump`ing, not `load`ing.
Comment: Answered your question. Hope it helped, if so please mark as correct and consider an upvote.
Answer: A workaround to this problem would be to use a normal csv.reader() and to then construct an OrderedDict() with each row. This can then be passed to json.dumps() to maintain the order:
from collections import OrderedDict
import csv, json, sys

fieldnames = ("Time", "MAC_ETH", "IP_ETH", "MAC_WiFi", "IP_WiFi", "Up_time", "Winsen", "ADC_2", "ADC_3", "VOC", "Temp", "Humidity", "Internet", "Lampa_UV", "CPU_temp", "Amph_P1", "Amph_P2", "PT100", "Brak",)

for row in csv.reader(sys.stdin):
    rowd = OrderedDict(zip(fieldnames, row))
    print(json.dumps(rowd, sys.stdout))
    sys.stdout.write('\n')

This would give you output as:
{"Time": "111117_055958", "MAC_ETH": "b8:27:eb:da:ef:6d", "IP_ETH": "0", "MAC_WiFi": "b8:27:eb:8f:ba:38", "IP_WiFi": "192.168.1.10", "Up_time": "171110_163200", "Winsen": "19", "ADC_2": "0", "ADC_3": "0", "VOC": "562", "Temp": "18.7", "Humidity": "50.1", "Internet": "ON", "Lampa_UV": "OFF", "CPU_temp": "44.0", "Amph_P1": "6593", "Amph_P2": "3697", "PT100": "-2.700629", "Brak": "X"}

Comment: THX. Now it works as should be !
Answer: json / dict does not preserve key order.
In python > 3.6 dict does support preserving key order by default.
However since reader has a .fieldnames which is order you could do the following (UNTESTED):
for row in reader:
    print(json.dumps({f: row[f] for f in reader.fieldnames}, sys.stdout))
    sys.stdout.write('\n')

Or simply use your own declared fieldnames:
for row in reader:
    print(json.dumps({f: row[f] for f in fieldnames}, sys.stdout))
    sys.stdout.write('\n')

Comment: Does not work. Syntax error is rapported because of last bracket


Manage domain by two mail servers

Question: For some reason (maybe stupid, probably it's about money) I need to setup domain to have two mail servers:

some mail addresses would be on Google
some mail addresses would be managed by our server

I know there is no way how to do it by MX records but is there any chance to get it working without several problems? Like (lets get domain.com):

setup MX records of domain.com to google servers
in google administration, create mail accounts that should be managed by google
create domain aliases for others like email@example.com -> firstname.lastname@example.com
setup MX records of f.domain.com to our server
configure domain.com as mail domain on our server
create user email@example.com
create domain alias on ourserver that will f.domain.com -> domain.com
lets that users to logging into system on ourserver and use our SMTP

Will this work? Has anybody better idea how to do it?
note: we are using postfix & dovecot combination on our server
Comment: Unless there's something clever you can do - in that current solution - users on your server will not be able to mail the users on the Google side.  Your server will try and deliver the mail locally (knowing it's the target for domain.com) and then bounce it with unknown recipient.
Comment: In Exchange this is called a shared SMTP namespace.  Not sure if postfix/dovecot/google offer the same type of setup or not though.  The term though might help with your search.
Comment: @EightBitTony yeah, I've missed that... But that can have some solution, not? Would be stupid to create a second (reverse) alias like `g.domain.com`...
Comment: It should be possible, but not easy.  Using masquerade_domains = domain.com in main.cf and using a g.domain.com subdomain with it's own mx record in dns.  It's been a while since I set up a google mail domain and from memory they are a bit restrictive.
Answer: There are two possibilities.

Use a subdomain of domain.com for one of the servers.

e.g.  gm.domain.com - for gmail.
and domain.com for your own server.

Set up all users on your own server, and pull mail from your server through to gmail using POP.  Of course, the gmail side will still need to be on a sub domain.

Not easy is it?  Yes I think it would be possible but it won't be elegant to implement.
If email was to pass out from your own smtp server use the

masquarade_domains = domain.com

option in main.cf.
The only thing is I don't think gmail allows mail relaying.  It would sure help to understand the driver for this.  I'd ask them straight because it may not be entirely possible.
Comment: Would be nice but sent e-mails needs to be from domain.com - meaning that this is not acceptable. Next thing is that client have some paranoia and he don't trust anyone but google (why google?) so e-mails managed by google would be only managers ones... Thanks for apache/nginx (etc.) proxies and other technics, this is annoying (or clients are :-D )
Comment: Basically I think all mail needs to go to your servers and be forwarded for certain users to google.  Then google needs to do the masquerade.  I don't think it's possible but I could be wrong.  I don't have a google mail domain that I'm managing anymore to look at.

Manage domain by two mail servers


How do I combine the error when multiplying a number by a proportion

Question: I wish to know how many seals are in an area. Seals have been counted in a portion of the area, once each month over multiple years. Separately, several seals in the area have been fitted with GPS tracking devices and the proportion of time they were available to be counted in the surveyed portion of the area has been determined.
e.g., Counts for July: (102, 192, 85, 90, 80, 102, 83) n = 7 counts, mean = 105, SD = 39.4
Proportions of time tracked seals spent in the surveyed portion: (0.51, 0.62, 0.20, 0.44, 0.72, 0.52) n = 6 seals, mean = 0.50, SD = 0.18
Estimate of total seals = 105/0.50 = 210
But what is the error on that estimate (SD, SE or Variance)?
Comment: This isn't my field at all. But I see very large uncertainty here. The observed range of seals is itself very wide, [80, 192]. Furthermore, you are trying to infer (by extrapolation) the population from the proportion of time spent in an area by 6 seals. 6 seals out of (what we know can be up to) 192 is very small.
Answer: It's called fractional uncertainty, and for a product or ratio like
\begin{equation}
q=\frac{x \times \cdots \times z}{u \times \cdots \times w},
\end{equation}
the fractional uncertainty of $q$ is
\begin{equation}
\frac{\sigma_q}{|q|}=\sqrt{\left(\frac{\sigma_x}{x}\right)^2 + \cdots + \left(\frac{\sigma_z}{z}\right)^2 + \left(\frac{\sigma_u}{u}\right)^2 + \cdots + \left(\frac{\sigma_w}{w}\right)^2}.
\end{equation}
Thus, the s.d. of $q$ is (after rearranging)
\begin{equation}
\sigma_q=|q|\sqrt{\left(\frac{\sigma_x}{x}\right)^2 + \cdots + \left(\frac{\sigma_z}{z}\right)^2 + \left(\frac{\sigma_u}{u}\right)^2 + \cdots + \left(\frac{\sigma_w}{w}\right)^2}.
\end{equation}
So, for your estimated ratio (105/0.5), the s.d. is
\begin{equation}
\sigma=210 \sqrt{\left(\frac{39.4}{105}\right)^2 + \left(\frac{0.18}{0.5}\right)^2 }
\end{equation}
Comment: Thanks, but that gives an SD on 210 seals of 0.5, which doesn't seem reasonable?
Comment: Apologies, the $\sigma$ needs to be multiplied by the absolute value of the ratio (in case it's negative). See modified answer.


$U$~$N(3,16)$ $V$ ~$\chi_{9}^{2}$ U and V are independent random variables. Find $P(U-3<4.33\sqrt{V})$

Question: $U$~$N(3,16)$
$V$ ~$\chi_{9}^{2}$
U and V are independent random variables.
Find $P(U-3<4.33\sqrt{V})$
(The notes I'm working through don't seem to approach this rigorously...)
The answer is $P(t_{9} <3\times\frac{4.33}{4})$ and I'd appreciate a rigorous solution. where $t_{9}$ is the t distribution with 9 degrees of freedom.
Answer: For independent random variables $Z$ and $V$ it can be proven that $\frac{Z}{\sqrt{\frac{V(m)}{m}}}\sim t_m \quad (1)$
with $Z$ is standard normally distributed, $V$ is Chi-sqared distributed with $m$ the degrees of freedom and $t_m$ is t-distributed with $m$ degrees of freedom as well. For a proof see here.
First of all we have to standardize $U$: $Z=\frac{U-3}{4}\sim \mathcal N(0,1)$
We have $\frac{U-3}4<\frac{4.33\sqrt{V}}4$
We can divide the equation by $\sqrt{V}$
$\frac{U-3}{4\cdot \sqrt{V}}<\frac{4.33}4$
To obtain $(1)$ we have to divide the denominator of by $\sqrt{m}=\sqrt{9}$. This is the same as multiplying the equation by $\sqrt{9}=3$.
$\frac{U-3}{4\cdot \sqrt{\frac{V}{9}}}<3\cdot \frac{4.33}4$, where $\frac{U-3}{\sqrt{4\cdot \frac{V}{3}}}\sim t_9$
Answer: Note that
$$
\frac{U-3}{4}\sim N(0,1)
$$
whence
$$
W=\frac{U-3}{4}\bigg/\sqrt {V/9}\sim t_9
$$
since $U$ and $V$ are independent. Now to compute the probability we note that
$$
U-3<4.33\sqrt{V}\iff W=\frac{3(U-3)}{4\sqrt{V}}<\frac{3}{4}\times 4.33
$$
so
$$
P(U-3<4.33\sqrt{V})=P\left(W<\frac{3}{4}\times 4.33\right)
$$
as desired.


How to keep a unique set of keys with an incremental transformation in Palantir Foundry?

Question: I am trying to save compute on a python transform in Foundry.
I want to run my code incrementally, but I want to keep a unique set of keys, without having to do a full snapshot read on the full dataset, and then run the unique.
If I try something like df_out = df.select("composite_key").dropDuplicates() I am afraid it uses the full dataset input, I want to make use of the previous deduplication I already did.
Answer: The trick here is use the previous version of the output dataset:
    df_out = df.unionByName(
        df_out.dataframe('previous', schema=df.schema).select("composite_key")
    ).drop_duplicates()

Using this pattern, you don't need to do a look up on the full dataset, you use the previously computed unique set of keys, union to the new data and then de-dupe.
Answer: If there are other columns in the new data but you still want to de-dupe by key you can use this approach.
# If there may be duplicates in the data do this step.
# df = df.dropDuplicates(['composite_key'])

df_prev = df_out.dataframe(mode='previous', schema=df.schema)
# This uses the new row for any existing key.
# You could do the opposite by swapping the places of the tables.
existing = df_prev.join(df, on='composite_key', how='leftanti')
result = existing.unionByName(df)


Combinations from two different groups

Question: I was asking myself the next scenario but I cant figure it out how to solve it. We have two group, first one with next characters:
qwerasdf
And the second one with:
zxcdvbnr
If a have to create a string with a substring of three characters from the first group and four characters from the second one (order doesnt matter) and this string cant have repeated characters, how many combinations can be generated?.
For example, a possible solution which is ok:
asdzxcv (three characters of the first group and four for the second, without any character repeated)
a wrong solution:
qwrvbnr (three for the first group and four for the second one, but this time final answer have a repeated character, which is not ok).
I know how to solve when both groups have no common characters, but not this case.
Answer: $\underline{Hint}$
qweasf$\quad\leftarrow$ dr$\rightarrow\quad$ zxcvbn

Combinations from two different groups


Convert Objective-C code to Java code

Question: I need a little help with converting an Obj-C code to Java code.
Obj-C code :
NSArray * tmp = [_user.userDbAdapter executeQuery:query];
if ([[[tmp objectAtIndex:0] objectForKey:@"mediaType"] intValue] == M_COLLECTION_ANTHEM)
{
    mtype = SECTYPE_AUDIO;
}

Java code :
ArrayList<Object> tmp = aUser.userDbAdapter.executeQuery(query);
if(tmp.get(0)){
    //code
}

Is this the right way to do this...and how can I get objectForKey in Java ?
Thanks in advance!
Answer: Your query will return an array of NSDictionary objects in the Obj-C code... in java my best guess would be it returns a array of objects that implement the Map interface (behaves the same way as the NSDictionary class: http://download.oracle.com/javase/1.4.2/docs/api/java/util/Map.html).
You can then use tmp.get(0).get("mediaType") (if the objects in the array implement the Map interface that is).
List<Map>* tmp = aUser.userDbAdapter.executeQuery(query);
if (tmp.get(0).get("mediaType").intValue() == M_COLLECTION_ANTHEM)
{
    mtype = SECTYPE_AUDIO;
}

Comment: That's because the objects in the List are not Maps... do this to figure out what they are: tmp.get(0).class.toString() <- print that to something where you can see it!
Comment: Damn, I didn't know that I can use get two times.Thanks really for the help!
Comment: There you go. Code edited and done for you. I think that should working (I am assuming the array returned contains objects implementing the Map interface though).
Comment: First get: List interface
Second get: Map interface
Comment: Actually If I use List, and after I change the type of query to List I can't use the both get methods. If i stick to the ArrayList type I still can't use the second get method....

Convert Objective-C code to Java code


Prove that the equation $u^a - u^b = n!$ has finitely many solutions $(a,b,n)$ in positive integers.

Question:
(China TST 2004 Day 1). Let $u$ be a fixed integer. Prove that the equation $u^a - u^b = n!$ has finitely many solutions $(a,b,n)$ in positive integers. Here is the original source of the problem.

Note: it's possible one has to additionally assume that $u\neq -1$. If it turns out that the claim is false for some $u\neq -1$, then just assume $u$ must be positive for convenience.

For a positive integer k and prime p, let $v_p(k)$ denote the highest exponent of p dividing k. If $u=0,1$, there are no solutions so assume $u\neq 0,1$. If $u=-1$, then $u^a - u^b$ can only take on the values $0, 2,$ and $-2$. But $2$ can be obtained infinitely many times, namely whenever a is even and b is odd, so it seems we need to assume $u\neq -1$. First consider the case when $u > 1$. Then $a > b$ clearly, so $u^b$ must divide $n!$. Also, $u^{a-b}-1 \ge 1$ is coprime to $u$ and divides $n!$. Equality only holds when $u=2$ and $a=b+1$. But in that case, we have $2^b = n!$, which only holds for $b=1$ as $n!$ is divisible by $3$ for $n\ge 1$. Since we're only interested in showing the given equation has finitely many solutions, we may as well assume $u^{a-b}-1 > 1$ so that it has a prime factor. I know that for any prime p, $v_p(n!) = \sum_{i\ge 1}\lfloor n/p^i\rfloor$. In particular, if we let $p$ be a prime dividing $u$ with $v_p(u) = k$, then from above, $v_p(n!) = kb$.  $n!$ grows faster than $u^n$ as $n\to\infty$ for any integer u. Clearly we cannot have $a = b,$ so it could be useful to consider when $a<b$ and when $a > b$ separately. It could be useful to use inequalities and show that when either $a$ or $b$ or $n$ exceed a certain bound, there are no more solutions. I know a fairly nontrivial fact that the last nonzero digit of $n!$ actually forms a nonperiodic sequence, though that's likely not useful for this problem. I'm also unsure whether it'll be useful to consider orders of powers of primes.
Comment: Related [AoPS thread](https://artofproblemsolving.com/community/c6h55883p346573)
Comment: @JohnOmielan That is an interesting observation, it might be that organizers missed this, but I think it is likely that whoever translated and typed the "official" source to LaTeX simply made a mistake and omit the word "positive" (since the original was in Chinese and typeset to LaTeX by someone else later, the page contains typeset TST's from 1986 too...). I guess we would have to see the original to know.
Comment: Have you tried figuring out how to express $n! = 2^{a_1}3^{a_2}3^{a_3}..... p_k^{a_k}$ yet.  For instance $10! = 22.214.171.124.220.127.116.11.9.10=18.104.22.168.10.3.5.7.9=2^220.127.116.11.22.214.171.124.9=2^{5+2}.126.96.36.199.5.7.9=2^{5+2+1}.184.108.40.206.7.9=2^{5+2+1}3^{2}.18.104.22.168=2^{5+2+1}3^{2+1}5^2.7$.
Comment: @Sil Interestingly, that AoPS thread says the problem is from the same source as what the OP stated, but the although the OP provided link provided has the question starting with "Let $u$ be a fixed integer", the thread's question starts with "Let $u$ be a fixed *positive* integer" (with the rest of the question also being a bit different, e.g., using $\alpha$ and $\beta$ instead of $a$ and $b$). I wonder if that was a question fix that was perhaps added later because, as the OP has noted, the problem is not true for $u = -1$. Also, note the same question is also being asked, but with no ...
Comment: @Sil (cont.) answers, at the AoPS threads [$u^{a}-u^{b}=n!$ has finitely many solutions](https://artofproblemsolving.com/community/c6h1845576p12424547) and [Old and nice](https://artofproblemsolving.com/community/c6h1287902p6800176), with the first one also being from the same problem set, and it *not* requiring $u$ to be positive, while the second one uses different variables, but also specifying the value, $b$ in this case, to just be an integer.
Comment: @user33096 FYI, using an [Approach0 search](https://approach0.xyz/search/?q=OR%20content%3A%24u%5Ea-u%5Eb%3Dn!%24%2C%20OR%20content%3Afinitely%2C%20OR%20content%3Apositive%20integers&p=1), in addition to finding the AoPS thread Sil mentioned, as well as the other $2$ related AoPS threads, there's also the specific question for $u = 2$, with it asking for all solutions here, at [Finding every triplet $(n,a,b)$ such that $n!=2^a-2^b$](/q/919173).
Answer: You've made a good start, including suggesting possibly using the orders of the powers of primes. Actually, the solution below uses the multiplicative order of a prime.
As Sil's comment indicates, the AoPS thread Chinese factorial equation references the same problem, with that thread providing several solutions having varying degrees of validity. I'm adapting (including making a few minor corrections) and expanding on the approach used in Post #$12$.
For any $u \gt 1$, rewrite the equation as
$$n! = u^b(u^m - 1) \tag{1}\label{eq1A}$$
where $m = a - b \gt 0$. Next, note there's at most one solution for any fixed $n$ since, for each prime factor $q$ of $u$, we require $b = \frac{\nu_q(n!)}{\nu_q(u)}$. If just one integral value of $b$ works in all cases, then $a$ would be uniquely determined from \eqref{eq1A} based on $b$ and $n$, with there then being just one solution if $a$ is an integer. Thus, it's sufficient to show that only finitely many $n$ are solutions, with this answer doing this by proving that $n$ is bounded above.
Consider any prime $p$ which doesn't divide $u$, with $d$ being the multiplicative order of $u$ modulo $p$, i.e., $d = \operatorname{ord}_p(u)$. For any $n$ at least as large as the smallest prime factor of $u$ (so $b \gt 0$), and also with $n \ge p$, we have $p \mid n! \; \to \; p \mid u^m - 1$, so $d \mid m$. Since $u^m = \left(u^{d}\right)^{m/d}$, then by the Lifting-the-exponent lemma,
$$\nu_p(u^m-1) = \nu_p(u^d-1) + \nu_p\left(\frac{m}{d}\right) \le \nu_p(u^d-1) + \log_p\left(\frac{m}{d}\right) \tag{2}\label{eq2A}$$
With the constant
$$c_1 = \nu_p(u^d-1) - \log_p(d) \tag{3}\label{eq3A}$$
as well as using $\log_p\left(\frac{m}{d}\right) = \log_p(m) - \log_p(d)$, we get from \eqref{eq2A} that
$$\nu_p(u^m-1) \le \log_p(m) + c_1 \tag{4}\label{eq4A}$$
Next, Legendre's formula gives that
$$\nu_p(u^m-1) = \nu_p(n!) = \sum_{i=1}^{\infty}\left\lfloor\frac{n}{p^{i}}\right\rfloor \gt \frac{n}{p} - 1 \tag{5}\label{eq5A}$$
Combining \eqref{eq4A} and \eqref{eq5A} results in
$$\log_p(m) + c_1 \gt \frac{n}{p} - 1 \; \; \; \to \; \; \; m \gt p^{\frac{n}{p} - c_2} \tag{6}\label{eq6A}$$
where $c_2 = c_1 + 1$. We then get
$$n^n \gt n! = u^b(u^m - 1) \gt u^m \gt u^{p^{\frac{n}{p} - c_2}} \tag{7}\label{eq7A}$$
Taking the natural logarithm of each side gives
$$n\ln(n) \gt p^{\frac{n}{p} - c_2}\ln(u) \tag{8}\label{eq8A}$$
However, this can be true for at most a finite number of $n$ since the left side is less than a polynomial of $n$ (e.g., $n^2$), but the right side is a growing exponential function in $n$.
Comment: Thanks. Out of curiosity, are there still only finitely many solutions when $u< -1$?
Comment: @user33096 You're welcome. Regarding the case where $u \lt -1$, let $u = -v$ so $v \gt 1$. Then depending on the parity of $a$ and $b$, we have $u^a - u^b$ becoming one of $v^a + v^b$, $v^a - v^b$, $-v^a + v^b$ and $-v^a - v^b$. The second & third cases are basically the same as in my solution, while the fourth one has no solution. With the first one, if $a = b$, then we have $2v^a$ which has a finite # of solutions since for any odd prime $p \nmid v$, there's no solutions when $n \ge p$. Otherwise, WLOG, have $b \lt a$. Then $n!=v^b(v^m+1)$. Then basically proceed as I did earlier. ...
Comment: @user33096 In particular, for any odd prime $p \nmid v$, we now have $v^m \equiv -1\pmod{p} \; \to \; v^{2m} \equiv 1 \pmod{p}$. Since $v^{2m}-1=(v^m+1)(v^m-1)$, then $p \nmid v^m - 1$, so $\nu_p(v^m+1) = \nu_p(v^{2m}-1)$. Thus, continue similarly as I did, replacing $m$ with $2m$ in most cases, to end up with a similar conclusion at the end.
Comment: Would you mind joining [my chat room](https://chat.stackexchange.com/rooms/141092/user33096s-chat-room) and answering some hopefully quick questions I have?


Compactness result PDEs

Question: There is an argument that I see is used often in Evans PDE book, that I don't really get.  We take a bounded sequence, say $(u_m) \in W^{1,q}(\Omega)$.  By some functional analysis results, we know that since this is a reflexive Banach space, there is a subsequence that converges weakly to some element, say $u_{m_j} \rightharpoonup u$ .  But now, there is the claim that this subsequence converges strongly in $L^q$.  Why is this?  I'm assuming that it has to do with the Rellich compactness result, $W^{1,p} \subset \subset L^q$ for all $1\leq q< p^{*}=\frac{np}{n-p}$.    How does this argument work then?
Answer: Let's begin with a weakly convergent sequence:  $u_j \rightharpoonup u$ in $W^{1,p}$, $1<p<\infty$. A weakly convergent sequence is bounded. By the Rellich-Kondrachov theorem, it has a subsequence $u_{j_k}$ that converges strongly in $L^p$. How do we conclude that the entire sequence $u_j$ converges strongly in $L^p$?
Note that we also have $u_j \rightharpoonup u$ in $L^p$. Therefore, the limit of $(u_{j_k})$ is nothing but $u$, because strong convergence implies weak, and weak limits are unique (the weak topology is Hausdorff). It remains to invoke an easy fact from general topology: if $u,u_j$ are elements of a Hausdorff space and every subsequence of $(u_j)$ has a subsubsequence that converges to $u$, then $u_j\to u$. (Proof is by contradiction, and is left as an exercise).
Answer: Compact linear operators on Banach spaces map weakly convergent sequences to strongly convergent sequences; such operators are also called completely continuous. The embedding $W^{1,q} \to L^q$ is such an operator.

Compactness result PDEs


Como criar um webgrid com colunas dinâmicas?

Question: Vi vários exemplos usando WebGrid mas todos com a quantidade de colunas fixas. Minha fonte de dados vai retorna uma lista dinâmica, ou seja, ora retorna 5 colunas e depois pode retornar 6 colunass, como fazer ?
Answer: Algo assim:
@model IEnumerable<dynamic>

@{
    WebGrid grid = new WebGrid(source: Model, rowsPerPage: 5, canPage: true);

    List<WebGridColumn> cols = new List<WebGridColumn>();
    cols.Add(grid.Column("ColunaManual", "Coluna Manual"));
    foreach (Dictionary<string, string> linha in Model.Select(m => m.ToPropertyDictionary()))
    {
        cols.Add(grid.Column(linha.Key, linha.Key,
            format: item => Html.Raw("<text>" + linha.Value + "</text>")
        ));
    }
}

ToPropertyDictionary() está aqui:
public static class DictionaryExtensions
{
    public static Dictionary<string, string> ToPropertyDictionary(this object obj)
    {
        var dictionary = new Dictionary<string, string>();
        foreach (var propertyInfo in obj.GetType().GetProperties())
            if (propertyInfo.CanRead && propertyInfo.GetIndexParameters().Length == 0)
                dictionary[propertyInfo.Name] = (propertyInfo.GetValue(obj, null) ?? "").ToString();
        return dictionary;
    }
}

Comment: entendi, o `Modelo` é convertido em um tipo `Dictionary`, mas agora eu me dei conta do seguinte: os dados com a quantidade dinâmica de atributos virão de uma **Procedure** e como fazer um **Modelo** pra refletir essa quantidade dinâmica de atributos ? Tem como ?, por exemplo na classe de contexto terei que fazer algo assim: `modelBuilder.Entity().MapToStoredProcedures();`. Os dados veêm de um a procedure devido a uma função Pivot que é utilizado nessa proc.
Comment: Faça outra pergunta, por favor.
Comment: Obrigado @Cigano Morrison Mendez eu criei um nova pergunta [link] https://pt.stackoverflow.com/questions/205530/atributos-din%C3%A2mico-de-um-modelo-entidade [link] .


Conflito de dependências em JSF com Maven

Question: Criei um projeto Maven no Eclipse com JSF 2, projeto vem evoluindo bem, rodando no Tomcat 8 sem problemas e foi ai que eu comecei a implementar os relatórios então coloquei as dependências para o Jasper conforme código abaixo, a partir dai o tomcat não sobe, entendo que isto é conflito das minhas dependências e que preciso excluir então já vasculhei a documentação e vi videos sobre o assunto mais não tive sucesso.
Neste projeto uso Spring Security, Hibernate etc....
<code>    <dependency>
        <groupId>net.sf.jasperreports</groupId>
        <artifactId>jasperreports</artifactId>
        <version>6.0.3</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>net.sf.jasperreports</groupId>
        <artifactId>jasperreports-fonts</artifactId>
        <version>6.0.0</version>
        <scope>compile</scope>
    </dependency>
</code>
Já olhei a hierarquia do pom ( o que não é fácil meu ),  Aqui segue minhas dependências completas sem colocar o Jasper e ai o projeto sobe sem problemas se alguém puder dar uma dica de como eu posso proceder agradeço.
<code>    <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

    <!-- Java -->
    <java.version>1.8</java.version>

    <!-- Primefaces -->
    <primefaces.version>6.0</primefaces.version>

    <!-- Hibernate / JPA -->
    <hibernate.version>4.3.10.Final</hibernate.version>

    <!-- Validator -->
    <validator.version>5.2.3.Final</validator.version>

    <!-- Weld - Implementação do CDI -->
    <weld.version>2.3.5.Final</weld.version>

    <!-- JSF Version -->
    <javax.faces.version>2.2.10</javax.faces.version>

</properties>

<repositories>
    <repository>
        <id>prime-repo</id>
        <name>PrimeFaces Maven Repository</name>
        <url>http://repository.primefaces.org</url>
        <layout>default</layout>
    </repository>
</repositories>

<dependencies>

    <dependency>
        <groupId>org.primefaces</groupId>
        <artifactId>primefaces</artifactId>
        <version>${primefaces.version}</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.primefaces.themes</groupId>
        <artifactId>bootstrap</artifactId>
        <version>1.0.10</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.primefaces.themes</groupId>
        <artifactId>bluesky</artifactId>
        <version>1.0.10</version>
        <scope>compile</scope>
    </dependency>

    <!-- Usado para geração de arquivos Excel -->
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi</artifactId>
        <version>3.11</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.glassfish</groupId>
        <artifactId>javax.faces</artifactId>
        <version>${javax.faces.version}</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.jboss.weld.servlet</groupId>
        <artifactId>weld-servlet</artifactId>
        <version>${weld.version}</version>
        <scope>compile</scope>
    </dependency>

    <!-- Weld depende do Jandex -->
    <dependency>
        <groupId>org.jboss</groupId>
        <artifactId>jandex</artifactId>
        <version>2.0.2.Final</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.hibernate</groupId>
        <artifactId>hibernate-validator</artifactId>
        <version>${validator.version}</version>
        <scope>compile</scope>
        <exclusions>
            <exclusion>
                <groupId>org.jboss.logging</groupId>
                <artifactId>jboss-logging</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>org.hibernate</groupId>
        <artifactId>hibernate-core</artifactId>
        <version>${hibernate.version}</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.hibernate</groupId>
        <artifactId>hibernate-entitymanager</artifactId>
        <version>${hibernate.version}</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>javax.servlet</groupId>
        <artifactId>javax.servlet-api</artifactId>
        <version>3.1.0</version>
        <scope>provided</scope>
    </dependency>

    <!-- Spring Security (autenticação e autorização) -->
    <dependency>
        <groupId>org.springframework.security</groupId>
        <artifactId>spring-security-core</artifactId>
        <version>3.2.5.RELEASE</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.security</groupId>
        <artifactId>spring-security-web</artifactId>
        <version>3.2.5.RELEASE</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.security</groupId>
        <artifactId>spring-security-config</artifactId>
        <version>3.2.5.RELEASE</version>
    </dependency>

    <!-- Database dependencies -->

    <!-- PostgresSQL dependencies -->
    <dependency>
        <groupId>commons-dbcp</groupId>
        <artifactId>commons-dbcp</artifactId>
        <version>1.3</version>
        <exclusions>
            <exclusion>
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging</artifactId>
            </exclusion>
            <exclusion>
                <groupId>xml-apis</groupId>
                <artifactId>xml-apis</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>postgresql</groupId>
        <artifactId>postgresql</artifactId>
        <version>9.1-901.jdbc4</version>
        <scope>provided</scope>
    </dependency>

    <!-- retrofit -->
    <dependency>
        <groupId>com.squareup.retrofit</groupId>
        <artifactId>retrofit</artifactId>
        <version>1.9.0</version>
    </dependency>

    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
        <scope>test</scope>
    </dependency>

    <!-- SLF4J <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId>
        <version>1.7.7</version> </dependency> -->
    <!-- Commons Logging (abstrai a implementação de logging) -->
    <dependency>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
        <version>1.1.3</version>
        <scope>compile</scope>
    </dependency>

    <!-- Commons Lang (utilidades) -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.1</version>
        <scope>compile</scope>
    </dependency>

    <!-- Log4J -->
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.17</version>
        <scope>compile</scope>
    </dependency>

    <!-- omnifaces -->
    <dependency>
        <groupId>org.omnifaces</groupId>
        <artifactId>omnifaces</artifactId>
        <version>1.14</version>
    </dependency>

    <dependency>
        <groupId>org.easymock</groupId>
        <artifactId>easymock</artifactId>
        <version>3.4</version>
        <scope>test</scope>
    </dependency>

    <!--google gson -->
    <dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
        <version>2.6.2</version>
    </dependency>

    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.4</version>
        <scope>compile</scope>
    </dependency>

</dependencies>
</code>
Answer: Utilize a tag exclusions do Maven. Se você estiver utilizando o Eclipse como IDE.
Por exemplo: na aba Dependency Hierarchy você poderá ver todas as dependências em formato de árvore.
Caso alguma dependência apareça mais de uma vez dê preferência para a mais atual e "exclua" a obsoleta.
<code>    <dependency>
        <groupId>com.group.name</groupId>
        <artifactId>project</artifactId>
        <version>1.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>com.anothergroup.name</groupId>
                <artifactId>projeto-em-conflito</artifactId>
                <version>1.0.0</version>
            </exclusion>
        </exclusions>
    </dependency>
</code>


text to speect AZURE add silence

Question: Hi I'm trying to add a pause in the text to speech Azure api, this code works fine but I can't add silence. I need to add silence when a specific character is found in the text, I tried with  or <mstts:ttsbreak strength="none" /> but the audio output speech the tags instead to make a pause.
       $doc = new DOMDocument();

       $root = $doc->createElement( "speak" );
       $root->setAttribute( "xmlns" , "http://www.w3.org/2001/10/synthesis" );
       $root->setAttribute( "xmlns:mstts" , "http://www.w3.org/2001/mstts" );
       $root->setAttribute( "xmlns:emo" , "http://www.w3.org/2009/10/emotionml" );
       $root->setAttribute( "version" , "1.0" );
       $root->setAttribute( "xml:lang" , "$spechlang" );

       $voice = $doc->createElement( "voice" );
       $voice->setAttribute( "name" , "$myvoice");

       $style = $doc->createElement( "mstts:express-as" );
       $style->setAttribute( "style" , "whispering"); //

       $prosody = $doc->createElement( "prosody" );
       $prosody->setAttribute( "rate" , "$rate.00%" );
       $prosody->setAttribute( "pitch" , "$pitch.00%" );

       $text = $doc->createTextNode( "$mytext" );

       $prosody->appendChild( $text );
       $style->appendChild( $prosody );
       $voice->appendChild( $style );
       $root->appendChild( $voice );
       $doc->appendChild( $root );
       $data = $doc->saveXML();

       $options = array(
        'http' => array(
            'header'  => "Content-type: application/ssml+xml\r\n" .
                    "X-Microsoft-OutputFormat: riff-24khz-16bit-mono-pcm\r\n" .
                    "Authorization: "."Bearer ".$access_token."\r\n" .
                    "X-Search-AppId: 07D3234E56TT426DAA29772419F436CA\r\n" .
                    "X-Search-ClientID: 1ECFAE91406677A480F00935DC390960\r\n" .
                    "User-Agent: TTSPHP\r\n" .
                    "content-length: ".strlen($data)."\r\n",
            'method'  => 'POST',
            'content' => $data,
            ),
        );

        $context  = stream_context_create($options);

        // get the wave data
        $result = file_get_contents($ttsServiceUri, false, $context);

a ssml generated could be
<speak xmlns="http://www.w3.org/2001/10/synthesis" xmlns:mstts="http://www.w3.org/2001/mstts" xmlns:emo="http://www.w3.org/2009/10/emotionml" version="1.0" xml:lang="it-IT">
<voice name="en-US-JennyNeural">
1. first point: <break strength="medium" />
lore lipso bla bla.
</voice></speak>

this SSML work fine on https://speech.microsoft.com/portal SPEECH STUDIO PORTAL but not in my php. thanks
Answer: I believe break does not use the mstts namespace. However, silence does. Here is the official docs:  https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-synthesis-markup-structure#add-silence
Answer: This is a full SSML script with the break  element.  It does not use the mstts namespace. The time is in milliseconds 1 sec = 1000
<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="string" xmlns:mstts="https://www.w3.org/2001/mstts">
  <voice  name="en-AU-NatashaNeural">
I have to think about it. Wait two seconds
<break  time="2000"/>
Yes I agree
</voice>
</speak>

text to speect AZURE add silence


Map characters a strings into dictionary with index as key

Question: i am new in python coding, I am trying to map characters a strings into dictionary with index as key.
I'm using a string "asleep" and want to get output as
{0: 'a', 1: 's', 2: 'l', 3: 'e', 4: 'e', 5: 'p'}

I'm using the following code:
myWord = "asleep"

char_count = map(lambda x: dict(enumerate(x)), myWord)

print(char_count)

Output:
{'a': 1, 's': 1, 'l': 1, 'e': 2, 'p': 1}

Comment: Why do you want to this though? I don't see any viable reason to doing this.
Answer: Simplest solution:
>>> word = 'asleep'
>>> dict(enumerate(word))
{0: 'a', 1: 's', 2: 'l', 3: 'e', 4: 'e', 5: 'p'}

This works because enumerate returns an iterable of pairs like (0, 'a'), (1, 's'), and so on; and the dict constructor works with an iterable of (key, value) pairs.
Comment: I was doing `{idx:char for idx, char in enumerate(myWord)}`, but this is even better :-)
Answer: You could try the following line:
dict(zip(range(len(myWord)), myWord))


Determining beta distribution parameters $\alpha$ and $\beta$ from two arbitrary points (quantiles)

Question: Suppose I have two points $(p_1,x_1)$ and $(p_2,x_2)$ where $p_i$ is a probability on the beta CDF and $x_i$ is a value on that same CDF.  How would I go about determining the beta distribution shape parameters $\alpha$ and $\beta$ with only these two points?
To clarify, I'm assuming I already know the lower and upper bounds of the distribution L and U.
Comment: @whuber Apologies, I am referring to a scaled, recentered beta.  The points would specified exactly.  I had thought this might require numerical methods, but I wasn't sure and wanted to check if there might be an analytic solution.
Comment: By definition a Beta distribution has bounds $0$ and $1$, so apparently you are referring to a scaled, recentered Beta.  No matter: what is of import is how you obtained the numbers $p_1, p_2, x_1, x_2$.  Are these given theoretically (that is, *exactly*), or have one or more of them been measured with some possibility of error?  In all cases you will need to resort to numerical methods, but *which* methods are appropriate to use and how the output is interpreted depends on these distinctions.
Answer: Solution
The transformation
$$(p, x) \to (p, (x-L)/(U-L))$$
converts these points to ones on CDFs for a Beta$(\alpha,\beta)$ distribution.  Assuming this has been done (so we don't need to change the notation), the problem is to find $\alpha$ and $\beta$ for which
$$F(\alpha, \beta; x_i) = p_i, i = 1, 2$$
where the Beta CDF equals
$$F(\alpha,\beta; x) = \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\int_0^x t^{\alpha-1}(1-t)^{\beta-1} dt.$$
This is a nice function of $\alpha \gt 0$ and $\beta \gt 0$: it is differentiable in those arguments and not too expensive to evaluate.  However, there exists no closed-form general solution, so numerical methods must be used.

Implementation Tips
Beware: when both $x_i$ are at the same tail of the distribution, or when either is close to $0$ or $1$, finding a reliable solution can be problematic.  Techniques that are helpful in this circumstance are

Reparameterize the distributions to avoid enforcing the "hard" constraints $\alpha\ge 0, \beta \ge 0$.  One that works is to use the logarithms of these parameters.
Optimize a measure of the relative error.  A good way to do this for a CDF is to use the squared difference of the logits of the values: that is, replace the $p_i$ by $\text{logit}(p_i) = \log(p_i/(1-p_i))$ and fit $\text{logit}(F(\alpha,\beta;x_i))$ to these values by minimizing the sum of squared differences.
Either begin with a very good estimate of the parameters or--as experimentation shows--overestimate them.  (This may be less necessary when following guideline #2.)

Examples
Illustrating these methods is the following sample code using the nlm minimizer in R.  It produces perfect fits even in the first case, which is numerically challenging because the $x_i$ are small and far out into the same tail.  The output includes plots of the true CDFs (in black) on which are overlaid the fits (in red).  The points on the plots show the two $(x_i, p_i)$ pairs.
This solution can fail in more extreme circumstances: obtaining a close estimate of the correct parameters can help assure success.

#
# Logistic transformation of the Beta CDF.
#
f.beta <- function(alpha, beta, x, lower=0, upper=1) {
  p <- pbeta((x-lower)/(upper-lower), alpha, beta)
  log(p/(1-p))
}
#
# Sums of squares.
#
delta <- function(fit, actual) sum((fit-actual)^2)
#
# The objective function handles the transformed parameters `theta` and
# uses `f.beta` and `delta` to fit the values and measure their discrepancies.
#
objective <- function(theta, x, prob, ...) {
  ab <- exp(theta) # Parameters are the *logs* of alpha and beta
  fit <- f.beta(ab[1], ab[2], x, ...)
  return (delta(fit, prob))
}
#
# Solve two problems.
#
par(mfrow=c(1,2))
alpha <- 15; beta <- 22 # The true parameters
for (x in list(c(1e-3, 2e-3), c(1/3, 2/3))) {
  x.p <- f.beta(alpha, beta, x)        # The correct values of the p_i
  start <- log(c(1e1, 1e1))            # A good guess is useful here
  sol <- nlm(objective, start, x=x, prob=x.p, lower=0, upper=1,
             typsize=c(1,1), fscale=1e-12, gradtol=1e-12)
  parms <- exp(sol$estimate)           # Estimates of alpha and beta
  #
  # Display the actual and estimated values.
  #
  print(rbind(Actual=c(alpha=alpha, beta=beta), Fit=parms))
  #
  # Plot the true and estimated CDFs.
  #
  curve(pbeta(x, alpha, beta), 0, 1, n=1001, lwd=2)
  curve(pbeta(x, parms[1], parms[2]), n=1001, add=TRUE, col="Red")
  points(x, pbeta(x, alpha, beta))
}

Comment: @whuber I'm a little confused here. It seems like the objective function relies on knowing the true parameter values in order to fit, which seems circular because if you knew the original parameter values then why fit at all? To me it seems the objective function requires prob which is provided as x.p which is itself a transform of pbeta with arguments  that are the original parameter values?
Comment: @whuber, why do you choose to use logits? I see why we would use the logarithms to remove the problem of the >0 constraint. I'm not quite certain what motivates the use of the logit. Could you please elaborate?
Comment: @Jan The log is useless when $p_i$ is close to $1.$  Moreover, the logit treats probabilities symmetrically in the sense that its derivative at $p$ is the same as its derivative at $1-p$ and the Beta family is similarly symmetric in the sense that whenever $F$ is a Beta distribution, so is the function $x\to 1-F(1-x).$  Although the logarithm alone might work, both these characteristics suggest the logit will be even better and it's scarcely any more expensive to compute.
Comment: @parvin You shouldn't be making any changes at all. As explained at the outset of this answer, the first thing my solution does is transform a scaled, shifted Beta variable into a standard Beta variable. In the code, the default parameters for `l.beta` (namely, `lower=0` and `upper=1`) are for a standard Beta.
Comment: @parvin Do it in three steps. **1: Initialize** the quantiles and the *logits* of the probabilities (as described in the text of this answer): `x <- c(.3, .8); x.p <- (function(p) log(p/(1-p)))(c(0.025, 0.975))`. **2: Solve:** `start <- log(c(1e1, 1e1));
sol <- nlm(objective, start, x=x, prob=x.p, lower=0, upper=1,
           typsize=c(1,1), fscale=1e-12, gradtol=1e-12);
parms <- exp(sol$estimate)`. **3: Check:** `qbeta(p = c(.025, .975), parms[1], parms[2])`.  The output will be "[1] 0.3 0.8", confirming the solution. Be sure to inspect `sol$code` to confirm convergence.
Comment: @parvin I just gave you *complete, working* code.  It will execute almost instantaneously if you have first run the example code in my answer.  If you have further questions about how to use `R`, might I suggest you inquire on [SO]?
Comment: Dear whuber, Thank you very much, very thorough answer, as always. I was wondering (and I can post a question if my question would have a possible answer) though if given some empirical information about the (***quantiles, mean, and variance***), we can find the  **$\alpha~ and ~\beta$** for a reasonably comparable beta-distribution that would have the *similar quantiles, mean, and variance* as provided by the empirical information?
Comment: @parvin It depends on what precisely is meant by "empirical information."  If you are referring to these properties of a *sample*, then in principle you could estimate $\alpha$ and $\beta$ using Maximum Likelihood.  This requires being able to compute the joint probability density for the specific statistics you have.
Comment: So, suppose I collect a sizable amount of information about the variability in a naturally [0, 1] parameter from a large pool of previously published research. I simply make a histogram of these obtained parameter values, and obtain quantiles, mean, and variance for this histgram. My goal is to use this empirical information and find an equivalent Beta distribution whose quantiles, mean, and variance match (are relatively close to) my initial histogram ? So, it's a kind of like a prior-finding problem.
Comment: @parvin You would be much better off estimating the parameters from all the data, rather than from a summary of them.
Comment: I see, Any specific method you might have in mind, are you referring to "empirical Bayes"?
Comment: @Stats It's no different than solving the equation $f(x)=y$ for $x$ given $y$: the solver systematically (and efficiently) produces a sequence of *trial* values $x_1, x_2, \ldots,$ that are plugged into $f$ until one is found for which the output is sufficiently close to $y$.  There's nothing circular about that.  I'm doing exactly the same thing where $x$ is a two-component vector and $y$ is a two-component vector.
Comment: @epp To answer your question: `x` and `x.p` are the values we are supposed to have already. For the purpose of this example, the answerer is creating their own `x` and `x.p` using `x in list(c(1e-3, 2e-3), c(1/3, 2/3))` and `x.p <- f.beta(alpha, beta, x)`. Neither of these are needed in your own code