Is it a valid claim, that by differencing a time series, it loses its memory, and as a result its predictive power?

Question: Marcos Lopez de Prado seems to be a well known and renowned machine learning expert in the field of finance.
I am very far from his level, as have not yet finished my PhD in economics, and only have an applied level statistical knowledge. I have encountered a much cited paper of Lopez de Prado. I can not say I completely understand all the mathematical parts. But there are some claims in the paper, which seem to completely contradict things, which I have learned on statistics and economics thus far, or at least seem to be illogical for me.
For a specific example, the paper under the section Pitfall #4 and Solution #4 suggests, that by differencing time series to make them stationary for classical statistical models (ARIMA etc.) removes the memory of the series and thus makes them lose the predictive power:

The conclusion is that, for decades, most empirical studies have
  worked with series where memory has been unnecessarily wiped-out. The
  reason this is a dangerous practice is that fitting a memory-less
  series will likely lead to a spurious pattern, a false discovery.
  Incidentally, this over-differentiation of time series may explain why
  the Efficient Markets Hypothesis is still so prevalent among academic
  circles: Without memory, series will not be predictive, and
  researchers may draw the false conclusion that markets are
  unpredictable.

In economics there really is a simplified theoretical model on equity returns, which posits, that it is a memory-less white noise, and the prices (the integrated returns) follow a random walk.
But from an empirical perspective, as far as I understand the memory-less attribute of returns pertains only to the individual data points themselves, not the series as a whole. A differentiated series still should have a "collective" memory put together, and it has almost the same information as the integrated version, only lacks a constant value. So it should have the same predictive power as well, should it not? Or it is me, who has a lack of understanding?
Comment: Why is nobody talking about fractional differentiation https://link.springer.com/article/10.1007/s12572-021-00299-5
Comment: BTW  (+1) Good question.
Answer: This tries to answer the original question and not get into Marcos's paper etc. If you think that the level of a variable ( say log price ) has information, then differencing the series ( to obtain returns ), throws out information. If you don't think that the level has information, then differencing is fine. Engle and Granger in their 1987  econometric paper showed how it is possible to consider both levels and changes in the relationship between two series ( X and Y) through the use of an ECM. But it does not mean that there can't be cases where one doesn't care about the levels and is only interested in changes ( or vice versa ).
On a different note, here's a piece of advice. Whenever you read anything about strategies and techniques and approaches in finance don't put too much weight in them because, if the author REALLY TRULY HAS SOMETHING THAT WORKS, he-she is not going to divulge it anyway. Most of the stuff you read will be purposely vague and general and unless you know the details of what the person actually does, not terribly useful. That's not to say that Marcos doesn't write interesting papers but he's not going to tell you what he actually does so  it's best to read his or anyone's presentations with that in mind.
Comment: +1. My intuition is that differencing is a stronger statement than "doesn't contain information" but rather is asking the model to average across what might be different regimes (high/low/etc), integrating out a nuisance variable, as it were.
Comment: Hi Wayne: To be honest, I'm not sure what Marcos is referring to when he says memory is wiped out. Possibly he's referring to where people use rolling windows so that information in the far past is ignored ?
Comment: gotcha. yes, regime changing also makes things difficult.
Comment: I think stationarity wipes memory by making snippets of the time series appear the same (in some summary, at least). So your process could've experienced a low regimine and then a high one, but that history (memory) is wiped out by differencing. It may have also acted differently under those two regimes, but your ARMA model will be stuck averaging across them, oblivious to the fact they existed.
Comment: Thinking from the perspective of TS models, if a certain pattern of returns has some signal, this will surely more readily be observed and learnt from a sequences of some form of differences, and therefore differencing would be preferable.
Answer: Since my similar question was flagged as duplicate (good debate in the comments!), I came across Simon Kuttruf's explanation on Medium:

for integer orders of differencing only a (small) finite set of past
  values is reflected in the resulting differenced series: the preceding
  value in first order differencing, two preceding values for second
  order differencing etc. While for fractional orders of differencing,
  all coefficients take on (asymptotically small) non-zero values and so
  past values get mixed into the differenced series, up to some chosen
  cutoff. This phenomenon is here referred to as 'long memory' (or
  'wipe-out of memory' resp).

If the fraction $d = .5$, then the first four coefficients (check my calculations) according to the recursive formula (see Simon's article)
$$
w_k = -w_{k - 1} \left( \frac{d - k + 1}{k} \right),
$$
the first four values would be 1, -.5, -1/8, and -1/16, leading to the transformation
$$z_t = y_t - \frac{1}{2} y_{t - 1} - \frac{1}{8} y_{t - 2} - \frac{1}{16} y_{t - 3} + \ldots$$
According to the terminology, the "memory is preserved" because the coefficients are positive for infinitely many past values of $y_{t-k}$.
The one example I can think of where memory is clearly lost by a first difference but without any apparent consequences is a true random walk. Where $\epsilon_i$ are i.i.d. $N(0, 1)$ random variables, let
$$
\begin{align}
y_1 &= \epsilon_1, \\
y_2 &= \epsilon_1 +\epsilon_2, \\
y_3 &= \epsilon_1 + \epsilon_2 + \epsilon_3, \\
y_4 &= \epsilon_1 +\epsilon_2 + \epsilon_3 +\epsilon_4.
\end{align}
$$
Then the first difference $y_4 - y_3 = \epsilon_4$ losses all of $y_4$'s information about $\epsilon_1, \ldots, \epsilon_3$, whereas the fractional differencing transformation would contain a bit of all of the epsilons. Not that they'd help you to forecast.
Another maybe more profound example where "memory" matters is a conintegrated time series regression problem described in page 11 of these notes by Eric Sims. There, taking first differences of two cointegrated random walks results in bias in the estimation stage, because the error in the difference vs difference regression includes a correction term to bring things in line with the long run relationship. If you rid yourself of that long term cointegrating relationship by differencing, you'll suffer bias.
Answer: The signal processing approach to such problems may be easier to understand. Among other things, the signal to noise ratio (SNR) for stocks may be so high as to render signal detection difficult, there are none-the-less many off-the-shelf algorithms for improving SNR, and that is perhaps of more interest to you. Such algorithms may sacrifice some signal (lossy algorithms) to accomplish noise reduction. The simplest algorithm to understand is perhaps addition. For example, if the trends are low frequency (long term) then reduction of the higher frequency noise by averaging (read as addition of) adjacent temporal signals or averaging over a short time 'window' will tend to reduce high frequency noise more so than low frequency signal thereby improving SNR. In the frequency domain this is called a low pass filter.
With respect to the question, addition and subtraction reduce information in the sense that when an average or difference is created, one loses the information as to what the numbers were before combining them.  For example, given a $5$ as a result that  could have come from $6-1$, $10-5$ or an uncountably infinite number of other number pairs.
Regarding information content such things are context dependent. If we assume that the noise contains no information, and that only the signal has information content, the goal of signal processing would then be to isolate the signal with the least possible signal loss while reducing the memoryless noise to zero. On  the other hand, if the context is that both signal and noise contain information, then obviously noise reduction would seriously reduce information content.
With respect to prediction, if the noise contains no information then one could establish bounds or an 'envelope' in which the noise is contained.  That is not exactly a prediction in the same sense as $y=f(t)$, but when coupled with the low pass signal may none-the-less offer a prediction range.
In the real world (assuming money is real) the high frequency data contains less information (but still some) and the low frequency data relatively more information. Still, signal processing may be useful.  For example, examining the data in the frequency domain may reveal repetitive occurrences that are not obvious in  the data itself, and such a process (Fourier transform) is lossless.
Comment: I think the point Marcos is making is that SARiMAX type models ignore the information content (if any) in the level of the I(I>i) signal. Yes the information is still there as you can just integrate an I(0) process to get the original I(1) process, but the model you feed the I(0) signal into does not reconstruct I(1) from I(0) (because of spurious regression concerns around unit root processes). The information is there yes, but the models ignore it because they can't handle it. That's the point I think. The proposed solution is to turn your SARiMAX into a SARFiMAX where i can be a fraction.
Comment: To use an analogy from Control Engineering, differencing and not using the original series is like attempting to design a PID controller and restricting the design ex-ante not to use the I (the integral) and instead (possibly) use the 2nd derivative. If the 2nd derivative has more information than the integral, that is an improvement, otherwise it is a downgrade. Typically the second derivative will contain the noise.
Comment: I understand the approach you are saying. But when we are doing the differencing, I think, that at that point everything can be considered "useful" information and not noise. Only the estimation method, for which we feed the series - let it be a regression or machine learning based - will decide what is noise and what is useful information. Until that point I would say, that we need every piece of information we can feed into.
Comment: That is why it is hard for me the grasp the concept, of loosing information by differencing. If we difference, we can easily recreate the original series by integrating, only we are missing the initial value, or the offset, which I think @mlofton is also referring to. But I do not think that the lack of that one constant can hinder the predictive power of a model, especially if we are thinking on stock price series - where the absolute magnitude of a stock is not much of interest to anyone.
Comment: An other explanation I can think beside this, that maybe people are implying, that we are loosing information because of a floating point error when calculating the difference d series as a percentage change or logarithmic return. But that is much more a computer science issue, than a theoretical statistical one, and I am sure we can have enough computational capacity, that can reduce it to being marginal.
Comment: Losing  information (spelling, common mistake, e.g., loose screw). [A linear difference equation](https://en.wikipedia.org/wiki/Linear_difference_equation)'s information loss would, I suppose, depend on exactly what was done to produce it.  In the particular case you are referring to, I do not know how much information would be lost; that depends on the exact algorithm used.
Answer: It's a well-know trade off between explanatory and predictive modeling.
We don't really do the difference nowadays but when you try to fit a model, the goal at the beginning is very important.
I can have the best predictive model of all time with non sense coefficient estimates or have a really good explanatory model that predicts poorly.
So basically when you do a differenciation of your series, you are more in a explanatory case than predictive so you sacrifice a little of the predictive power to have a better model in quality. Or at least you are ready to sacrifice but i don't think it can be proved that it will always make loose predictive power, maybe 95% of the cases base on personal experience.
Comment: If you have terrible predictive power, why do you believe your explanation?
Comment: When I said poorly, I should have said worse than an other modeling option with less explanatory power.
Comment: @Betelgeux: I'm not clear on what you mean by the lack of one constant in your comment ? In time-series, differencing means that you know longer have the levels of the series at any point in time because you are differencing adjacent values. so, you lose a lot of information. But, particularly in finance, differencing can often make a series more stationary  so it can be a useful thing. In machine learning, you could be correct in that it's less harmful to keep everything but time-series is pretty different. Note that you only lose one constant if the series had a constant level.
Comment: lose predictive power, not loose.


How to delete definite lines of a file which strats and ends with specific characters

Question: I have a test file with many lines. I want to delete lines with a specific start and end characters.
here is my code:
with open('test.txt', 'r') as f, open('output.txt', 'w') as out:
    for i, line in enumerate(f):
        if (line.startswith('E3T') and line.endswith('3')):
           out.write(line)
        elif (line.startswith('E4Q') and line.endswith('3')):
           out.write(line)
        elif (line.startswith('E4Q') and line.endswith('4')):
           out.write(line)
        elif (line.startswith('E4Q') and line.endswith('3')):
           out.write(line)
        elif line.startswith('BC'):
            break

this is my test.txt file
E3T 1 2 1 3 3
E3T 2 4 2 5 1
E3T 3 3 5 2 4
E3T 3326 2001 2008 1866 10
E4Q 3327 1869 2013 2011 1867 9
E4Q 3328 1867 2011 2012 1868 8
E4Q 3329 1870 2014 2013 1869 4
E3T 8542 4907 4908 4760 5
E3T 8543 4768 4909 4761 9
E3T 8544 4909 4763 4761 6
E3T 17203 9957 9964 10161 3
E3T 17204 9957 10161 9959 2
BC  1 "Zulauf: Temperatur" 12 0 1 "HYDRO_WT-2D"
BC_DEF 12 1 "Temperatur [°C]" 5 "Zeit [s]" "Temperatur [°C]"

And the output should be like this:
E3T 1 2 1 3 3
E3T 3 3 5 2 4
E4Q 3329 1870 2014 2013 1869 4
E3T 17203 9957 9964 10161 3

I think, it does not work because of the spaces. Is there any pythonic way of doing this or I have to split the lines and then compare the first and last characters?
Answer: When you read a line in that way there is either a new-line or a new-line/line-feed character at the end of it that is usually 'invisible' to you. You need to deal with that somehow, otherwise endswith will process it rather than the character you want to process. Then, when you output a line you need to put the new-line character back.
with open('test.txt', 'r') as f, open('output.txt', 'w') as out:

    for i, line in enumerate(f):
        line = line.strip()
        if (line.startswith('E3T') and line.endswith('3')):
           out.write(line+'\n')
        elif (line.startswith('E4Q') and line.endswith('3')):
           out.write(line+'\n')
        elif (line.startswith('E4Q') and line.endswith('4')):
           out.write(line+'\n')
        elif (line.startswith('E4Q') and line.endswith('3')):
           out.write(line+'\n')
        elif line.startswith('BC'):
            break

In this case I've used strip to discard white space at the beginning and end of each line. This is a very crude approach. It would be better to use,
line = line.rstrip()

which strips white space from only the right end of a string.
EDIT, in answer to the question in a comment:
Replace the last line above with these lines,
    out.write(line+'\n')
else:
    continue

Comment: thank you for the solution! What should I do if I want to write the rest of the lines and don't break! which means the rest of the file should be as same as the input file!


Calculating goodness of fit and choosing the right model (R)

Question: I'm trying to evaluate the value of an object, depending on his characteristics. In order to do this, I'm building the following regression model price ~ ., using similar objects and for each variable I got min 20 observations
I encountered following problem: none of the regression models worked for all my data, so I decided to use all of the followings methods:
model.lm <- lm(price ~ .)
model.lmLog <- lm(log(price) ~ .)
model.ltsReg <- ltsReg(price ~ .)
model.ltsRegLog <- lts(log(price) ~ .)
model.lmrob <- lmrob(price ~ .)
model.lmrobLog <- lmrob(log(price) ~ .)
model.lmRob <- lmRob(price ~ .)
model.lmRobLog <- lmRob(log(price) ~ .)
model.glm <- glm(price ~ .)
model.glmLog <- glm(price ~ ., family=gaussian(link="log"))

My question is: how can I decide which of this models fits best for the current data, without plotting the results?
As far as I know, the r-squared aren't trusty, because I the data is corrupted, so will be the r-squared.
Any ideas?
Thank you!
[UPDATE]
what do you think about using BIC or AIC and choosing the one with the lowest value?
what do you think about choosing the variables for the regression upon the analysis of anova?
I have 17 variables from which 10 are dummy variables, is that a problem?
Answer: When you entertain more than about 3 models, model uncertainty will greatly distort your final result, especially with regard to precision, confidence intervals, and $P$-values.  AIC is generally preferred over BIC, but AIC does not solve the above problem.  I would choose a robust semi-parametric model such as proportional odds or proportional hazards (see the orm function in the R rms package).  These are $Y$-transformation invariant.  Or use AVAS to make estimation of the transformation of $Y$ part of the process (function areg.boot in rms).  Note that with 20 observations you cannot afford to model more than a couple of parameters on the right hand side of the model, and any linearity assumptions you make in doing so are questionable.
Traditional robust regression involves trimming, mean absolute deviation (MAD) minimization, etc., and it is very dependent on how $Y$ is transformed.  Robust regression also, except for MAD (quantile regression) produces hard-to-interpret estimates.  A large class of ordinal regression models is based on cumulative probabilities and allows one to directly estimate any exceedance probability or other probability of interest.  It also allows you to estimate means, and in the case of continuous $Y$, quantiles.  In rms the pertinent functions are orm, Quantile, Mean, and ExProb, the latter 3 functions being applied to a fit object produced by orm.  A detailed case study may be found at the link to course handouts at http://biostat.mc.vanderbilt.edu/rms and is covered in detail in the upcoming 2nd edition of my book Regression Modeling Strategies.  Except for the estimation of the conditional mean, ordinal regression in the cumulative probability family is completely invariant to the $Y$-transformation.
Note that when printing the result of orm the intercepts are not printed by default.  There are options for printing them, or use coef(fit).
Comment: There is k-1 where k is the number of unique Y values.  Think of the intercepts as the empirical distribution of Y given X
Comment: with 20 observations pro variable, I mean that I have a min #obs = 20*#variables
Comment: I can't install package `rms` due to problems with it's dependencies, not all of them are available for `R 3.0` .
I decided to use only `lmRob` . what do you think about using `drop1.lmRob` or `step.lmRob`?
Comment: If you are running the latest version of R (and if you are using RStudio you have installed its latest version too) `rms` and its dependencies work fine.
Comment: I tried now the `orm` and it returned totally different results than `lmRob`. I still have some questions: 1. why would you choose `orm` over a robust linear method. 2. I haven't seen the coefficient for the intercept in the result, where is it?
Comment: I can't understand why are there so much values for the intercept (coef(fit)).


First remove seasonal trend or long-term trend in time series?

Question: I have a time series (quarterly data) which has both a long-term trend and seasonality. Taking seasonal differences will make the series stationary, according to the Augmented Dickey-Fuller test. On the other hand, if I first take non-seasonal differences, the series also become stationary according to the Augmented Dickey-Fuller test, but the ACF still shows seasonal correlation.
How do I tackle this? Should I take both seasonal -and non seasonal differences because there is a trend and seasonality, even though taking only of them already makes the series stationary?
I am confused because examples I found seem to be contradictory.
Comment: You are right to be confused: there are many different analysis styles. For example, one common answer is not to think of trend or seasonality as in any sense stuff to be removed before concentrating on the rest, but to build a single model that adequately captures trend, seasonality and other structure.
Comment: You might want to add more information about your goal. The question is currently asking about a particular solution, which may or may not be appropriate, and if it is appropriate the way its done may still depend on the goal.
Answer: As you suggested you one can observe non-stationarity (symptom) but the correct remedy (medicine) is unclear. The correct remedy could be multiple level shifts , multiple trends , seasonal pulses too name a few. Assuming any one approach is both simple and potentially damaging to good statistical analysis.. The high road is to to "listen to the data" ala Bacon,Box,Tukey et al and form the appropriate form of non-stationarity adjustment (much like a good drug prescription) to render the data stationary without incurring damage. The whole idea is to keep the model simple but not too simple.
Non-stationarity can be induced by changes in model form , changes in parameters, changes in error variance besides what has been previously listed here. The message is avoid presumptive cook-book rules suggested by some textbooks and commentators particularly x-11 and it's variants and use the best statistical tools available to form/identify usable solutions.
For example review the outliers and changing error variance when using x11 on the classic airline series to construct the irregular or error process.
Answer: Remove them simultaneously. This example shows you how to do it. Henderson filter extracts the trend while S(3,3) filter extracts the seasonality. In fact all de-seasonalizing software will do this, I think.


Scraping svg Content from a webpage

Question: I am trying to scrape pdf files of Safety Data Sheets from this link: https://www.sigmaaldrich.com/PK/en/search/2127-03-9?focus=products&page=1&perpage=30&sort=relevance&term=2127-03-9&type=cas_number
The pdf link seems to be part of the SVG content on the webpage. I found Scraping a webpage for link titles and URLs utilizing BeautifulSoup link and am trying to use the answer to get SVG content.
However, the code does not seem to extract SVG content.
<code>base_url = 'https://www.sigmaaldrich.com/PK/en/search/2127-03-9?focus=products&page=1&perpage=30&sort=relevance&term=2127-03-9&type=cas_number'
headers = {"User-Agent": "Mozilla/5.0 (X11; CrOS x86_64 12871.102.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.141 Safari/537.36"}

with requests.Session() as session:
    # extract the link to svg
    res = session.get(base_url, headers = headers)
    soup = BeautifulSoup(res.content, 'html.parser')
    svg = soup.select_one("object.svg-content")
    svg_link = urljoin(base_url, svg["data"])
</code>
Error
Comment: There is no class `svg-content` in the response, please double check the url
Comment: or if the data is generated dynamically use selenium
Comment: Let me know if you need a selenium solution
Comment: Can you provide a selenium solution?
Comment: Can you post the element as that you want to scrape
Comment: Basically, on the link, you can see four chemicals with an icon to open their Safety Data Sheets (SDS), which I want to scrape. I can open each SDS and use their link to scrape. But I want the code to find the embedded pdf links and then scrape them. I want to automate the process of opening each SDS as well.
Comment: Hi @DanishRajani, so all you need is to download those 4 pdfs using selenium, right?
Comment: Yes, that is the aim.
Comment: So if I give you the code, that clicks on 4 pdf download button, would that do? @DanishRajani
Comment: Yes, I am trying to write the code using selenium, but somehow it's not working. I am unable to locate the SVG icon and click it. I am trying to locate SVG using XPATH but somehow not working. Actually, the larger is to input the CAS number of different chemicals and scrap all SDS pdf files as for each CAS number, there are multiple. First, I am trying to solve the problem of scraping pdfs, then I will work on how to automate the whole process.
Answer: There isn't a SVG. I clicked and it brought a pdf. Could you try to download by a link, as https://www.sigmaaldrich.com/BR/pt/sds/sigma/d5767 e.g.?
Also, an alternative which doesn't require BS: Selenium. It's quite simple to achieve the desired effect with this tool.
Doc:

https://selenium-python.readthedocs.io/
https://www.browserstack.com/guide/download-file-using-selenium-python
Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Comment: Yes, this is one of the pdfs, but there are four chemicals; hence I was wondering if there is a way to scrap all four pdfs from the link above.
Comment: There is a way: you just need to get those links by using selenium, to click and watch over the page opened.

Scraping svg Content from a webpage


I don't understand this summation simplication (Simple problem).

Question:
I don't how the highlighted occurred.
EDIT:Optional:  Does anyone know where I can practice these basic summation problems? Specifically  like a site perhaps..
Comment: $\sum \text{(stuff)} = \text{(stuff)}\sum 1$ and $\sum \text{(stuff)}\cdot i = \text{(stuff)}\sum i$ where $\text{(stuff)}$ is not a function of $i$.
Comment: so the -1 from highlighted summation went inside ?
Comment: The first term: $$\sum (n(n+1)(2n+1)/2) = (n(n+1)(2n+1)/2)\sum 1$$ and the second term:$$-\sum (2in) = -\sum (2n)\cdot i = -(2n)\sum i$$
Answer: Note that each of your sums is over the parameter $i$. All of the $n$'s are a fixed value and do not change within the sum. As such they can be taken out of the sum as a constant. Compare with this easier to see example:
$$\sum_{i=1}^nf(n)\cdot g(i)$$
We could write this out in full as:
$$f(n)\cdot g(0)+f(n)\cdot g(1)+f(n)\cdot g(2)+...+f(n)\cdot g(n-2)+f(n)\cdot g(n-1)+f(n)\cdot g(n)$$
Then we can factor out the $f(n)$ to give:
$$f(n)\left(g(0)+g(1)+g(2)+...+g(n-2)+g(n-1)+g(n)\right)$$
And then rewriting it in sum notation gives:
$$f(n)\cdot\sum_{i=1}^ng(i)$$
Comment: OHHHHH I get it
Comment: I didn't know what was acting on what. This really helps.
Comment: quick question... the summation of a constant 1 will be 1n right? is that you got rid of the sigma? im 95% sure on this, just wanted to know concretely know,
Comment: Yes $\sum_{i=1}^n1$ is equal to $n$.
Comment: Ahhh Ok i get it. Perfect Answer thank you . 2x more confident with summations now! Thanks.


Testing significance of peaks in spectral density

Question: We sometimes use spectral density plot to analyze periodicity in time series. Normally we analyze the plot by visual inspection and then try to draw a conclusion about the periodicity. But have the statisticians developed any test to check whether any spikes in the plot are statistically different from white noise? Have the R-experts developed any package for spectral density analysis and for doing that kind of test?
Comment: Pressed by @Wesley, I deleted my quick thoughts about autocorrelation functions and periodogram (may be he indeed is a frequency domain analysis guru, but I personally don't think Bartlett, while working with autocorrelations in time domain), but still think that my second suggestion about `bootspecdens` may be helpful.
Comment: I'm basing my assumption about people's response to 'what is an autocorrelation?' on literature appearances, in which almost all instances where an autocorrelation is used being of the standard, time-domain computed, Barlett autocorrelation. And, unfortunately, this is bad! :)

I do appreciate the suggestion of `bootspecdens` from Dmitrij; looking forward to checking it out.
Answer: You should be aware that estimating power spectra using a periodogram is not recommended, and in fact has been bad practice since ~ 1896. It is an inconsistent estimator for anything less than millions of data samples (and even then ...), and generally biased. The exact same thing applies to using standard estimates of autocorrelations (i.e. Bartlett), as they are Fourier transform pairs. Provided you are using a consistent estimator, there are some options available to you.
The best of these is a multiple window (or taper) estimate of the power spectra. In this case, by using the coefficients of each window at a frequency of interest, you can compute a Harmonic F Statistic against a null hypothesis of white noise. This is an excellent tool for detection of line components in noise, and is highly recommended. It is the default choice in the signal-processing community for detection of periodicities in noise under assumption of stationarity.
You can access both the multitaper method of spectrum estimation and the associated F-test via the multitaper package in R (available via CRAN). The documentation that comes with the package should be enough to get you going; the F-test is a simple option in the function call for spec.mtm.
The original reference that defines both of these techniques and gives the algorithms for them is Spectrum Estimation and Harmonic Analysis, D.J. Thomson, Proceedings of the IEEE, vol. 70, pg. 1055-1096, 1982.
Here is an example using the included data set with the multitaper package.
require(multitaper);
data(willamette);
resSpec <- spec.mtm(willamette, k=10, nw=5.0, nFFT = "default",
                    centreWithSlepians = TRUE, Ftest = TRUE,
                    jackknife = FALSE, maxAdaptiveIterations = 100,
                    plot = TRUE, na.action = na.fail)

The parameters you should be aware of are k and nw: these are the number of windows (set to 10 above) and the time-bandwidth product (5.0 above). You can easily leave these at these quasi-default values for most applications. The centreWithSlepians command removes a robust estimate of the mean of the time series using a projection onto Slepian windows -- this is also recommended, as leaving the mean in produces a lot of power at the low frequencies.
I would also recommend plotting the spectrum output from 'spec.mtm' on a log scale, as it cleans things up significantly. If you need more information, just post and I'm happy to provide it.
Comment: Admittedly, this was 8 years ago, when I was a lot younger and crankier. But for scientific applications with many lines embedded against a quasi-stationary background, the periodogram and the smoothed versions remain less than useless. And that's the primary area I work in. If you have a handful of lines, you can do whatever you want, and it'll basically work. Provided your model matches reality, which is always questionable. Even with smoothing, the periodogram is not robust to violations.
Comment: To Burr, Silva and Celov - thanks a lot for your interesting answers and suggestions. I look forward to test these estimators. Best regards
Comment: (+1) this night I thought carefully about your suggestions, and decided that time domain indeed is the last thing (due to lag truncation and weak properties in small samples) to try searching for cycling behavior. What I am personally concerned about are the assumptions for F statistics and the small sample size properties of the suggested scheme. Well and probably it is good to start a separate question regarding the optimal window selection, because there are indeed many.
Comment: There are indeed many window choices, although the two most common are the Discrete Prolate Spheroidal Sequences (or *Slepians*) and the sine tapers. If you are looking for maximal concentration of energy in a local bandwidth, the Slepians have been proved to be optimal, and in fact are the output from the integral equation form of the spectral density (see the paper I mentioned for full details). As far as the F statistics go, there certainly are some issues with degrees of freedom, but on the whole they work quite well, with ~ 2k-2 dof available.
Comment: Smoothed periodogram also uses taper, allows for FFT, David Stoffer's book teaches you how to calculate confidence intervals as well. This `multitaper` package seems to have employed more advanced techniques for tapering and calculating confidence interval. But I think the idea was the same, according to David Stoffer. This is the only thing I could think of that teaching vanilla peridogoram actually still makes sense today.
Comment: ok so you are one of the authors of this package and you have used some very strong words against periodogram I hope you could one day come back with more evidence. Common pros and cons for Periodogram are well known, like its explosive variance which is why it's not a good consistent estimator for spectrum but the smoothed periodogram isn't really that bad, not as bad as you stated here I think.
Comment: Any suggested python implementation?
Answer: Ronald Fisher proposed an exact test of the maximum periodogram
coordinate in R.A. Fisher, Proc. R. Soc A (1929) 125:54. The test is based on the g-statistic. Specifically, the null hypothesis of Gaussian white noise is rejected if g is significantly large, that is, if one of the values of $f(\omega_k)$ is significantly higher than the average of all signals.
You can get more details about the test in MB Priestley, Spectral Analysis and Time Series, Academic Press, London, 1981, page 406.
In R, the package GeneCycle  contains the function fisher.g.test():
library(GeneCycle)
?fisher.g.test

Hope this helps.
Comment: this is great but the package's g test relies on its own periodogram function which has very limited options for calculating power spectra...
Answer: We have tried an attempt to address this issue by a wavelet transform of a spectral-based test recently in this paper. Essentially, you need to consider the periodogram ordinates distribution, similarly to the article of Fisher, mentioned in the earlier answers. Another paper from Koen is this.
We have recently published an R package hwwntest.
Answer: Use the spectrum.test function in the ts.extend package
You can conduct a "permutation spectrum test" on your data using the ts.extend package.  This is a permutation-based variant of the classic Fisher test that looks at the maximum spectral intensity of the data and compares it to its null distribution under the null hypothesis of exchangeability.  (The advantage of this test over the Fisher test is that it does not assume normal error terms in the data; i.e., it works for any underlying distribuiton.)  Here is an example where we generate data with a periodic signal and then test for the presence of the signal.  The test output and resulting plot easily detects the signal.
#Load the package
library(ts.extend)

#Generate mock data
set.seed(1)
m      <- 100
SIGNAL <- 0.8*sin(0.3*(1:m))
NOISE  <- rnorm(m)
SERIES <- SIGNAL + NOISE

#Conduct permutation-spectrum test
TEST <- spectrum.test(SERIES)
TEST
        Permutation-Spectrum Test

data:  real time-series vector SERIES with 100 values
maximum scaled intensity = 3.6428, p-value = 0.000208
alternative hypothesis: distribution of time-series vector is not exchangeable
(at least one periodic signal is present)

#Plot the test results
plot(TEST)


Adicionar objetos de tipos diferentes em um mesmo vetor

Question: Não consigo adicionar a classe Video dentro do vetor.
public class SongDriver {
    public static void main(String[] args) {
        Playlist lista = new Playlist();
        Musica num1 =new Musica("Agua de beber","Antonio Carlos Jobim","Astrud Gilberto", 140);
        Musica num2 =new Musica("O mar serenou","Candeia"," Clara Nunes", 179);
        Musica num3 =new Musica("Rapaz Folgado","Noel Rosa"," Martinho da Vila Mart'nália",180);

        Video num4 = new Video("Musica"," Despacito", " Luis Fonsi",5926796,280);

        lista.add(num1);
        lista.add(num2);
        lista.add(num3);
        lista.add(num4);

        lista.print();

    }
    public static class Playlist{

        private Musica []musicas;
        private int cont;

        public Playlist(){
            musicas = new Musica[1000];
            cont = 0;
        }
        public void add(Musica a){
            if(cont==musicas.length){
                System.out.println("ERROR: playlist atingiu sua quantidade máxima de itens.");
            }
            musicas[cont]= a;
            cont++;
        }
        public void add(Video b){
            if(cont==musicas.length){
                System.out.println("ERROR: playlist atingiu sua quantidade máxima de itens.");
            }
            musicas[cont]= b;
            cont++;
        }
        public Musica get(int i){
            if(cont> i){
                return musicas[i];
            } else {
                return null;
            }
        }
        public void print(){
            String result="Total de Musicas e Videos Alocados = "+cont+"\n";
            for(int i =0; i <cont; i++){
                result+=( i +" => "+musicas[i]+"\n");
            }
            result+="Tempo Total da PlayList: "+formatoTotalTempo();
            System.out.println(result.toString());
        }
        public int totalTempo(){
            int totalTempo=0;
            for(int i =0; i <cont; i++){
                totalTempo+=musicas[i].getDuracao();
            }
            return totalTempo;
        }

        public String formatoTotalTempo(){
            return formatoTempo(totalTempo());
        }
    }

    public static class Musica{

        public String nome;
        public String interprete;
        public String compositor;
        public int duracao;

        public Musica(String musicaNome,String interprteNome,String compositorNome,int tamanhoDuracao){
            this.nome=musicaNome;
            this.interprete=interprteNome;
            this.compositor=compositorNome;
            this.duracao=tamanhoDuracao;
        }

        public void setNome(String musicaNome){
            nome=musicaNome;
        }

        public String getNome(){
            return nome;
        }

        public void setInterprte(String interpreteNome){
            interprete=interpreteNome;
        }

        public String getInterprete(){
            return interprete;
        }

        public void setCompositor(String compositorNome){
            compositor=compositorNome;
        }

        public String getCompositor(){
            return compositor;
        }

        public void setDuracao(int tamanhoDuracao){
            duracao = tamanhoDuracao;
        }
        public int getDuracao(){
            return duracao;
        }
        public String toString(){
            return"Nome: "+getNome()+", Interprete:"+getInterprete()
                +", Compositor:"+getCompositor()+", Duracao: "+formatoTempo(getDuracao());
        }
    }
    public static class Video{
        String titulo;
        String assunto;
        String canal;
        int visualizacoes;
        long duracao_video;

        public Video(String tituloNome,String assuntoNome,String canalNome,int visualizacoes_video, long tempo_video){
            this.titulo = tituloNome;
            this.assunto = assuntoNome;
            this.canal = canalNome;
            this.visualizacoes = visualizacoes_video;
            this.duracao_video= tempo_video;
        }
        public void setTitulo(String tituloNome){
            titulo = tituloNome;
        }

        public String getTitulo(){
            return titulo;
        }
        public void setAssunto(String assuntoNome){
            assunto = assuntoNome;
        }

        public String getAssunto(){
            return assunto;
        }
        public void setCanal(String canalNome){
            canal = canalNome;
        }

        public String getCanal(){
            return canal;
        }
        public void setVisualizacoes(int visualizacoes_video) {
            visualizacoes = visualizacoes_video;
        }
        public int getVisualizacoes() {
            return visualizacoes;
        }
        public void setDuracao_video(long tempo_video) {
            duracao_video = tempo_video;
        }
        public long getDuracao_video() {
            return duracao_video;
        }
        public String toString(){
            return"Titulo: "+getTitulo()+", Assunto:"+getAssunto()+", Canal:"+getCanal()+
                    ", Visualizações: "+getVisualizacoes()+" Duração do video:"+formatoTempo(getDuracao_video());
        }

    }
    public static String formatoTempo(long tempo){
        int hora = 60;
        int minuto = 60;
        long h= tempo /hora;
        long m= tempo % minuto;
            return String.format("%02dh %02dmin", h, m);
    }

}

Comment: Isso não vai funcionar pq vc está tentando adicionar um `Video` em um array de `Musica`. Como as duas classes não tem relação nenhuma entre si, seus tipos são incompatíveis e não é possível colocar uma em um array que só tem a outra. Crie um outro array só para `Video` (na verdade, tem outras soluções, como refatorar as classes, criar uma interface `Playable`, por exemplo, para qualquer tipo de mídia "tocável", etc). Mas por ora, ter outro array deve bastar...
Comment: Hkotsubo, como faço para criar um vetor generico para as duas classes musica e video.
Answer: O problema é que Musica e Video são classes diferentes que não tem relação nenhuma entre si. Por isso, ao tentar colocar um Video em um array de Musica, dá erro de compilação. Existem algumas alternativas para resolver esse problema.
Obs: eu criei cada classe em um arquivo .java separado, por isso no meu código abaixo elas não são static. Talvez você tenha colocado tudo em um arquivo só "para facilitar", mas criar classes aninhadas tem usos bem específicos que não parece ser o caso aqui. Então sugiro que separe cada classe em seu próprio arquivo .java.

Usar outro array para Video
Isso é uma solução que não requer alteração nas classes Musica e Video. Basta manter um array separado para cada uma delas. Como você está controlando a quantidade de músicas com outra variável, vai precisar de mais uma para os vídeos também (e mudei o nome dos contadores para algo mais significativo).
public class Playlist {
    private Musica[] musicas;
    private Video[] videos; // novo array só para vídeos
    // manter um contador para cada tipo de mídia
    private int qtdMusicas;
    private int qtdVideos;

    public Playlist() {
        musicas = new Musica[1000];
        videos = new Video[1000];
        qtdMusicas = 0;
        qtdVideos = 0;
    }

    public void add(Musica a) {
        if (qtdMusicas == musicas.length) {
            System.out.println("ERROR: playlist atingiu sua quantidade máxima de músicas.");
        }
        musicas[qtdMusicas] = a;
        qtdMusicas++;
    }

    public void add(Video b) {
        if (qtdVideos == videos.length) {
            System.out.println("ERROR: playlist atingiu sua quantidade máxima de vídeos.");
        }
        videos[qtdVideos] = b;
        qtdVideos++;
    }
}

Obviamente que será preciso adicionar um método getVideo, e eu também mudaria o nome do método get atual para getMusica. O método print também deverá percorrer o array de vídeos logo depois do loop já existente no array de músicas.
Mas essa solução não é muito boa, pois duplica muito código: o algoritmo para adicionar, obter e imprimir músicas e vídeos é praticamente o mesmo. E se futuramente você quiser adicionar uma outra classe (por exemplo, Filme), terá que duplicar código novamente.
Talvez seja melhor generalizar a solução, mas isso exigirá uma alteração nas classes.

Generalizando...
Se você quer que sua Playlist tenha tanto músicas quanto vídeos, pode fazer com que ambas as classes Video e Musica compartilhem alguma característica que permita agrupá-las. No caso, ambas podem ser "tocadas" em um player de mídia. Então eu posso criar uma interface que represente esta característica:
// representa qualquer coisa que pode ser "tocada" em um player
public interface Playable {
    public String getNome();
    public int getDuracao();
}

No caso, a interface Playable representa "algo que pode ser tocado em um player". Eu também defini os métodos getNome() e getDuracao(), mas repare que esses métodos não tem código. A interface só define o que um Playable faz, mas não como isso é feito.
Agora basta fazer com que Musica e Video sejam "coisas que podem ser tocadas em um player". Em outras palavras, ambas as classes devem implementar a interface Playable:
public class Musica implements Playable {
    // o resto da classe permanece igual
}

Como a classe Musica já possui os métodos getNome() e getDuracao() (e eles retornam respectivamente uma String e um int, tal como definido na interface), não é preciso fazer mais nada. Já a classe Video precisará de algumas adaptações, já que ela não possui esses métodos (abaixo coloquei apenas os trechos que mudei):
public class Video implements Playable {
    // mudei a variável duracao_video, pois é redundante (ela faz parte de Video, então claramente é a duração do vídeo)
    int duracao; // mudei para int, o tempo é em minutos, não precisa de long

    public Video(String tituloNome, String assuntoNome, String canalNome,
                 int visualizacoes, int tempo) {
        ....
        this.duracao = tempoVideo;
    }

    @Override
    public String getNome() {
        return this.titulo; // o nome do vídeo é o seu título
    }
    public void setDuracao(int tempo) {
        duracao = tempo;
    }

    @Override
    public int getDuracao() {
        return this.duracao;
    }
}

Mudei algumas coisas: como o tempo está em minutos (pelo que entendi), não precisa usar um long. Um int suporta até pouco mais de 2 bilhões, o que é suficiente para qualquer vídeo.
Também mudei o nome de duracao_video para duracao. Se a duração pertence à classe Video, é redundante dizer que esta é a duração do vídeo. Outro ponto é evitar nomes_de_variaveis_assim, já que a convenção do Java é usar nomesDeVariaveisAssim.

Bom, agora que tanto Musica quanto Video são Playable, podemos mudar nosso array:
public class Playlist {
    private Playable[] items;
    private int qtdItens;

    public Playlist() {
        items = new Playable[1000];
        qtdItens = 0;
    }

    public void add(Playable item) {
        if (qtdItens == items.length) {
            System.out.println("ERROR: playlist atingiu sua quantidade máxima de itens.");
        }
        items[qtdItens] = item;
        qtdItens++;
    }

    public Playable get(int i) {
        if (qtdItens > i) {
            return items[i];
        } else {
            return null;
        }
    }

    public void print() {
        String result = "Total de Musicas e Videos Alocados = " + qtdItens + "\n";
        for (int i = 0; i < qtdItens; i++) {
            result += (i + " => " + items[i] + "\n");
        }
        result += "Tempo Total da PlayList: " + formatoTotalTempo();
        System.out.println(result.toString());
    }

    public int totalTempo() {
        int totalTempo = 0;
        for (int i = 0; i < qtdItens; i++) {
            totalTempo += items[i].getDuracao();
        }
        return totalTempo;
    }

    public String formatoTotalTempo() {
        return formatoTempo(totalTempo());
    }
}

Repare que agora eu só trabalho com Playable, ou seja, o array vai aceitar qualquer classe que implemente a interface Playable. Então eu posso adicionar Musica, Video e qualquer outra classe que eu criar futuramente (desde que ela implemente Playable).

Próximos passos
Isso é apenas uma introdução e serve como os primeiros passos para modelar melhor suas classes. Mas sugiro estudar a fundo polimorfismo, modelagem de classes e o próprio mecanismo de interfaces e classes abstratas do Java. É algo amplo demais para abordar aqui, então deixo apenas mais uma dica "bônus".
Em vez de um array, você pode usar java.util.List, pois assim você não precisa ficar controlando a quantidade de itens, já que a própria lista aumenta seu tamanho conforme a necessidade. A classe modificada ficaria assim:
import java.util.ArrayList;
import java.util.List;

public class Playlist {
    private List<Playable> items;

    public Playlist() {
        this.items = new ArrayList<>(); // não precisa dizer o tamanho, a lista controla isso para você
    }

    public void add(Playable item) {
        if (this.items.size() == 1000) { // se quiser, ainda pode limitar a quantidade máxima (opcional)
            System.out.println("ERROR: playlist atingiu sua quantidade máxima de itens.");
        }
        this.items.add(item);
    }

    public Playable get(int i) {
        if (i > this.items.size()) {
            return null;
        }
        return this.items.get(i);
    }

    public void print() {
        String result = "Total de Musicas e Videos Alocados = " + this.items.size() + "\n";
        for (int i = 0; i < this.items.size(); i++) {
            result += (i + " => " + items.get(i) + "\n");
        }
        result += "Tempo Total da PlayList: " + formatoTotalTempo();
        System.out.println(result.toString());
    }

    public int totalTempo() {
        int totalTempo = 0;
        for (Playable item : items) {
            totalTempo += item.getDuracao();
        }
        return totalTempo;
    }

    public String formatoTotalTempo() {
        return formatoTempo(totalTempo());
    }
}

Outro detalhe é que se você usar o método get, irá obter um Playable, e para saber se ele é uma música ou um vídeo, terá que usar instanceof e fazer o cast:
Playable item = lista.get(0);
// testar o tipo de Playable retornado
if (item instanceof Musica) {
    Musica musica = (Musica) item;
    // usar a Musica
} else if (item instanceof Video) {
    Video video = (Video) item;
    // usar o Video
}

Comment: Eu estava refatorando o código do Bruno mas aí a página foi atualizada e já tinha a sensacional explicação do hkotsubo :D
De qualquer forma a solução que eu estava elaborando está aqui https://github.com/joellobo/stackoverflow/blob/master/364318/src/com/stackoverflow/SongDriver.java
Comment: Agradeço ao hkotsubo e ao Joel lobo, são profissionais que fazem com que tenhamos prazer em aprender programar  , vocês deram explicações perfeitas, aprendi muito com vocês, abrigado amigos, meu sonho é chegar no nivel de conhecimento de vocês e poder ajudar outros ..
Comment: Amigos só mais uma pergunta, eu quero implementar uma maneira de aumentar o tempo do vídeo em porcentagem, EX: If(vizualizacoes < 1000) o tempo é o mesmo , else If(visualizacoes > 1000 && vizualizacoes < 10000) aumento em 2% o tempo, duracao= tempo_video *(2/100), minha duvida é onde implemento, posso colocar de ntro do setDuracao na classe Video.
Comment: Ok Obrigado pela atenção.


Singular value decomposition procedure in R

Question: I'm trying to follow Prof Strang exercise
but I have a problem with the signs of the resultant matrix.
He also had a problem with the signs during the exercise, so I have no way to find what is the error.
Can somebody point me what I'm doing wrong?
Thanks in advance,
Diego
These are my results:
> A
     [,1] [,2]
[1,]    4    4
[2,]   -3    3
> SVD_of_A
     [,1] [,2]
[1,]   -4    4
[2,]   -3   -3
> U
     [,1] [,2]
[1,]   -1    0
[2,]    0   -1
> sigma
     [,1]     [,2]
[1,] 5.656854 0.000000
[2,] 0.000000 4.242641
> V
      [,1]       [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068  0.7071068

These are the results of the svd function
> svd(A)
$d
[1] 5.656854 4.242641

$u
     [,1] [,2]
[1,]   -1    0
[2,]    0    1

$v
       [,1]       [,2]
[1,] -0.7071068 -0.7071068
[2,] -0.7071068  0.7071068

This is the code I'm using for that exercise:
rm(list=ls())

#------------------------------------------------------------------
# Find the Singular Value Decomposition of the matrix A
#------------------------------------------------------------------

A = matrix(c(4,-3,4,3),2)

#------------------------------------------------------------------
# We want to obtain:
# SVD = U*sigma*t(V)
#------------------------------------------------------------------

#------------------------------------------------------------------
# First we need to obtain the transpose of A
# t = Given a matrix or data.frame x, t returns the transpose of x.
#------------------------------------------------------------------

At= t(A)

#------------------------------------------------------------------
# Multiply the 2 matrices, At and A to obtain AtA
# %*% = Matrix Multiplication
#------------------------------------------------------------------

AtA=At%*%A

#------------------------------------------------------------------
# Now we need to obtain the spectral decomposition of the matrix AtA
# eigen = Computes eigenvalues and eigenvectors of real
#         (double, integer, logical) or complex matrices.
#------------------------------------------------------------------

eig1 = eigen(AtA)

#------------------------------------------------------------------
# With the eigenvalues of AtA we can build sigma.
# I did this manually here for the purpose of the procedure.
# However note that sigma is equal to sqrt(AAt)
#------------------------------------------------------------------

sigma=matrix(c(sqrt(eig1$values[1]),0,0,sqrt(eig1$values[2])),2)

#------------------------------------------------------------------
# The eigenvectors of AtA are the values of V
#------------------------------------------------------------------

V=as.matrix(eig1$vectors)

#------------------------------------------------------------------
# Next we need to compute A*At
#------------------------------------------------------------------

AAt=A%*%At

#------------------------------------------------------------------
# Now we need to obtain the spectral decomposition of the matrix AAt
#------------------------------------------------------------------

eig2=eigen(AAt)

#------------------------------------------------------------------
# U is equal to the eigen vectors of AAt
#------------------------------------------------------------------

U=as.matrix(eig2$vectors)

#------------------------------------------------------------------
# Finally the singular value decomposition of A is equal to U*sigma*V
#------------------------------------------------------------------

SVD_of_A=U%*%sigma%*%V

#------------------------------------------------------------------
# Results
#------------------------------------------------------------------

A
SVD_of_A
U
sigma
V

PS: I've also tried V transpose, but the results are still wrong...
Comment: You're not doing anything wrong, nor is the software. The SVD is not unique. If we perturb the column vectors of both of the basis matrices by the same sign matrix, we obtain another valid SVD for the same matrix. This is also true for eigendecompositions of symmetric matrices. I didn't run your code, but what almost undoubtedly happened was that each of your `eigen` calls ended up choosing a different sign pattern for the basis vectors. There's no algorithmic error here.
Comment: For an extended illustration of this phenomenon (and some ways to deal with it) please see http://stats.stackexchange.com/questions/34396/im-getting-jumpy-loadings-in-rollapply-pca-in-r-can-i-fix-it/34401#34401.  In addition to what @Cardinal said, there are also the rare cases where the dimension of an eigenspace is greater than $1$: the software is then free to pick any arbitrary orthonormal basis of such an eigenspace. It would be strange for two software solutions ever to agree in such instances, unless they used exactly the same underlying code.
Comment: Thank you very much cardinal and whuber for the explanations! I'll take a look at the link. Just wondering... Why or better How the SVD function in R is able to produce always the same result?
Answer: I did the exact same thing with Matlab/Octave and got the right result.
A = [4 4;-3 3]

At= transpose(A)
AtA = At * A
AAt = A * At

[vec1, val1] = eig(AtA)
V = vec1

[vec2, val2] = eig(AAt)
U = vec2

sigma=sqrt(val1)

SVD_of_A = U * sigma * transpose(V)

Result:
SVD_of_A =

   4.0000   4.0000
  -3.0000   3.0000

The error is that $eigen(AAt)$ should produce eigenvector equals to have the right result to:
      [,1] [,2]
[1,]    1    0
[2,]    0    1

See comment of Cardinal
Comment: No, $R$ did not do anything "wrong" in this instance. This comes down to understanding that the SVD is not unique. The vectors are invariant to common sign flips, and this is precisely what happened here.


root of $x^3+x+1$ over $\mathbb{F}_5$

Question: What is the simplest radical expression of some root $a \in \overline{\mathbb{F}_5}$ of the polynomial $x^3+x+1  \in \mathbb{F}_5[x]$? I wonder if one can simplify the general formulas in this special case. I think $\sqrt[3]{\sqrt{2} - 3} - \sqrt[3]{\sqrt{2} + 3}$ is ok for the beginning, but this minus in the middle and the nested roots make it rather complicated.
Answer: I think, this cannot be much simplified.
All elements ($0,\pm 1,\pm 2$) are cubic roots, so one $\sqrt[3]\ $ is not enough. The simplest you can get then (if you insist on root signs) is to introduce a $\sqrt\ $.
But our field extension $\mathbb F_{125} \cong \mathbb F_5(a)$ has degree $3$, so doesn't contain any pure square root, so one additionally need a cubic root finally, and this minus trick with the conjugate, to get back of degree $3$..
Comment: Yeah, but I hope for something like $\sqrt[3]{\sqrt{...}}$ without $\pm$.
Comment: Perhaps even two such simple radical expressions which generate the field.
Answer: $\mathbb F_{5^3}$ is a cubic extension of $\mathbb F_5$, and the first thing to ask if it we can write it as $\mathbb F_5(\sqrt[3]a)$. Unfortunately, 5 is not congruent to 1 mod 3, so $x \mapsto x^3$ is a permutation of $F_5$, and the polynomials $X^3 - a$ are always reducible.
In order to express $\alpha$ with radicals, you will need a cube root, so we have no choice but to go to $F_{5^2}$ first by taking a square root, then use a cube root to go to $F_{5^6}$, and then express $\alpha$ just like you did.
Now, rearranging your expression, we have $\alpha = \sqrt[3]{2 + \sqrt{2}} + \sqrt[3]{2 - \sqrt{2}}$. It is clear it is in $\mathbb F_{5^3}$ because it is invariant by the automorphism of $\mathbb F_{5^2}$ switching $\sqrt 2$ with $ - \sqrt 2$.
However it is unclear because we pick two different cube roots and someone might make a mistake and not take conjugated roots. It may be better to write $\sqrt[3]{2 - \sqrt{2}}$ = $\frac 1 {\sqrt[3]{2 + \sqrt{2}}}$ instead, so $\alpha = \sqrt[3]{2 + \sqrt{2}} + \frac 1 {\sqrt[3]{2 + \sqrt{2}}}$
Comment: You don't offer a simplification, so why do you answer? I already know that $3$ is coprime to $4$ ;).
Comment: @ Martin : well I still explain why there is no simpler combination of radicals (unless you accept useless descriptions like x = 124th root of 1), and I give an alternate less ambiguous description, which is the best you can expect. I don't know what you want more, and of course yes 3 is also coprime to 7 =D

root of $x^3+x+1$ over $\mathbb{F}_5$


Cálculo de IMC automático em INPUT

Question: Estou com problemas com esse codigo, quando eu realizo ela dá um resultado errado e não consigo limitar casa decimais.

function calcular() {
  var peso = parseInt(document.getElementById('peso').value);
  var alt = parseInt(document.getElementById('alt').value);
  document.getElementById('imc').innerHTML = peso / Math.pow(alt, 2);
}
 <form action="#" method="post" enctype="multipart/form-data">

Peso

    <input type="text" class="form-control" id="peso" name="peso" placeholder="Insira Peso (kg)" onfocus="calcular()">

Altura

    <input type="text" class="form-control" id="alt" name="alt" placeholder="Insira Altura (m)" onblur="calcular()">

IMC

</form>
Comment: Possível duplicata de [Cálculo de índice de massa corporal (IMC)](https://pt.stackoverflow.com/questions/27440/c%c3%a1lculo-de-%c3%adndice-de-massa-corporal-imc)
Answer: Basicamente eu fiz algumas correções no seu seu código principalmente em JavaScript.
No seu html eu apenas alterei o onfocus e onblur para onkeyup porque pelo que entendi e análise pelo código você quer que cada número que o usuário digitar seja feito o cálculo do imc e para isso é necessário o onkeyup e também alterei o tipo do input para number.
O código JavaScript abaixo esta todo comentado e o HTML tem as alterações.
JavaScript
/**
* Sugestao(oes):
*  - Verificar se o usuario digitou a altura
*    realmente em metros.
*/

function calcular() {
  // Pega o valor do peso.
  var peso = document.getElementById('peso').value;

  // Verifica se o simbolo de casas decimais e
  // o do Brasil.
  if (peso.indexOf(",") > -1) {
    // Se sim ,corrige para o padrao americano
    // e programavel.
    peso.replace("," ,".");

  // Verifica se o valor de 'peso' nao esta vasio.
  } else if (!(peso)) {
    // Se sim ,define para o valor para 1.
    peso = 1;
  }

  // Converte 'peso' para numero decimal.
  peso = parseFloat(peso);

  // Verifica se 'peso' e igual a zero.
  if (peso == 0) {
    // Se sim ,define 'peso' como 1.
    peso = 1;
  }

  // Pega o valor da altura.
  var alt = document.getElementById('alt').value;

  // Verifica se o simbolo de casas decimais e
  // o do Brasil.
  if (alt.indexOf(",") > -1) {
    // Se sim ,corrige para o padrao americano
    // e programavel.
    alt.replace("," ,".");

  // Verifica se o valor de 'alt' nao esta
  // vasio.
  } else if (!(alt)) {
    // Se sim ,define para o valor para 1.
    alt = 1;
  }

  // Converte 'alt' para numero decimal.
  alt = parseFloat(alt);

  // Verifica se 'alt' e igual a zero.
  if (alt == 0) {
    // Se sim ,define 'alt' como 1.
    alt = 1;
  }

  // Cacula e exibe o imc ao usuario.
  document.getElementById('imc').innerHTML = (peso / Math.pow(alt, 2)).toFixed(2);
}

HTML - Form

<form action="#" method="post" enctype="multipart/form-data">

Peso

      <input type="number" class="form-control" id="peso" name="peso" placeholder="Insira Peso (kg)" onkeyup="calcular()">

Altura

      <input type="number" class="form-control" id="alt" name="alt" placeholder="Insira Altura (m)" onkeyup="calcular()">

IMC

</form>

Comment: Muitooo obrigado, funcionou perfeitamente...... valeu mesmo amigo!!
Comment: Muito por nada.
Answer: Basta fazer (remova o parseInt)
 function calcular() {
  var peso = 68.00;
  var alt = 1.0;
  var imc = parseFloat( peso / Math.pow(alt, 2)).toFixed(2);

  alert(imc)

}

calcular();

Comment: Não funcionou !
Comment: você tem que remover o ParseInt()

Cálculo de IMC automático em INPUT


Supremum $\sup_{x\in (0,+\infty)}|F(x,t)|=\sup_{x\in (0,+\infty)}\dfrac{e^{-t^2}}{x+|t|}$

Question:
Let :
  \begin{aligned}F \colon (0,+\infty)\times \mathbb{R} &\longrightarrow \mathbb{R} \\(x,t) &\longmapsto F(x,t)=\dfrac{e^{-t^2}}{x+|t|}.\end{aligned}

How they do to find  $\displaystyle \sup_{x\in (0,+\infty)}|F(x,t)|$

$$ \fbox{ $\sup_{x\in (0,+\infty)}|F(x,t)|=\sup_{x\in (0,+\infty)}\dfrac{e^{-t^2}}{x+|t|}=\dfrac{e^{-t^2}}{|t|} $} $$
Indeed,
$\forall t\in \mathbb{R}\quad F(.,t)=\dfrac{e^{-t^2}}{x+|t|}$
$$\frac{\partial F}{\partial x}(x,t)=\dfrac{-e^{-t^2}}{(x+|t|)^2} $$
I'm stuck here
Answer: The supremum of this value is achieved when the denominator has the infimum value which is achieved for $x=0$.
Answer: For any fixed $t$, one may see it as a function of a single variable $x$:
$$
f(x)=\dfrac{e^{-t^2}}{x+|t|}, \quad x>0,
$$ giving
$$
f'(x)=-\dfrac{e^{-t^2}}{(x+|t|)^2}<0,
$$ thus $f$ is decreasing over $(0,\infty)$, its supremum is obtained as $x \to 0^+$:
$$
\lim_{x \to 0^+}f(x)=\lim_{x \to 0^+}\dfrac{e^{-t^2}}{x+|t|}=\dfrac{e^{-t^2}}{|t|}.
$$


Posterior distribution of parameter

Question: I am looking at pg. 257 of the textbook "Statistical computing with R" by Maria L. Rizzo, where an example is given of a simple model of stock returns:
We have 5 stocks, and at the end of a year of 250 trading days, we have a vector [$x_1$, $x_2$, ..., $x_5$], where $x_i$ is the number of days that stock $i$ was the best performing stock. This is a multinomial joint distribution, with probability vector $p = [\frac{1}{3}, \frac{1 - \beta}{3}, \frac{1 - 2\beta}{3}, \frac{2\beta}{3}, \frac{\beta}{3} ]$, with $\beta \in (0, 0.5)$.
Now, what I don't understand, is that in the textbook they now say that the posterior probability distribution of $\beta$, which now represents the parameter of the multinomial distribution here, is
$Pr[\beta | (x_1, ..., x_5)] = \frac{250!}{x_1!x_2!x_3!x_4!x_5!}p_1^{x_1}p_2^{x_2}p_3^{x_3}p_4^{x_4}p_5^{x_5}$
which is also the pmf (probability mass function) of the multinomial distribution, (given the parameter) i.e.
$Pr[(x_1, ..., x_5) | \beta] = \frac{250!}{x_1!x_2!x_3!x_4!x_5!}p_1^{x_1}p_2^{x_2}p_3^{x_3}p_4^{x_4}p_5^{x_5}$
According to Bayes
$Pr[\beta | (x_1, ..., x_5)]  = \frac{Pr[(x_1, ..., x_5) | \beta]Pr[\beta]}{Pr[(x_1, ..., x_5)]}$
So the only way $Pr[\beta | (x_1, ..., x_5)] = Pr[ (x_1, ..., x_5) | \beta]$ then is if $Pr[\beta] = Pr[(x_1, ..., x_5)]$, but I am assuming here that $\beta$ is uniformly distributed, making $Pr[\beta]$ a constant. $Pr[(x_1, ..., x_5)]$ is not a constant, if I am correct, because of combinatorics.
So what am I missing here?
Comment: The data are **fixed** and so the (marginal) probability of the data **is** a constant.
Comment: @mef: as a _function_ of $(x_1,x_2,\ldots,x_5)$ the normalising constant of the posterior on $\beta$ varies for different samples $(x_1,x_2,\ldots,x_5)$.
Answer: While I do not have access to this book, I think the first equation is missing a proportionality constant, i.e., that the posterior on $\beta$ should be
$$\begin{align*}\pi(\beta | (x_1, ..., x_5)) &\propto \dfrac{250!}{x_1!x_2!x_3!x_4!x_5!}p_1^{x_1}p_2^{x_2}p_3^{x_3}p_4^{x_4}p_5^{x_5}\\
&\propto p_2(\beta)^{x_2}p_3(\beta)^{x_3}p_4(\beta)^{x_4}p_5(\beta)^{x_5}\\&\propto \{1 - \beta\}^{x_2}\, \{1 - 2\beta\}^{x_3}\, \beta^{x_4}\, \beta^{x_5}\\&\propto \{1 - \beta\}^{x_2}\, \{1 - 2\beta\}^{x_3}\, \beta^{x_4+x_5}\end{align*}$$
with a manageable normalising constant since this is a polynomial in $\beta$. You are right that the marginal $p(x_1, ..., x_5)$ is not a constant. For instance,
$$\begin{align}\int_0^{1/2} \{1 - \beta\}^{3}\, &\{1 - 2\beta\}^{6}\, \beta^{9} \text{d}\beta = (1/2)^{10}/10 - (15(1/2)^{11})/11 \\
&+ (33(1/2)^{12})/4 - 29(1/2)^{13} + (456(1/2)^{14})/7 - (484(1/2)^{15})/5\\
&+ 95(1/2)^{16} - (1008(1/2)^{17})/17 + (64(1/2)^{18})/3 - (64(1/2)^{19})/19 \\ &= 4.38\, 10^{-9}
\end{align}$$
and
$$\begin{align}\int_0^{1/2} \{1 - \beta\}^{1}\, \{1 - 2\beta\}^{2}\, \beta^{3} \text{d}\beta &=
(1/2)^4/4 - (1/2)^5 + (4(1/2)^6)/3 - (4(1/2)^7)/7 \\
&= 7.44\, 10^{-4}
\end{align}$$
For a similar problem with a Gibbs sampling resolution, see Tanner & Wong (1987) [also processed in our book, pp.347-348].
Comment: Excellent answer. Thanks for clearing things up (and giving some extra insight)!

Posterior distribution of parameter


Fully delete roaming profile on terminal server upon user logoff

Question: We're using roaming profiles on our terminal servers (running Win2k8). Occasionally, it is necessary to delete some of them, for example because of corruption or to test something. They store only preferred settings (e.g. displayed views in applications), no user data.
Of course, we can only delete profiles while the affected user is logged off. We'd like to be able to delete a profile at any time, though.
Is it possible to configure that a profile is not updated on the profile server when a user logs out, but deleted instead? This should only happen once, so when he logs in again, a new profile would be created. Alternatively, it would be fine as well if the server profile would be deleted immediately and the update process be prevented once.
Or to put it differently: How can we ensure a specific user will receive a new, clean profile when he logs himself in the next time only?
Comment: Can I ask the most important question: Why? .. Why do you want to delete the profile whilst the user is logged in?
Comment: @TomO'Connor he basically wants to *schedule* the deletion so he would not need to wait for the user's logoff to run the necessary commands.
Comment: And if the user's logged in, whilst you delete their profile, doesn't all hell break loose?
Comment: syneticon-dj is right. If the user is not logged in, delete the profile. If he _is_ logged in, delete it as soon as it's safe to do so (= when he logs off).
Answer: While there is a "Prevent Roaming Profile Changes From Propagating to the server" GPO setting from "Computer Configuration\Administrative Templates\System\User Profiles" which changes the "ReadOnlyProfile" value in the HKLM\Software\Policies\Microsoft\Windows\System regkey, it is a per-machine configuration setting valid for all users logging on to your terminal servers, and thus probably undesirable. Also, I am not sure if it would affect profiles from already logged-on users.
I believe the only method to achieve what you want would be through scripting.

parse a file for the list of user profiles to delete
iterate through this list

check if user is logged on
if not, delete her roaming profile and the local copy

wait for a while before proceeding
go to 1.

This looks simple enough even to be scripted as a CMD batch, but obviously using PowerShell would produce more elegant and resilient code.
Answer: Create a new GPO Object, and link it to the OU of the users (or computers).
<code>Computer Configuration => Policies => Administrative Templates => System/User Profiles => Delete cached copies of roaming profiles == Enabled
</code>
This worked for me in the past.
Comment: the OP is looking for a mechanism to force a one-time profile reset while the user is logged on. The GPO setting to delete local profile copies only would delete the local copy after updating the profile on the server store, so it would not be of much use for this case.
Answer: Set this script as a powershell "logoff" script, you will need to make sure users have proper perm on the "ProfileList" registry folder. ( our users did not  )
Essentially what it does is during the logoff process it changes the Profile State to a "128" which is guest account and nukes they profile while logging off.
Just copy and paste this as a powershell script
$SID = ([Security.Principal.WindowsIdentity]::GetCurrent()).User.Value
Set-ItemProperty -path "HKLM:\Software\Microsoft\Windows NT\CurrentVersion\ProfileList\$SID\" -name State -value 128
This is handy when you have apps like SKYPE that do not allow manadatory profiles or guest acccounts that do not have permissions to write to the cert store.


Ordenação Array PHP

Question: Como posso resolver o seguinte problema:
Possuo 4 arrays que são concatenados (array_merge) e viram apenas um, porém, agora preciso concatenar os vetores de uma forma diferente.
Preciso pegar o índice 1 de cada array, depois o índice 2, e assim sucessivamente. (Nem todos tem a  mesma quantidade de índices) até preencher um único array.
Comment: cade os `arrays` em um exemplo minimo? poderia postar na sua pergunta?
Comment: Entendo, eu particularmente não gosto de pedir ajuda com códigos, pelo fato de pessoas acharem que estou me aproveitando, neste caso gostaria apenas de mais cabeças para pensar em uma solução.
Comment: Descrevendo o problema geral você vai conseguir apenas uma resposta geral. Exemplo: "Como construir uma casa?", "Use blocos e cimento, construa paredes sólidas.". Em vez disso pergunte algo específico e respondível de forma útil: "Como levantar uma parede usando esse tipo de bloco com tal inclinação e de tal altura de forma segura?", resposta: "Posicione os blocos no formato tal, execute esse algoritmo de colocação de blocos, não use essa ferramenta pois há o risco tal, aqui tem um exemplo funcionando de uma parede pronta para você ver como faz [link]". Vê? Perguntas muito amplas não ajudam.
Comment: Pode fazer isso com `array_map()`
Answer: Não programo em PHP mas fiz um algoritmo super simples que resolve o problema com javascript.
Basicamente ele verifica qual o array mais longo e faz um for em cima do tamanho máximo. Em cada passada pelo looping ele verifica se cada um dos arrays possui o índice atual e adiciona em um novo array.
var a = [1,2,3,4,5];
var b = ['a','b','c'];
var c = ['aa','bb','cc','dd','ee','ff'];

var elem = document.getElementById('new_array');

var new_array = [];

var max_length = 0;

// Verifica qual o maior array
if (a.length > b.length && a.length > c.length) {
    max_length = a.length;
} else if (b.length > a.length && b.length > c.length) {
    max_length = b.length;
} else {
    max_length = c.length;
}

// Percorre todos os arrays com base no maior deles
// Verifica se o indice atual existe e adiciona ao novo array
for (var i = 0; i < max_length; i++) {
  if (a[i]) {
    new_array.push(a[i]);
  }

  if (b[i]) {
    new_array.push(b[i]);
  }

  if (c[i]) {
    new_array.push(c[i]);
  }
}

elem.innerHTML = JSON.stringify(new_array)

Output: [1,"a","aa",2,"b","bb",3,"c","cc",4,"dd",5,"ee","ff"]
Comment: Leandro, obrigado, era exatamente isso que você citou. testei isso em código php e funcionou perfeitamente. No inicio achei que essa opção era inviável pois iria estourar em algum dos vetores o index, mas com seu if antes de atribuir solucionou o problema.

Ordenação Array PHP


time slice per day for two timestamps

Question: I've two timestamps with a gap between a few hours and a few days.
Is there a way in PHP and / or MySQL to get the time slice for every day?
INPUT
Timestamp 1: 2013-10-30 10:00:00
Timestamp 2: 2013-10-31 11:00:00

OUTPUT
2013-10-30: 14 hours
2013-10-31: 11 hours

Comment: Check the [DateTime](http://php.net/manual/en/class.datetime.php) object of PHP or if they are MySQL timestamps check the DATE_ functions of MySQL to handle date calculations. But its not totally clear for me where you get 14h and 11h from...
Comment: Timestamp 1 is the start. This day (30th) has 14 hours until midnight. The end is timestamp 2. From 0:00 a.m. until 11:00 its 11 hours.
Answer: Easiest way to perform operations on dates is to convert them to timestamps.
Here, what you can do is to get three timestamps :
$ts1 = strtotime("2013-10-30 10:00:00");
$ts2 = strtotime("2013-10-31 00:00:00");
$ts3 = strtotime("2013-10-31 11:00:00");

$first_time_slice = ts2 - ts1; //here you got your first time slice in seconds, convert it to hours
$second_time_slice = ts3 - ts2; //idem for the second time slice

Answer: It can be done in MySql like this:
SELECT TIMEDIFF(TIMESTAMP(DATE('2013-10-30 10:00:00') + INTERVAL 1 DAY), '2013-10-30 10:00:00'),
TIMEDIFF('2013-10-31 11:00:00', TIMESTAMP(DATE('2013-10-31 11:00:00')))

time slice per day for two timestamps


Postfix tls errors with gmail as relayhost

Question: while configuring a php contact form ive run into a bizarre issue where postfix refuses to send an email, I've narrowed the errors down to one and occasionally a second which.
The first is :
from /var/log/maillog
 connect to smtp.gmail.com[2607:f8b0:400d:c03::6c]:587: Network is unreachable
Jun 11 16:57:59 site postfix/smtp[1315]: E79467A7: to=<apache@site.[domain].com>, relay=smtp.gmail.com[126.96.36.199]:587, delay=0.19, delays=0.01/0/0.16/0.02, dsn=5.5.1, status=bounced (host smtp.gmail.com[188.8.131.52] said: 530-5.5.1 Authentication Required. Learn more at 530 5.5.1  https://support.google.com/mail/?p=WantAuthError y131-v6sm20355089qka.30 - gsmtp (in reply to MAIL FROM command))

Allow less secure apps is enabled as recommended in the error log, and I've added tls support in main.cf
dump of main.cf
queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
mail_owner = postfix
#myhostname = [site].[domain].com
#mydomain = [domain].com
myorigin = $mydomain
inet_interfaces = all
inet_protocols = all
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
unknown_local_recipient_reject_code = 550
#mynetworks = <192.168.122.0/24, 192.168.100.0/24, 127.0.0.0/8>
relayhost = [smtp.gmail.com]:587
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases

debug_peer_level = 2
debugger_command =
         PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
         ddd $daemon_directory/$process_name $process_id & sleep 5
sendmail_path = /usr/sbin/sendmail.postfix
newaliases_path = /usr/bin/newaliases.postfix
mailq_path = /usr/bin/mailq.postfix
setgid_group = postdrop
html_directory = no
manpage_directory = /usr/share/man
sample_directory = /usr/share/doc/postfix-2.10.1/samples
readme_directory = /usr/share/doc/postfix-2.10.1/README_FILES
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_policy_maps = hash:/etc/postfix/tls_policy
smtp_tls_security_level = encrypt
smtp_tls_CAfile = /etc/pki/tls/cert.pem

I'm not familiar with postfix, how it works, or what it requires.
Any help would be appreciated.
Comment: Good answers that will probably get you on the right track. But here's a little help that might go a long way to explain this authentication issue: http://postfix.state-of-mind.de/patrick.koetter/smtpauth/smtp_auth_mailservers.html
Answer: As for the first message:
connect to smtp.gmail.com[2607:f8b0:400d:c03::6c]:587: Network is unreachable

This means that the network configuration of your server doesn't allow postfix to reach an IPv6 address, but that you configured postfix to try and use IPv6.
You should check your host configuration and eventually if you don't have access to an IPv6 network, disable ipv6 processing in postfix:
postconf -e inet_protocols=ipv4

As for the second:
Jun 11 16:57:59 site postfix/smtp[1315]: E79467A7: to=<apache@site.[domain].com>, relay=smtp.gmail.com[18.104.22.168]:587, delay=0.19, delays=0.01/0/0.16/0.02, dsn=5.5.1, status=bounced (host smtp.gmail.com[126.96.36.199] said: 530-5.5.1 Authentication Required. Learn more at 530 5.5.1  https://support.google.com/mail/?p=WantAuthError y131-v6sm20355089qka.30 - gsmtp (in reply to MAIL FROM command))

The message can be read like this:

relay=smtp.gmail.com[18.104.22.168]:587, delay=0.19, delays=0.01/0/0.16/0.02

Postfix sent the mail to the relay host specified

dsn=5.5.1, status=bounced

The host refused to handle the message

host smtp.gmail.com[18.104.22.168] said: 530-5.5.1 Authentication Required. Learn more at 530 5.5.1  https://support.google.com/mail/?p=WantAuthError y131-v6sm20355089qka.30 - gsmtp (in reply to MAIL FROM command)

This is an explaination sent by the relayhost you use for the rejection. In this case it's google way of saying that you didn't send an login/password to authenticate on the relay, or that the login/password used is incorrect.

To explain/help you further the content of /etc/postfix/sasl_passwd and /etc/postfix/tls_policy would be useful.
/etc/postfix/sasl_passwd should contain a line like that
smtp.gmail.com <gmail-accound-username>:<gmail-password>

Comment: Oh, Looks like i was an idiot and used smtp.google.com for some friggin reason instead...
Comment: `postconf -e ...` so elegant! :+1:
Answer: Regarding the first error I do agree with silmaril.
You've enabled IPv6 for postfix but v6 networking is not configured on your system, or a firewall is blocking v6 connections to smtp.gmail.com.
Regarding the second issue I guess you probably missed to run the following while configuring postfix for using Gmail SMTP as relay:
sudo postmap /etc/postfix/sasl_passwd


DRF IntegrityError: NOT NULL constraint failed: user_id

Question: I can't figure out how to pass user object to the following serializer:
class ReviewSerializer(serializers.ModelSerializer):
    user = UserSerializer(read_only=True)

    class Meta:
        model = Review
        fields = ('pk', 'title', 'user', 'movie', 'timestamp', 'review_text',)

I have this viewset:
class ReviewsViewSet(viewsets.ModelViewSet):
    queryset = Review.objects.all()
    serializer_class = ReviewSerializer

and this model:
class Review(models.Model):
    title = models.CharField(max_length=255)
    user = models.ForeignKey(User, on_delete=models.CASCADE, related_name='reviews')
    movie = models.ForeignKey(Movie, on_delete=models.CASCADE, related_name='reviews')
    review_text = models.TextField()
    timestamp = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return '{movie} review by {user}'.format(user=self.user, movie=self.movie)

My javascript request looks like this:
return axios({
  method: 'post',
  url: 'http://localhost:8000/api/reviews/',
  data: { // Using data from Vue
    title: this.review_title,
    movie: this.id,
    review_text: this.review_text,
    user: JSON.stringify(this.user)
  },
  headers: {
    'Content-Type': 'application/json',
    Authorization: `JWT ${token}`
  }
})

It gives me this traceback.
How should I pass the user object to the request?
Thanks in advance.
Answer: Remove read_only=True from serializer
class ReviewSerializer(serializers.ModelSerializer):
    user = UserSerializer()

    class Meta:
        model = Review
        fields = ('pk', 'title', 'user', 'movie', 'timestamp', 'review_text',)

If you set read_only=True, the DRF will not takes the value from input source even if it's there

From the doc,

Read-only fields are included in the API output, but should not be
  included in the input during create or update operations. Any
  'read_only' fields that are incorrectly included in the serializer
  input will be ignored.
Set this to True to ensure that the field is used when serializing a
  representation, but is not used when creating or updating an instance
  during deserialization.
Defaults to False

UPDATE
You should override the create() method of ReviewSerializer as
class ReviewSerializer(serializers.ModelSerializer):
    user = UserSerializer()

    def create(self, validated_data):
        user_dict = validated_data.pop('user')
        user_obj, created = User.objects.get_or_create(**user_dict)
        return Review.objects.create(user=user_obj, **validated_data)

    class Meta:
        model = Review
        fields = ('pk', 'title', 'user', 'movie', 'timestamp', 'review_text',)

for debug purpose only
class ReviewsViewSet(viewsets.ModelViewSet):
    queryset = Review.objects.all()
    serializer_class = ReviewSerializer

    def create(self, request, *args, **kwargs):
        print(request.data)  # print here <<<<
        return super(ReviewsViewSet, self).create(request, *args, **kwargs)

Comment: Oh, ok, now it's working. Thank you for detalied answer, I learned a bit how to debug DRF now.
Comment: Yes, but If I take out `read_only=True` it gives me: `"A user with that username already exists."`. So, I don't want to post a new user, just to take the user data needed to create that particular review.
Comment: Can you add the value of `user` that you're passing from front-end?
Comment: @yierstem Anyway I've updated the answer. Hope it helps
Comment: I pass the whole user object that I stored in Vuex so I can display user details and stuff. So it looks like this `"{"pk":1,"username":"test","first_name":"test","last_name":"test","email":"email@example.com"}"`. I sent a request with all of the fields required, especially user field, and returns another error: `"user": [ "This field is required." ]`.
Comment: Yes, I edited the serializer according to your answer.
Comment: The error says that the `user` data not reaching to DRF view
Comment: @yierstem print the data in DRF view and add it here, (check the updated answe)
Comment: It does not reach create() method. I tried your updated answer, it displays nothing, only `Bad request` and the status code. Note: I tried to display some random string, to see if it displays anything at all.
Comment: create method of view or serializer?
Comment: Oh, sorry, It was for the viewset, since it has request, my bad. Here: ``
Comment: Can you add the result of `print(type(request.data['user']))` in `create()` method of **view** ?
Comment: `` is the output
Comment: Then, Change `user: JSON.stringify(this.user)` to `user: this.user` in your JS code
Answer: just add this
class ReviewSerializer(serializers.ModelSerializer):
    user = UserSerializer(read_only=True)
    # new line
    user_id = serializers.IntegerField(write_only=True)

    class Meta:
        model = Review
        fields = (
            'pk', 'title', 'user', 'movie', 'timestamp', 'review_text',
            'user_id', # new field
        )

read_only
Read-only fields are included in the API output, but should not be included in the input during create or update operations. Any 'read_only' fields that are incorrectly included in the serializer input will be ignored.
Set this to True to ensure that the field is used when serializing a representation, but is not used when creating or updating an instance during deserialization.
Defaults to False

write_only
Set this to True to ensure that the field may be used when updating or creating an instance, but is not included when serializing the representation.
Defaults to False


Como fazer uma tabela em um scaffold rails com os atributos de outro scaffold

Question: pessoal estou com uma dificuldade!
Criei um scaffold chamado livro e um chamado emprestimos
depois criei um chamado relatorio no relatorio quero fazer uma busca de emprestimos por livros
estou usando a gem ransack para fazer buscas dentro do mesmo scaffold, mas como nesse caso e dentro de outro scaffold nao estou conseguindo
criei o scaffold relatorios asssim:
rails g scaffold relatorio livro:references emprestimo:references
Answer: Quando você usa o scaffold ele gera apenas uma parte do código.
Se você quiser dar mais vida ao software é preciso escrever os relacionamentos.
Você precisa abrir as classes Livro, Emprestimo e Relatorio e edita-las, adicionando o relacionamento entre elas.
Você pode usar o has many through association
<code># model/livro.rb
class Livro < ApplicationRecord
  has_many :relatorios
  has_many :emprestimos, through: :relatorios
end

# model/relatorio.rb
class Relatorio < ApplicationRecord
  belongs_to :livro
  belongs_to :emprestimo
end

# model/emprestimo.rb
class Emprestimo < ApplicationRecord
  has_many :relatorios
  has_many :livros, through: :relatorios
end
</code>


Is it possible to use bifactor model to predict another construct in SEM?

Question: Background info: I've explored the dimensionality of a construct with a four-correlated-factor model, which yielded a good model fit but the factor correlations were too high (all above .70). A bifactor model could better capture the dimensionality of this construct, with one general factor and four uncorrelated specific factors.
Problem: I want to check the relation between this construct and an outcome variable. Given that the discriminant validity of the four-correlated-factor model was inadequate, can I use the bifactor model to represent the construct in SEM? I've never come across a research article that did this and I'm wondering if it is feasible to do this analysis.
Answer: You are correct; there are relatively few applications of SEM that include latent variables modeled using the bifactor structure. This being said, a handful of articles have used this approach. For example, Tarescavage, et al. (2015) used the bifactor SEM in the context of constricted validation, so you may find it helpful. I also know for a fact that mplus can estimate bifactor SEM models, and I believe the R package lavaan can as well. So to answer your question, yes, you can use the bifactor model to represent your construct in the context of SEM.
If you do decide to take the bifactor SEM approach, there are a couple of things you should be aware of. First, the bifactor model has a tendency to fit even randomly generated data sets well (e.g., see Bonifay & Cai, 2017 for more info), so you should not be surprised if your bifactor SEM model fits better than competing models. Second, SEM  fit indices (e.g., CFI, TLI, RMSEA, SRMR) are known to be sensitive factors such as the number of items, factor loadings, sample size, and estimation. Given this, and also the fact that bifactor SEM model fit is an area that has received little attention in the methodological literature, I would caution you to be wary when using established cutoffs (e.g., those given by Hu & Bentler, 1999) to interpret how well your bifactor SEM model fits the data.
References
Bonifay, W., & Cai, L. (2017). On the complexity of item response theory models. Multivariate Behavioral Research, 52(4), 465-484.
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: a multidisciplinary journal, 6(1), 1-55.
Tarescavage, A. M., Finn, J. A., Marek, R. J., Ben-Porath, Y. S., & van Dulmen, M. H. (2015). Premature termination from psychotherapy and internalizing psychopathology: The role of demoralization. Journal of Affective Disorders, 174, 549-555.


File field label

Question: I have created a custom content type that has a file field.  When I upload a file the node displays the complete file name, test_file.pdf. I would rather display its file name as clickable link. I have tried manage display and set the format to URL to file and it displays "File: site_name.com/sites/default/files/test.txt," which is only text.
Is there anything I am missing?
Comment: Are you using the development snapshot of Drupal 7? I am using Drupal 7, but for a file field I can only select "File," as widget.
Answer: You are on the right track! Go to "Manage Display" and set the format to "Generic File", it will give you a clickable filename with a link to the file!
Comment: I am using that now and it's ok but I am looking for 'cleaner' look where the node displays the field label "File" and a link like "Download" rather than the file name, file.txt for example.  Hope I am being clear with what I am trying to accomplish.  Thanks!
Comment: Ok now I see, as far as I know you can't do that out of the box. You can either create a new formatter (probably kinda complicated) or just override the template for that kind of node (either using contemplate or plain .tpl.php)
Comment: @LandonAB Under the arrow to down vote, there is a check mark. I cannot see it in this question, as it has been asked by you, but that is the place where to look at. You should see [something like this](http://i.imgur.com/bzQAH.png).
Answer: I've created a module on drupal.org that provides a formatter that does just that: https://www.drupal.org/project/file_formatters.
Comment: Sweet!  I subscribed to it on drupal.org, anticipating a 7.x version.

File field label


Is there any way the adjusted $R^2$ might decrease by adding predictors?

Question: Let's consider a multiple linear regression formula:
$ \hat{y} = \beta_0 + \beta_1 \hat{x}_1 + \beta_2 \hat{x}_2 $ (1)
which produces adjusted $R^2 = r_1$.
Now I want to add to one predictor to the (1) which turns into:
$ \hat{y} = \beta_0 + \beta_1 \hat{x}_1 + \beta_2 \hat{x}_2 + \beta_3 \hat{x}_3 $ (2)
which produces adjusted $R^2 = r_2$.
If the data fed into (1) and (2) are exactly the same, is there any way to explain $r_2 < r_1$ apart from a code bug?
Comment: i think you might be confusing {R-sq} and {adjusted R-sq} here.
Comment: The very large number of threads on our site discussing [adjusted R-squared](http://stats.stackexchange.com/search?q=adjusted+squared) are worth reviewing in this context.
Comment: @charles indeed
Answer: Yes, it's definitely possible for adjusted $R^2$ to decrease when you add parameters.
Ordinary $R^2$ can't decrease, but adjusted-$R^2$ certainly can. We can write the relationship between the two like so:
$R_{adj}^2 = R^2-(1-R^2)\frac{p}{n-p-1}$
Note that both terms in the product $(1-R^2)\cdot\frac{p}{n-p-1}$ are positive (unless $R^2=1$), so if $R^2<1$, $R_{adj}^2 < R^2$.
If $R^2<\frac{p}{n-1}$, then adjusted-$R^2$ will be negative.
$R_{adj}^2$ will decrease if the $R^2$ for a model with an additional term if the second model's $R^2$ didn't increase from that for the first model by at least as much as would be expected for an unrelated variable.
We can see this happen quite easily: I just generated three unrelated variables in R (via)
x1=rnorm(20);x2=rnorm(20);y=rnorm(20)

If we fit a linear regression with just the first $x$ (lm(y~x1)), the adjusted $R^2$ is smaller than with the null model (which is 0):
Multiple R-squared:  0.0007048,   Adjusted R-squared:  -0.05481
If we fit both independent variables (lm(y~x1+x2)), the adjusted $R^2$ goes down again (and the $R^2$ – necessarily – goes up):
Multiple R-squared:  0.00199,   Adjusted R-squared:  -0.1154

For adjusted $R^2$ to increase, its addition has to explain more additional variation in the data than would be expected from an unrelated variable; it's possible for an unrelated variable to add less than it would be expected to, just by chance.


Flicker API issues, set_id

Question: Can somebody suggest me , why set_id doesn't work properly for Flicker API object call.
IN slideshow it works very well, it selects just images from this set
http://www.flickr.com/slideShow/index.gne?group_id=&user_id=26339121@N07&set_id=72157623827367793&tags=Cars,Lotus,Exige

but when I tru to use this one :
http://api.flickr.com/services/rest/?format=json&method='+
                'flickr.photos.search&api_key=' + apiKey + '&user_id=' + userId + '&set_id=72157623827367793&per_page=' + perPage + '&jsoncallback=?'

It select all images, seams it doesn't work at all !
Thanks!!
Answer: per the api documentation, it is not a valid field for that method
if you use the getList method of a set, you can specify a user.

Flicker API issues, set_id