Improving a forest model by dropping features below a percent importance threshold?

Question: I'm wondering if there is a term for this process, where I can find more reading/information about it, and if this is valid or hacky overall. I've only used this process for tree-based models which have easily accessible feature importances (random forest, xgboost, etc.)
The process is as follows: I train a preliminary model with all the features, and track each feature's % importance. I then loop through different thresholds (e.g. 0%, 2%, 4%...) and drop all the features, from the original model, that have a percent importance below the threshold. I then choose the model, features, and threshold that yield the best accuracy, since that is what I am interested in for this particular case.
Is this a known approach, and is there anything wrong with it? Where can I find more information about it?
Comment: I guess this is what is implemented in Python's `sklearn` as `SelectFromModel`. However, the docs do not come with references to papers etc. Maybe interesting to know anyway: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html#sklearn.feature_selection.SelectFromModel
Comment: How could that "improve" your model?

Dropping features might make it simpler or more efficient but how could it not also make it less accurate?
Answer: In theory, you could take such an approach, but there's a number of caveats.
Firstly, whether this is helpful for predictive performance will be rather case specific and it could certainly make your model performance worse on new data. E.g. you could imagine that it would be helpful in case you think that there could be a number of predictors that have no effect at all, while if all predictors may have some potentially useful information, it may heavily depend on the amount of data you have, how correlated predictors are and various other factors. Note that correlation between predictors might be a serious issue for your proposed approach (e.g. if you have many very closely related variables, you could end up with each having lowish feature importance for each, but together they are very important to the model).
Secondly, how do you decide what's a good threshold value to use? You definitely would not want to rely on performance on training data, but rather you'd want to pick the threshold e.g. through cross-validation. I.e. it is like any other hyperparameter of your algorithm (and note: other hyperparameters may have different optimal values after removing this variable, which is e.g. obvious for the proportion of features to randomly pick for each tree and/or split within a tree).
Thirdly, this will affect the interpretation of the final model. Given the models you talk about, I'm assuming you're primarily interested in predictive performance, but just to be clear: interpreting just the final created model may be rather tricky due to the preceding model selection. In this context, it's useful to be aware that the variable selection might be a pretty variable in its outcome. If you just do the same thing a few times with different random number seeds, you might get different results, but that still understates it, to get a fuller picture bootstrapping the data and looking how much additional variability that adds in what variables get selected can provide useful insights. For many model selection procedures it turns out that it's rather variable (hey, maybe you'd even get better results by doing this repeatedly and averaging the resulting models) and thus, we should not pretend that such a selection procedure truly selects the variables that are really important. On the other hand, a "well, we can't know whether other variables are maybe important, but we're predicting decently well with these" interpretation is more reasonable.


What happens when both players are on the black ball?

Question: In 8-Ball Pool, when you are on the black ball, if you foul (eg. don't hit the black ball), the other player doesn't get 2 shots because they are on the black ball.
What stops both the players from just tapping the ball every time? They are not at a loss if they do, because the other player only gets 1 shot.
Answer: By Pool, I take you are referring to a game of 8-ball (there are many variations of billiards games). As for your question, there are a few different ways that a player can foul on the eight ball. When fouling, the advantage is that the next player has ball in hand, allowing them to place the cue ball into a favorable position. So I would gladly have the player just tap the eight ball, as it increases the likelihood of leaving the cue ball in a favorable position for me to win the game.
Take a look at the rules for 8-ball to get some of the other specifics regarding fouls, etc.
http://en.wikipedia.org/wiki/Eight-ball
Comment: Yes, I mean the 8 ball game. But, when you are on the black ball, I thought the rule was that there is no such thing as a foul - you can do anything and the other player won't get 2 shots. Is that wrong?
Comment: @ṧнʊß What I've heard is if a player is on the 8-ball (the black ball) and they scratch (foul), then they automatically lose.  Also, I've never heard of a player being allowed a second shot if they didn't sink a ball on the first shot.
Comment: I agree, those seem like different rules then I have ever played with as well (I've always played that scratching on the 8 ball is an automatic loss as well). Perhaps it was some rule variation, or some home rules, that you played as you can most certainly foul with only the 8-ball left. Take a look at the wiki entry I provided, and then search from there for 'official' rules if you want the full scoop.
Answer: BCA standard (and European Standard) rules for 8 ball pool state that

After hitting any ball, the cue ball or the object ball or another ball contacted after contacting the right ball (something has to) has to touch a cushion/rail.
In case the rule above was violated, it results in a foul and the opponent is awarded a Ball in Hand

In case the 8 ball is stuck on a cushion/rail, after contacting the ball, something has to touch a DIFFERENT cushion/rail. It cannot be considered legal that you tap a frozen ball without contacting an entirely different rail/cushion.


Playing Hammond B3 organ parts in rock music. Need advice!

Question: I need advise about creating Hammond B3 parts in rock music. Recently I joined to rock in an rock-n-roll band. And here's the challenge in front of me: I do not know how to arrange and create my Hammond B3 parts to make it fit.
Any advice?
Answer: First, I suggest you study the organ playing of the keyboardists for classic rock bands, including:

Deep Purple
The Doors
Boston
Booker T. and the MG's
Spencer Davis Group & Traffic (Steve Winwood playing B3 in both bands)
Emerson, Lake, & Palmer
A very few Led Zeppelin songs

You might also check out some great B3 players who aren't exactly rock players:

Jimmy Smith (IMHO the all time king of B3)
Robert Walter
Larry Young
Richard "Groove" Holmes
So many others

Some things I would start off with to just get going right away:

Start with the drawbars at 888 000 000 or 808 000 000 or 888 800 000 or something like that. That's a basic sound. I'd also start with percussion on, soft, short/fast, 2nd or 3rd to taste, and the C3 setting on the chorus/vibrato. I like Leslie a lot, but starting without it simplifies things. I would also have just a touch of overdrive if you can get - just enough so that it sounds mostly clean with a little dirt when you play low and/or loud.
Start off with power chords (5ths) doubling the guitar and/or bass. If there's a bass player in the band, lay off the bass pedals if you have them. You can figure out how to work them in once you're really grooving with the band, but without careful attention, they will likely fight with the bass guitar player.
Once you're comfortable with the chord changes, add flavor to the chords and/or little melody notes or riffs when there's a gap. A gap means when there isn't anything else interesting going on in the music. When the singer is singing, keep it simple. If there's a guitar solo, keep it simple. If the singer sings a line and then there's a pause in the vocals, that's a time to try a little melody (unless the guitarist is playing something interesting there). Make sure you stop playing the melody again before the singer comes back in.
Get a volume pedal if you don't have one and learn to use it to keep your level sitting well with the other musicians. Start off playing more quietly than you think you should and ask the band members what they think of your level after a few songs. Constantly pay careful attention to your volume compared with the other instruments and try to keep your instrument sitting well. Sometimes you should be a little louder, like if they dig your little melody notes, kick the volume up a bit when you play them, then dial it back when the singer comes back in.
Listen carefully and constantly to both what they other players say about your playing and what they are playing. Sometimes the best thing to do is not play at all. Other times, being barely audible is the right thing. You have to figure out when to play very percussive and fast versus holding long notes (since the organ can do both). Listening to how it's done by the players I mentioned above should give you good ideas.
Comment: A great answer. The volume pedal is most important. Changing between the various B3 sounds means some are more powerful, and you do need to be able to balance them against each other and the band. Hands are busy enough! A pedal to turn Leslie on/off is also a great boon. +1.


How to evaluate $\lim_{n\to \infty }(\frac{n^{2}+1}{n-1})^{-2n^{2}}$?

Question:
How to evaluate the following limit?
  $$
\lim_{n\to \infty }(\frac{n^{2}+1}{n-1})^{-2n^{2}}
$$

I have tried the following:
$$
\begin{align}
\lim_{n\rightarrow \infty }(\frac{n^{2}+1+2n-2n}{n-1})^{-2n^{2}}
&= \lim_{n\rightarrow \infty}(\frac{(n-1)^{2}+2n}{n-1})^{-2n^{2}}\\
&= \lim_{n\rightarrow \infty}(\frac{(n-1)^{2}}{n-1}+\frac{2n}{n-1})^{-2n^{2}}\\
&= \lim_{n\to \infty }({n-1}+\frac{2n}{n-1})^{-2n^{2}}
\end{align}
$$

I know that:
$$
\lim_{n\rightarrow \infty }(1+\frac{1}{n})^{n}=e
$$
But I can't come to that form in my equation.
Can you help me, please?
Answer: $$\lim_{n\to\infty}\left(\dfrac{n^2+1}{n-1}\right)^{-2n^2}=\dfrac1{\lim_{n\to\infty}\left(n+1+\dfrac2{n-1}\right)^{2n^2}}=?$$
Answer: Note that:
$$\left(\frac{n^{2}+1}{n-1}\right)^{-2n^{2}}=\left(\frac{n-1}{n^{2}+1}\right)^{2n^{2}}<\left(\frac1n\right)^{2n^2}\to 0.$$
Answer: We have that
$$\left(\frac{n^{2}+1}{n-1}\right)^{-2n^{2}}=n^{-2n^2}\left(\frac{n+\frac1n}{n-1}\right)^{-2n^{2}}=n^{-2n^2}\left(\frac{n-1+1+\frac1n}{n-1}\right)^{-2n^{2}}=$$
$$=n^{-2n^2}\left(1+\frac{n+1}{n(n-1)}\right)^{-2n^{2}}=n^{-2n^2}\left[\left(1+\frac{n+1}{n(n-1)}\right)^{\frac{n(n-1)}{n+1}}\right]^{\frac{-2n^{2}(n+1)}{n(n-1)}}=$$$$=\frac1{n^{2n^2}\left[\left(1+\frac{n+1}{n(n-1)}\right)^{\frac{n(n-1)}{n+1}}\right]^{\frac{2n^{2}(n+1)}{n(n-1)}}}\sim \frac{1}{n^{2n^2}e^{2n}}$$
Comment: Great solution, thank you very much.
Comment: @violettagold It can be simplified a lot, I did in that way in order to use the knwn limit for $e$ but it is not necessary of course.
Comment: Yes, I have noticed. I had simplified my answer at the end.


SSH bounce box with transparent forwarding to backend hosts

Question: First, is such a thing possible? Consider the following setup:
One public facing "SSH Bounce Box"
bouncer.mydomain.tld

Four private *nix systems
host1.mynet.local
host2.mynet.local
host3.mynet.local
host4.mynet.local

All running Open-SSH on port 22.  Each private system has its own unique root password and unique users.
I want to be able to log into host1.mynet.local as root with this single command:
ssh firstname.lastname@example.com

The above would sign me on to host1.mynet.local SSH as root using the SSH bounce box without having to manually setup tunnels.
Also, consider the following possibility:
ssh email@example.com

This should connect to host3.mynet.local as the user janet.  The last "_" separated the username from the host identifier.
The problem is that this should be completely transparent to the end user.  Meaning the work is done server side and not client side.  The end user does not need to edit configuration files, setup tunnels, or edit .ssh/config files to make it work on there end.
Comment: Get IPv6 and skip the "bounce box".
Answer: I'll first describe a [unfortunate, hacky but easy to set up] solution. Then I'll describe why I don't think that exactly what you have described would be easy.

The machine at bouncer.mydomain.ltd will have one user for every user/host tuple. In each of the users on the bouncer machine will have a .bashrc:

ssh firstname.lastname@example.com
exit

Furthermore, public keys from the bouncer machine for each user can be placed in the authorized_keys directory for the respective host's intended user.
The failure of this is that information is redundant, and any change to the specific hosts will not propagate (unless you introduce some hacky scripting)

The previous solution is really an ssh connection opened up from the shell of an ssh connection. I think you really were imagining something more like a reverse proxy that routed based on whatever you put in the user area.
Unfortunately, it is hard to extract that user area information. (And there is no software that I know of that does this already) When you type

ssh email@example.com

ssh does not immediately just extract janet_host3 and send it over to the server. Rather..

SSH client and server negotiate over keys and algorithms
client/server establish a secure(encypted) communication channel

Only then can the authentication happen. Through an encrypted channel.
It's important to note that your server cannot know which machine to proxy the connection information to until after some of the communication has been completed.
For reference, the following is an example some of the first pieces of information that a server receives (I ran ssh to my localhost).

SSH-2.0-OpenSSH_6.0p1 Debian-3ubuntu1
......%{A.Q..3....*~\.....ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-ex

So to solve this:

Your server would have to negotiate its way through the transport stage, set up the encrypted communication channel, and then snipe the user information.
Then, it can parse the user information and establish a new connection with the intended destination.
After this, it could funnel everything the client has sent, and everything the client says from then on to the intended server.

Depending on why you want to set this up, maybe I can propose some alternate solutions :

If you just want aliases so that users have to remember different information, use DNS and cnames so that hostn.bouncer.mydomain.tld sends users to the machine that they intend.
If you wanted to be able to monitor traffic using the bouncer machine.. you'll need to do one of the things described above. To sniff encrypted traffic, you have to be a middleman such that you decrypt the information then encrypt it again before sending it off.
Answer: Im not sure but can you please have a look into sshuttle https://github.com/apenwarr/sshuttle/  maybe one of the common uses for it can meet your requirement.
Answer: Googling around provided this solution:
http://blog.gidley.co.uk/2009/03/tunnelling-ssh-over-socks-proxy.html
I find it simple and clever, unfortunately it does not address the last requirement...
But using nc could be the answer.


Average values in same month python

Question: I have the following dataframe:
    Date     Water   Milk
2020-01-25    21      58
2020-01-21    25      54
2020-01-14    64      22
2020-02-25    54      58
2020-02-16    21      94
2020-03-28    45      95
2020-03-23    64      95

I want to take the average of the Water and Milk values that have the same month and store the averages in a new dataframe so that it looks like:
Date     Water      Milk
 1      36.6667   44.6667
 2       37.5       76
 3       54.5       95

Where the date column only outputs the month, and the water and milk values are averaged for those months. Is this possible using the python pandas library?
Answer: Use Series.dt.month with aggregate mean:
df['Date'] = pd.to_datetime(df['Date'])
cols = df.columns.difference(['Date'])
df[cols] = df[cols].astype(float)
#alternative
#df[cols] = df[cols].apply(lambda x: pd.to_numeric(x, errors='coerce'))

df1 = df.groupby(df['Date'].dt.month).mean().reset_index()
print (df1)
   Date      Water       Milk
0     1  36.666667  44.666667
1     2  37.500000  76.000000
2     3  54.500000  95.000000

Comment: I'm getting an error "No numeric types to aggregate." Any ideas how to fix this? I think it may stem from how I'm storing my dates in the initial dataframe

Average values in same month python


What does "a testing procedure is valid" mean?

Question: How is "a testing procedure is valid" defined?
For example, in Wikipedia

Fisher's exact test is a statistical significance test used in the
  analysis of contingency tables. Although in practice it is employed
  when sample sizes are small, it is valid for all sample sizes.

Thanks and regards!
Answer: (Moving my comment to an answer so the question doesn't stay unanswered.)
I don't think the intend the word in a statistically technical sense.
I think they simply mean that the procedure works as a formal test at large sample sizes rather than being exclusively a small-sample test.
There's nothing about the way the test is constructed that assumes a particular sample size.
I suppose you could interpret it to mean something formal -- that they're claiming it has (no more than) the nominal type I error rate -- but I don't think they actually meant that.


Test significance of weighted average of multiple regression coefficients from different models

Question: I have 3000 independent time series samples (customers) where I fit a dynamic regression model with ARIMA errors to each sample and estimate regression coefficient of interest (intervention), $B_1{_i}$ from the following model
$Y{_i} = B_0{_i} + X_1{_i}B_1{_i} + .... + e{_i}$
where $Y_i$ is sales per customer $i$.
I used ARIMA to take into account any seasonality and trend and am okay with ARIMA terms capturing other unexplained variance.
I end up with 2 vectors of size 3000; one for the $B_1{_i}$ and another for their standard errors, $SE_1{_i}$. Some coefficients are significant and others are not.
An overall estimate of $B_1$ is needed so I use a weighted average (a weight has been derived based on prior year sales proportion out of the total) to calculate the overall estimate and use wtd.t.test from the weights package in R to test for the significance of the overall estimate.
My questions are

Is it valid to test the significance of the overall estimate using a weighted one-sample t-test?
Or do I need to combine the standard error estimates, $SE_1{_i}$ from all the models and calculate an overall standard error?
And how would i calculate the overall standard error integrating the weights?
Answer: If I understand correctly, you are interested in estimating some overall population effect. Instead of first doing a regression and then trying to do subsequent weighted tests, there is a more efficient way to use your data to achieve your objective.
A mixed-model (a.k.a. hierarchical or mixed-effects model) allows you to estimate your overall population effect while accounting for deviations from the population effect at an individual level using random effects. Random effects (intercepts and slopes) characterise the variation around the population effect. The mixed modelling approach benefits from partial pooling, in which each customer's regression is informed/constrained by the data from other customers; this is described as the individual-level random effects being 'shrunk' towards zero. This might seem unintuitive, but it has good theoretical and empirical support.
Comment: I understand that mixed-models are one way to achieve my objective, but my questions are regarding the an alternative method of modeling each customer separately instead of building one model.
Comment: @MoeSam Why do you want to use a t-test instead?
Comment: that is my question, is there anything invalid about using t-test?
Comment: @MoeSam As I've said here, it is an inefficient use of data and provides results that are likely to be less accurate than mixed models. Isn't that good enough reason to avoid a t-test? If not, you should define what criterion you are using to decide on a method.
Comment: Since each sample is a time series, I am using dynamic regression with ARIMA errors to take into account any seasonality and trend. I am only interested in the effect of the intervention, $B_1$, and would be okay with ARIMA terms capturing other unexplained variance. If I go with a mixed-model, it would further complicate the analysis having to deal with omitted variable bias.
Comment: @MoeSam It would have been useful if you had included these details in your question.
Comment: Just did that now.


Zend Route Regex - matching route with optional parameters AND plus sign

Question: Following THIS question, I managed to create regex for use with Zend_Controller_Route_Regex that would match not only required, but optional parameters.
Having links like:
fooroute,alles,p0000,1
fooroute,alles,p0000
fooroute,alles
fooroute

Starting with:
fooroute,([\+\sa-zA-Z]+),([p0-9]+),(\d+)

I ended up with following regex:
fooroute(?:\,|$)([\+a-zA-Z]+|$)(?:\,|$)([p0-9]+|$)(?:\,|$)([0-9]+|$)

Links listed above work just fine.
The problem is that \+ in first parameter does not seem to be working fine.
Trying to hit following route:
fooroute,This+Is+Test,p0000,1

I end up with Page not found exception.
I built my pattern using regexr.com, and result of my work can be found HERE.
What I'm missing?
UPDATE
Code:
$router->addRoute(
            'www-search-results',
            new Zend_Controller_Router_Route_Regex(
                    'fooroute(?:(?:,([a-z_]+))?(?:,(p\d+))?(?:,(\d+))?)?$',
                    array(
                        'module'=>'www',
                        'controller'=>'index',
                        'action'=>'searchresults',
                        'city' => 'alles',
                        'searchParams' => 'p0000',
                        'page' => 1

                        ),
                    array(
                            '1'=>'city',
                            '2'=>'searchParams',
                            '3'=>'page'

                        )
                    , 'fooroute,%s,%s,%d'
                )
    );

Answer: You don't need to escape + inside the characters set. Neither you need to escape the ,.
Also, using the i (ignorecase) flag will make your pattern smaller as you won't have to use a-zA-Z everytime. Use the following pattern:
^fooroute(?:(?:,([a-z+]+))?(?:,(p\d+))?(?:,(\d+))?)?$

Comment: Thanks! Gonna try it out. BTW which software produces such a beautiful images? ;)
Comment: Unfortunately, this isn't working. While it turns out that `Zend_Controller_Route_Regex` adds `/i` by default, it still won't get the `+` sign.
Comment: @ex3v Can you put the actual code you're using in your question? The website is https://www.debuggex.com
Comment: @ex3v You have `[a-z_]` instead of `[a-z+]` there.
Comment: Sorry, forgot to mention that I tested this with both `_` and `+` (backup strategy) and, strangely, with `_` sign it works, but with `+` it won't... It looks like it's some kind of Zend-related issue


'Undoing' a square root transformation for linear regression

Question: I have a time series that I'd like to perform regression on. It's not linear, or homoscedastic.
After applying a square root transformation to the dependent variable, which made the graph linear.
I applied the sklearn LinearRegression model, and checked the distribution of the residuals, which are normally distributed. Additionally, I checked for homoscedascity, and found that to be the case.
My question is, is it valid for me to just square the values of the regression line, and plot it against the graph of my original values to undo the transformation? Can I do the same with standard error lines?
(Edit: typos)
Comment: If your estimate for the square root is say $\widehat{\sqrt{y_i}}=20$ then you can square this to give an estimate for the value of $\hat y_i =400$, but you need to realise that this transformation makes this estimate biased and the uncertainty asymmetric (so a range for the square root of $10-30$ when squared becomes $100-900$ and $400$ is not in the middle of that)
Comment: @Henry you could probably make the comment an answer.
Answer: From comment:
If your estimate for the square root is say $\widehat{\sqrt{y_i}}=20$ then you can square this to give an estimate for the value of $\hat y_i=400$,
but you need to realise that this transformation makes this estimate biased and the uncertainty asymmetric (so a range for the square root of $10−30$ when squared becomes $100−900$, and $400$ is not in the middle of that)
Comment: If the transformation makes the distribution symmetric, then the reverse-transformed values estimate the median of the distribution on the original scale.  If you want to estimate the mean without assuming a distribution you can use the smearing estimator.
Comment: Thank you, this is helpful!

Is there a way to account for, and offset, the bias, or rather do you just have to be aware that it's there?
Comment: @AGriffen You have to be aware that it is there.  Similar things would happen with taking logarithms or other non-linear transformations
Comment: @FrankHarrell Is that the distribution of the residuals, sorry?
Comment: Correct; distribution of residuals on the transformed scale.
Comment: Yes, there are ways to account for the bias, such as [Duan's smearing estimator](https://stats.stackexchange.com/search?q=Duan%27s+smearing+estimator+score%3A1).


Is there any tool to test the tendency toward/away from stationarity?

Question: I have time series data, like the following:
<code>Time       Value
   1          20
   3          40
   8         -10
   9          30
       ...
</code>
I'm wondering if there is any test if this time series is toward 0 (like "decrescendo" pattern in the traceplot), or away from 0 (like "crescendo" pattern in the traceplot).
Any recommendation for the tool to visualize/summarize this pattern will be appreciated.
Answer: There are many such tests.
Generally, they fall into two categories:

Tests for unit root (I.e. tests of stationarity)
$H_{0}:$ Time series is/are stationary.
$H_{A}:$ Time series has/have unit root (i.e. is non-stationary).
Example: The Hadri-Lagrange multiplier test.
Tests for stationarity (I.e. tests of unit root)
$H_{0}:$ Time series has/have unit root.
$H_{A}:$ Time series is/are stationary.
Example: The Im-Pesaran-Shim test.

There are many versions of these tests, for example, for making such inferences about multiple time series (e.g., a time series for each country, city, etc.). Which tests are appropriate for you depends on your study design (e.g., single or multiple time series, balanced measurements, missing observations, etc.).
I think a wise way to approach making use of such tests is to combine inference from an appropriate version of a test for stationarity, coupled with an appropriate version of a test for unit root:

Reject test for stationarity, and fail to reject test for unit root: conclude data are stationary.
Fail to reject test for stationarity, and reject test for unit root: conclude data have unit root (are non-stationary).
Fail to reject test for stationarity, and fail to reject test for unit root: conclude data are under-powered to make any inference one way or the other.
Reject test for stationarity, and reject test for unit root: think hard about your data! For example with tests like those I mentioned above it may be the case that some time series have unit root, and some time series are stationary. It may also be the case that your data are autoregressive (values have memory of their prior states), but are not fully unit root (i.e. the memory decays eventually, instead of carrying forward infinitely).

For a single time series, the augmented Dickey-Fuller test for stationarity, and the complementary Kwiatkowski–Phillips–Schmidt–Shin test for unit root may be appropriate tools, and these tests are commonly implemented in statistical software (e.g., R, Stata, etc.).
Further, explicit modeling of autoregressive process will allow you to make inferences not only about stationarity vs. unit root, but the middle-ground of autoregressive (memoried) processes, which will be stationary in the long run (possibly very long run), but still present difficulties for valid inference over shorter time scales. For example, using OLS regression or some other estimator, one could model the first difference of a time series ($\Delta y_{t} = y_{t} - y_{t-1}$) using a single lag:
$$\Delta y_{t} = \alpha + \rho y_{t-1} + \zeta_{t}$$
A value of $\rho$ which is undifferentiable from $0$ (equivalence tests can help with this inference) is evidence of stationarity (trend stationarity if $\alpha \ne 0$) over the time scale of your data. A value of $|\rho| \approx 1$ is evidence of unit root. A value of $0 < \rho < 1$ is an autoregressive process which is likely to demand an appropriate time series model (see de Boef and Keele) the closer $\rho$ is to 1 (i.e. the longer its memory gets). (There can even be $|\rho|>1$, implying a runaway process... in my discipline, I don't see these much.) Of course, models with more lags are also possible.
References
De Boef, S. (2001). Modeling equilibrium relationships: Error correction models with strongly autoregressive data. Political Analysis, 9(1):78–94.
De Boef, S. and Keele, L. (2008). Taking time seriously. American Journal of Political Science, 52(1):184–200.
Dickey, D. A. and Fuller, W. A. (1979). Distribution of the estimators for autoregressive time series with a unit root. Journal of the American Statistical Association, 74(366):427–431.
Hadri, K. (2000). Testing for stationarity in heterogeneous panel data. The Econometrics Journal, 3(2):148–161.
Im, K. S., Pesaran, M. H., and Shin, Y. (2003). Testing for unit roots in heterogeneous panels. Journal of Econometrics, 115:53–74.
Kwiatkowski, D., Phillips, P. C., Schmidt, P., and Shin, Y. (1992). Testing the null hypothesis of stationarity against the alternative of a unit root: how sure are we that economic time series have a unit root? Journal of Econometrics, 54(1-3):159–178.
Answer: You can plot the time-series and inspect it visually. A stationary time-series will have constant variance and have a constant expected value (edit to incorporate @Richard Hardy's comment). plot(Value~Time)
You could look at the ACF and PACF plots. They should die down if the series is stationary.
There is also the Augmented Dickey-Fuller Test for being stationary. In R use ?adf.test for more information.
Comment: @Alexis, expected value of zero has nothing to do with stationarity. In a stationary process, the expected value can be any constant.
Comment: @RichardHardy I assumed "expected value of zero" carried an implied "about the mean or mean trend", but you are quite right.
Comment: "A stationary time-series will have constant variance and have an expected value of zero." This is true, but it may take a long time—possibly a much longer time than the period studied—for long-memoried processes which to prove stationary in statistical inference.


Do I need separate rows for each patient for different covariates in a regression model?

Question: patient  outcome_var   covar1 covar2 covar3
1          y                1     0    0
1          y                0     0    1
1          y                1     0    0
2          n                1     0    0
2          n                0     1    0
2          n                0     0    0
Do I need to use the table above as such in a logistic regression or for each patient bring a single observation for each covariate for example, make the above table look like this
patient  outcome_var     covar1 covar2 covar3
1           y               1      0      1
2           n               1      1      0
i.e, to just bring a single observation for each covariate and get a single for each patient with a single 1/0 value for each covariate to use in a logistic regression model.
Thanks
Answer: How you structure your data depends on the type of analysis you're doing and the software you're using. If you are running a standard logistic regression predicting a single outcome at one time point from several covariates, you should have one row per patient. That row should contain all the information about a patient.
The first table would be used if you were taking different measurements at different times for each patient, and at each occasion, one of the covariates could be active. For example, in a sequential trial, you might give one treatment at time 1 and record the response to that treatment, then another treatment at time 2 and record the response to that treatment, etc. Then you would have one row per time point per patient. The key is that you are measuring the outcome multiple times; when you do this, you should have a row for each time the outcome is measured. You would fit a mixed effects logistic regression to this kind of data. You haven't indicated that your data are like this.
Comment: Thanks Noah for the answer. I am doing a standard multiple logistic regression in R. so, I think I should do one patient per line with all the covariates in different columns on a single line. Thats what I have done but I wasn't sure. When I run the model there is quite a big difference in coefficients values for many covariates between the univariate model outcome and multivariate. I read if there is a big difference then it means that you have dependent variables.


Adding action button (icon) on top right corner of the shiny dashboard box

Question: I would require an action button icon on the top right corner of the shiny dashboard box. The below code appends the icons 'refresh' and 'plus' adjacent to the 'Title1'. However, I would require the icons to be placed at the right side end of the header bar (Similar to the positions of minimize, restore and close button in windows application).
library(shiny)
library(shinydashboard)

body <- dashboardBody(
  fluidRow(
    box(
      title = p("Title 1",
                actionButton("titleBtId", "", icon = icon("refresh"),
                             class = "btn-xs", title = "Update"),
                actionButton('titleBtid2', '', icon = icon('plus'),
                             class='btn-xs', title = 'update')
      ),
      width = 4, solidHeader = TRUE, status = "warning",
      uiOutput("boxContentUI2")
    )
  )
)

ui <- dashboardPage(
  dashboardHeader(title = "Row layout"),
  dashboardSidebar(),
  body
)

server = function(input, output, session) {

  output$boxContentUI2 <- renderUI({
    input$titleBtId
    pre(paste(sample(LETTERS,10), collapse = ", "))
  })
}

shinyApp(ui = ui, server = server)

Answer: Add a style declaration with absolute positioning to your action buttons.
library(shiny)
library(shinydashboard)

body <- dashboardBody(
  fluidRow(
    box(
      title = p("Title 1",
                actionButton("titleBtId", "", icon = icon("refresh"),
                             class = "btn-xs", title = "Update", style = "position: absolute; right: 40px"),
                actionButton('titleBtid2', '', icon = icon('plus'),
                             class = 'btn-xs', title = 'update', style = "position: absolute; right: 10px")
      ),
      width = 4, solidHeader = TRUE, status = "warning",
      uiOutput("boxContentUI2")
    )
  )
)

ui <- dashboardPage(
  dashboardHeader(title = "Row layout"),
  dashboardSidebar(),
  body
)

server = function(input, output, session) {

  output$boxContentUI2 <- renderUI({
    input$titleBtId
    pre(paste(sample(LETTERS,10), collapse = ", "))
  })
}

shinyApp(ui = ui, server = server)


How can I add a custom tab to my Facebook pages?

Question: How to add a custom tab to my Facebook page, without the use of static iframes? I have followed the documentation of Facebook regarding the installing of tabs but couldn't find the solutions for my problem.
I have fetched the pages of my users using the manage_permissions but am unable to get the custom tab install to a page.
Answer: Just go to:
http://facebook.com/add.php?api_key=YOUR_APP_ID&pages, you should get a select with all the pages where you are an administrator, so you can install that app's tab to any of them.
You need to have an application created (https://developers.facebook.com/) and properly configured.
You should have the app's Page Tab section configured also.
Another way of installing the tab to a page is going to your app's profile page (http://www.facebook.com/apps/application.php?id=YOUR_APP_ID) and click on Add to My Page.
Comment: I have created an app nd used that method in creating the tab by  configuring but i need to add a custom tab to a particular page by using php code ...
`http://facebook.com/add.php?api_key=YOUR_APP_ID&pages`
when i click and use the above link, it is redirecting to my facebook homepage....how to use this code in php..?
Comment: you can't add a tab to a page programmatically, "http://facebook.com/add.php?api_key=YOUR_APP_ID&pages" is as close as it gets.
Comment: even better, if you have the page id:

http://facebook.com/add.php?pi_key=YOUR_APP_KEY&pages=1&page=YOUR_PAGE_ID
Comment: read http://stackoverflow.com/questions/7162071/add-a-tab-to-facebook-page-programatically, maybe that HTTP-POST-to-the-graph thing could work


Swapping the row data and column data in react aggrid

Question: Using react ag-grid for displaying the data in the table format.
Trying to swap the row and column data in the below but didn't work.
import React, { useState } from 'react';
import { AgGridReact } from 'ag-grid-react';

import 'ag-grid-community/styles/ag-grid.css';
import 'ag-grid-community/styles/ag-theme-alpine.css';

const App = () => {
   const [rowData] = useState([
      {make: "Toyota", model: "Celica", price: 35000},
      {make: "Ford", model: "Mondeo", price: 32000},
      {make: "Porsche", model: "Boxster", price: 72000}
   ]);

const [columnDefs] = useState([
    { field: 'make' },
    { field: 'model' },
    { field: 'price' }
]);

return (

        <AgGridReact
            rowData={rowData}
            columnDefs={columnDefs}>
        </AgGridReact>

);
};

export default App;

Expected in the below table format
Make       Toyota      Ford      Porsche
Model      Celica      Mondeo    Boxster
price      35000       32000     72000

Answer: I think it's a hacky solution but in my opinion, there is no built-in way to transpose data in react-ag-grid. So you can try to transpose data using custom function.
For example:
const transposeData = (data, columns) => {
  const colNames = columns.map((col) => col.field);
  const transposed = colNames.map((colName) => {
    const obj = { name: colName };
    data.forEach((row, index) => {
      obj[index] = row[colName];
    });
    return obj;
  });
  return transposed;
};

const App = () => {
  /* define rowData and columnDefs */

  const transposedData = transposeData(rowData, columnDefs);
  const transposedColumns = [
    { field: "name" },
    ...rowData.map((row, index) => ({ field: index.toString() }))
  ];

  return (

      <AgGridReact
        rowData={transposedData}
        columnDefs={transposedColumns}
        defaultColDef={{ width: 150 }}
        headerHeight={0}
      ></AgGridReact>

  );
};

export default App;

You can see the whole example here: codesandbox.io


Naive Bayes: Continuous and Categorical Predictors

Question: It's my understanding that most types of common classifiers (Support Vector Machine, for example) can take a mixture of categorical and continuous predictors.
However, this doesn't seem to be true for Naive Bayes, since I need to specify the likelihood distribution a priori.
What should I do if I want to run Naive Bayes for a mixture of categorical and continuous predictors?
Thanks!
Answer: You can use any kind of predictor in a naive Bayes classifier, as long as you can specify a conditional probability $p(x|y)$ of the predictor value $x$ given the class $y$. Since naive Bayes assumes predictors are conditionally independent given the class, you can mix-and-match different likelihood models for each predictor according to any prior knowledge you have about  it.
For example, you might know that $p(x|y)$ for some continuous predictor is normally distributed. Simply estimate the mean and variance for this variable under each class in the training set; then use PDF of the Normal distribution to estimate $p(x|y)$ for new unlabeled instances. Similarly, you can use the sufficient statistics and PDF of any other continuous distribution as appropriate.
If some other predictor in the classifier is categorical, that's fine. Simply estimate $p(x|y)$ using a Bernoulli or multinomial event model as you normally would, and multiply the two conditional probabilities together in the final prediction (since they are assumed to be independent anyway).
Side Note: It isn't strictly the case that SVMs and other discriminative linear models take a mixture of categorical and continuous predictors. You can interpret SVMs as only taking continuous predictors, with values in {0,1} for categorical variables as a special case.
Comment: You know what... I spoke too soon. The nb.predict_proba(X) method will return posteriors normalized over all classes. The correct thing would be to multiply the *unnormalized* probabilities. So it looks like the only way to mix predictor types with sklearn is to roll your own subclass of BaseNB that knows how to treat the differepent predictors differently.
Comment: You can certainly do this, and it would be an ensemble classifier, but it still isn't exactly the same prediction as a single "mixed" naive Bayes model, since nb.predict_proba(x1) returns normalized class marginal probabilities. Say we have two classes {0,1}, a continuous variable $x$, and a discrete variable $z$. The "proper" posterior of class 1 for the mixed NB classifier is $p(1|x,z) = \frac{p(1)p(x|1)p(z|1)}{p(1)p(x|1)p(z|1)+p(0)p(x|0)p(z|0)}$. If you expand out the equation for the product of two differen NB classifiers in this case, you'll see that the math is different.
Comment: Ok, I see what's going on.  The problem is that in the Python library sklearn's implementation of Naive Bayes, you have to specify the likelihood distribution as Gaussian, Binomial, etc.  Thanks!
Comment: Ah, yes. You could play a trick by fitting a different classifier for each subset of predictor types (Gaussian, Binomial, etc.) using the different naive Bayes implementations. Then multiply their posterior distributions together and re-normalize to make predictions. Again, since the predictors are assumed to be conditionally independent, this should give you the same answer... though it's a little sloppy from a software engineering perspective. :)
Comment: Ok, let me make sure I understood you properly.  The new predictor, created by the product of their posterior distributions, would just be a single number, right?  Thanks again!
Comment: I think you can just multiply if the probability of each class is equal.  Otherwise, you have to normalize by the probability of the class.  Right?
Comment: But if the probabilities of each class is roughly equal, it shouldn't matter, right?
Comment: Specifically, nb.predict_proba(x1) should give P(C1|x1)...P(CN|x1).  Can't I just then use the following formula to combine posterior probabilities?
P(C1|x1,x2)=P(C1|x1)*P(C1|x2)/P(C1)

(see http://www.mpia-hd.mpg.de/Gaia/publications/probcomb_TN.pdf)
Answer: Another simple approach to handling continuous predictors is to "bin" your continuous variables:
A common example is to split time of day (continuous, numeric) into AM and PM, for instance.
You can potentially capture more information by increasing the # bins (e.g. split 24 hours into 4 6-hour periods); however, this also increases your model's sensitivity to noisy data so you need to be careful.
Based on my experience I'd recommend this approach if you have one/few continuous predictors among many categorical predictors.
Comment: Would you use a Multinomial or Bernoulli Naive Bayes model for your time example?
Answer: The way to do this as sketched above - "Another simple approach to handling continuous predictors is to bin your continuous variables" - is available in a ready-to-use webservice.
Real-valued numeric variables are 'binned' while maximizing the retained discriminative performance with respect to the classifier outcomes to predict. After this preprocessing step, the classifier is built 'on-the-fly', and its generalization ability tested with N-fold cross validation.
You can try this webservice yourself at Insight classifiers.
When you want real insight into how classification takes place in your domain, you need to substitute continuous-valued classifiers such as neural networks and support-vector machines with discrete classifiers.
A multivariate mixture distribution that comprises discrete and continuous predictive variables - any mapping of this to the classification outcomes involves complex probability integrals Egmont-Petersen et al. And in most classification domains, these probability densities are not Gaussian.
So performance-retaining discretization of the predictive variables ensures a distribution-free (non-parametric) classifier, which is also a white-box. This means that you can comprehend the classifier and thereby the underlying domain.


The formula for conditional expectation in terms of joint cdf

Question: We know that covariance can be written as a function of marginals and joint CDFs, namely
$$\newcommand{\cov}{\operatorname{cov}}\newcommand{\d}{\mathrm{d}}\cov(X,Y) = \iint (F_{X,Y}(x,y) - F_X(x)F_Y(y))\,\d x\,\d y$$
How can we rewrite $E(Y|X<a)$ in the same way as function of CDFs?
Answer: $$P[Y<y|X<a]=\frac{F_{X,Y}(a,y)}{F_X(a)}$$
So,
$$E[Y|X<a]=\int{1-\frac{F_{X,Y}(a,y)}{F_X(a)}dy}=\int{\frac{F_X(a)-F_{X,Y}(a,y)}{F_X(a)}dy}$$
Comment: Could you please elaborate on the requirement "Y | X < a needs to have positive support"?  What does it mean?  Can you provide example and counterexamples of applying the formula where this coition holds / does not hold?  Thank you
Comment: @Confounded It means not possible for Y to be less than or equal to 0;  that is $P[Y>0|X<a]=1$.
Comment: We are assuming that $Y$ has positive support here, right?
Comment: Yes, in the first step.
Comment: In the first step, the conditional distribution of Y given X<a needs to have positive support. If the distribution of Y given X<a has support that is bounded below by -M where M is a positive number, then it can still work with some easy modification since Y+M now has positive support and $E[Y|X<a]=-M+E[Y+M|X<a]$.


How to remove dynamically created textbox and labels in C# Windows Forms?

Question: I want to know how can I remove dynamically created textboxes and labels, I want to remove all dynamically created textboxes and labels from this event cmbMethodActions_SelectedIndexChanged. Because all of the textboxes and labels is based from database. Every the event changes, the textboxes and labels will change too.
As of now here is my code:
private void cmbMethodActions_SelectedIndexChanged(object sender, EventArgs e)
    {
        GetActionFields(int.Parse(cmbMethodActions.SelectedValue.ToString()));
    }

    private void GetActionFields(int i)
    {
        MySqlCommand cmd = new MySqlCommand("call GetActionField("+i+")", cn);
        MySqlDataAdapter adapter = new MySqlDataAdapter(cmd);
        DataTable dt = new DataTable();
        adapter.Fill(dt);

        for (int x = 0; x < dt.Rows.Count; x++)
        {
            TextBox txtfields = new TextBox();
            txtfields.Name = dt.Rows[x]["field_name"].ToString();
            txtfields.Width = 175;
            flowLayoutPanelText.Controls.Add(txtfields);

            Label txtlbl = new Label();
            txtlbl.Name = dt.Rows[x]["field_name"].ToString();
            txtlbl.Text = txtlbl.Name;
            flowLayoutPanelLabel.Controls.Add(txtlbl);
        }
    }

Answer: flowLayoutPanelText.Controls.Clear();
flowLayoutPanelLabel.Controls.Clear();


Restore Angular 4 project from dist folder

Question: So I accidentally deleted my Angular 4 project but I still have dist folder with all bundles and code packed. So is there any way I can restore all the project layout so I can continue coding in my editor or is it gone forever?
Thanks.
Comment: Use any software for deleted file recovery instead.
Comment: ctrl + z? recycle bin? My thoughts are that this will not be possible. It would make it too easy to take other people's angular websites.
Comment: I think it's hard luck
Comment: Unfortunately I deleted it somehow with GIT, so I don't have them in my bin. It was few days ago, I know I can the system restore, but that is not a option.
Answer: No.
If you do not have project files, you cannot reverse-engineer Angular application to get source back. You can take a peek at compiled code to get some business logic if it's not too tied with Angular's view engine (which it shouldn't be if you've been developing with best practices) and try restoring that, though.
It's like trying to go back to 2 + 3 from having only 5. You have no idea if it was 3 + 2, 1 + 4, 4 + 1. Angular compiles your application, and there's no 1-1 mapping. Especially when you consider that you've also been using TypeScript. Angular's build output it two compilers and a bundler away from original. Irreversible.
Here's a more detailed breakdown of how compilation looks like.

Your Angular code written in TypeScript is being compiled to a different TypeScript code by Angular compiler. This process in theory might or might not be reversible, depending on how Angular's compiler works (I'm in no position to speak about that).
Then TypeScript is compiled down to JavaScript. This is irreversible
A bundler (such as Webpack, Rollup, SystemJS or Parcel) messes with your code, including who-knows how many plugins mess with your code. Most of them are irreversible. Actual bundling might be reversible depending on implementation.
Resulting bundles (or a single one) are minified. Irreversibility of this also depends on how heavy the minification was. For example, if you're using Google's Closure Compiler with advanced settings, getting back the original code is irreversible by definition.
Comment: Yeah, that is what I was thinking, Thanks for the answer!
Comment: What @Eliseo is suggesting is the easiest step and would not lead you anywhere. The chain is as follows: (1) source code (2) output of Angular compiler (3) output of TypeScript compiler (4) output of bundler such as webpack (5) output of a minifier. Deobfuscation gets you from 5 back to 4, but that's it.
Answer: Deploy the app code in the Dist folder, and then use Google Chrome Developer tools (F12). Under debugger tab, look under webpack -> src It will show all typescript files. you can copy and past the code provided.


Why is there no mixer functionality that allows a track to be aux send/return-ed while preserving the track's pan setting?

Question: (This is ramble-y, please don't ream me in comments.)
For my own purposes, when using a mixer, it would be nice if I could send a left or right panned track out the aux send and then return it to the aux return with the returning signal being panned left or right identical to the original track.  When I first started learning to use mixers I assumed aux sends and returns would behave like this.
I now understand the fact that aux sends don't necessarily send to aux returns and aux returns don't necessarily receive from aux sends, and therefore neither has any reason to hold any of the settings of any given track that might be going from or returning to them.
That said, the use case in the first paragraph doesn't seem to me so unreasonable that a mixer designer wouldn't have thought of it, and I don't imagine I'm the only person who either made the false assumption that sends and returns would work like that or would find it convenient if they could work like that.
Given these two presumptions (which I realize could both be false!), I would think that at least some mixer makers would have implemented this functionality.  In my experience with mixers though (which I acknowledge is far from exhaustive and therefore potentially misinformed!), there is no mixer maker that does.
Would this be an unreasonable or insufficiently practical feature?  If so, what are alternate use cases that make it such?  If not, is there some concrete reason why it is never seen?
Comment: Your effects need to be true stereo in/stereo out. It's pointless if they're mono in. It's de rigueur on DAWs these days where many FX are true stereo, so sends & returns are too. On hardware, it costs more, so it won't be on budget gear.
Comment: There is no way on earth you can send to a mono effect & have it know where in the stereo field it ought to be placed. For that you need a true stereo-capable effect & stereo send/returns. It's like running warm water through a pipe, then trying to separate it into hot & cold again… or trying to unbake a cake.
Comment: You can do that with a mono send/return. Just pan the return the same as the track you're sending from. You can't expect a mono effect to know where it's supposed to be in the stereo field.
Comment: You 'just' need stereo auxes, or sacrifice 2 aux busses to each send/return. idk at what price point that starts to appear, it's been a long time since I used a hardware mixer.
Comment: Do you mean both a stereo aux send and return?
Comment: Ah, so there are mixers for which you can pan the return.  I didn't know this.
Comment: If you can't pan your return, bring it up a regular channel instead & pan that.
Comment: They're all just channels, some just have more control. Everything you do 'costs' a channel. The more you have, the freer you can be with how you use them. Simple logistics.
Comment: You can't do it with a single mono effect, whatever else you have to play with. On a DAW, you'd just use multiple effects, probably as inserts rather than use up a dedicated return channel.
Comment: Of course, unless you bring it back up regular channels.
Comment: I can't wrap my head around how that would actually work with an effects processor in between, seems like pan info would still get lost somewhere...  In any case, stereo sends sounds like an esoteric feature; are you suggesting then that the feature I'm looking for is equally esoteric?
Comment: To give a concrete example, I would like to pan a track 50% left, send it out to a guitar pedal (mono), return it and have the wet signal be panned 50% left.  Are you saying that that is not a realistic use case?  (That's not a rebuttal, just a question.)
Comment: I know that method but in addition to using up a track, for someone who doesn't understand mixers and signal flow it doesn't seem intuitive that that should need to be done.
Comment: Back to the mono effect, even if you can pan the return, it seems it would be convenient to be able to somehow apply a mono effect to multiple tracks with various pan settings and have the wet signal retain those original pan settings.  That can be done in a DAW, it would be nice for me personally if I could do that with a hardware mixer.
Comment: When I did 'proper' studio work, 48 channels, 8 busses, 6 send & returns were just about enough for most things. Sometimes you'd have to pull in a sub-mixer from one of the other rooms. Now I work in software 120 channels is pretty standard fayre. Unlimited channels & busses, limited but very generous sends/inserts & unlimited returns.
Comment: I understand that I can't do it,  I just don't understand why the functionality doesn't exist, because it seems so intuitive to me that it also seems like there must be a reason.  And I acknowledge that that presumption could be wildly off base.
Comment: I understand.  For my imaginary functionality I'm imagining a different signal flow existing.  I'm trying to assert that the idea itself doesn't seem that unreasonable.  But I guess the fact that it doesn't exist is evidence that it is unreasonable.  Thanks for your patience, I won't unnecessarily wear it any further.
Comment: The more audio engineering you do, the more you'll want to at least dabble in some electrical engineering/electronics. It will help make sense of 95% of the tools of audio engineering. A mid-level understanding of electronics would make it easy to see why that functionality isn't available. As described, it's electronically impossible.
Comment: Could you expand the first paragraph to clarify what exactly are you trying to achieve? Does a mixer with stereo sends solve your issue? If not, why?
Answer: I think you've misunderstood what an FX send is for. Sends are for bus effects, i.e. effects shared by multiple different tracks. Of course you can use it with just a single track, but in general what comes in on the return will be a mix of signals that originated from multiple tracks. The only way that a send loop can preserve stereo-field information is if the entire thing is in stereo. Now, that is actually perfectly possible, indeed standard in DAWs. But even if, it's kind of missing the point if the return has the same stereo position as the sending track, because the whole point of having an effect in the send loop is that it should merge together the signals.
This is most evident in reverb, the by far most common send effect: reverb should facilitate that the different instruments sound like they're actually in the same room. It is in principle possible to apply mono reverb to each channel individually and then pan them together, but this sounds completely different – the reverbs won't sound roomy anymore but will instead be perceived as each instrument having a "wetter", but still isolate sound.
If you want to apply an effect to a single channel and have it "follow the position" of that channel, then you should insert the effect into the channel itself, not use any send. In that case, the effect is applied before the pan control and is thus, indeed, panned to exactly the same position as the dry component on this channel.
Answer:
Would this be an unreasonable or insufficiently practical feature?

I don't think so. I would think of it as a stereo, post-pan aux send. The pan control is effectively where the mono channel gets 'turned into' stereo - a certain amount sent to L, and a certain amount to R. If you had a stereo aux send after this point, you could effectively send the panned signal to the effect.
This could be useful too, musically, in some cases. If the effect you are sending to could deal with the panned signal in a stereo way - e.g. a delay effect that maintained stereo separation (i.e. that was effectively two mono delays with the same settings), it could allow each instrument and its delays to appear in the same place in the stereo image. The same would be true of other linear-response effects, but it wouldn't be true of e.g. distortion based effects, because the way distortion works means that the channels would interfere with each other.

If not, is there some concrete reason why it is never seen?

It costs extra, and there are a limited number of cases where it is useful. As leftroundabout says, reverb is the most common send effect, and maintaining complete separation between L and R doesn't make for a convincing reverb effect - you want to hear the sound bouncing around the soundstage. Often, stereo reverbs sum to mono before the actual reverb processing - they will be stereo in that the dry signal separation will be maintained, and in the characteristics of the wet signal, but not in any way that makes a panned-left input signal give a different wet output than a panned-right one. You could just about imagine modern technology being able to create a reverb that did something sensible with the pan position, but it would have to be very clever and effectively 'unmix' the sent signals before processing (remember that sends are usually used to share the same effect for a number of channels).
People who really want to achieve what you can achieve with a panned send will find another way to do it in the studio - e.g. use multiple insert effects, or the same insert effect applied in turn to different tracks in the recording process.
But your idea isn't a crazy one at all, and it would actually make a lot of sense with certain types of effects.
Answer: We're straying beyond the original remit of the question, so to cover that first.
It is physically impossible to send a mono effect to the same position as the original panned mono instrument. There is no correlation between the two. As the return of your effect pedal is mono, then you can place that in the stereo field manually, but all sounds sent through it must, of necessity, be panned to the same location.
You get around this, especially on a modern DAW or large studio desk, by having true stereo sends to true stereo effects, which can then return signal in the same part of the stereo field as the send.
This isn't always the case. Many effects are 'fake' stereo - mono in, stereo out. Others are true stereo right through.
I disagree with some of the other answers here in as much as if I send several instruments to a given room reverb I want the returns to simulate each of those instruments being in their own place in the room & not just generically splashing the whole lot with generic room sound, placed centrally. This way, each sound retains a sense of placement in the final mix, rather than what I would consider a very 80s way of the reverb being a complete stereo wash that ignores that placement.
Comment: I didn't mean reverb should necessarily act _the same_ on every instrument, but it should have the same general character. Adjusting pre-delay, initial panning and early reflections for each channel separately sure makes a lot of sense, but if the general length, character and stereo pos is significantly different for the different instruments then it just sounds weird. (In fact, to me, the most steretypical "80s wash" is what you get from that completely unnatural gated reverb on the snare.) And in a real room, the later reflections are indeed never from the position of the instrument itself.
Answer: There are.  They are called "inserts" and are per track since you could not otherwise preserve the pans of different channels independently.  If you have effects where you are ok with FX-ing a stereo mix, you can use them with two FX channels: you have to establish your pan manually, but then pan controls are not intended for changing all the time like faders may be, and you often have some post-fader (or switchable post-fader) FX buses where the channel volume is being tracked.
Or you just add the effect on the main out: if you have a power mixer, it tends to have separate inserts for that so that you can add, say, extensive equalisation.  Without a power mixer, you just add your whole-mix effects at the end anyway.
Comment: If you read through the myriad comments, you will find the OP's effect is a mono guitar pedal; very much unsuited to use as an insert effect, especially on a stereo bus.
Comment: @Tetsujin a guitar pedal is almost certainly _more_ suitable for use as an insert effect, than send effect. Mono of course, but in the analogue realm most channels are mono.
Comment: Assuming it's designed to run at the same dBV as the insert, which is unlikely.
Answer: I'll assume that this comment of yours is the actual question:

To give a concrete example, I would like to pan a track 50% left, send
it out to a guitar pedal (mono), return it and have the wet signal be
panned 50% left.

You can do this with an insert effect. (Let's not care about the effect's input being designed guitars now.) Does your mixer have channel inserts? That would be mono-in, mono-out though. For real stereo, you'll need a stereo-in, stereo-out, true stereo effect device.
With an effect used in an aux send configuration having the return panned differently for every channel becomes complicated and its usefulness is not clear to me. If you send channel A and channel B to the same mono-input effect, A and B sends are mixed together as A+B and then sent to the effect input, and the effect's output (wet signal) will have both A and B baked in the same signal. If you pan channel A hard left and channel B hard right, that effect return will still be from A+B. If you pan the effect return left for channel A, and right for channel B, then it will be just effect(A+B) on the left and exactly the same effect(A+B) on the right as well. The same thing on the left and on the right! Why would you want that?
If you absolutely want, you could split the effect return to a number separate channels and pan/balance each of them differently. But I still fail to understand what sense it would make to have the exact same thing going to a bunch of channels that are just panned differently.
Edit. Maybe it could make sense to have e.g. reverb appear in the same spot in the stereo field. But trying to simulate that with a mono-input reverb feels very difficult, and it would require lots of components. And even with a complicated multiple-return-channels contraption, the reverb will still have all channels baked in the same signal. What IS often needed however, is having the reverb send post-fader, so that the amount of reverb follows the amount of dry signal.


SELECT calculando a média de nota em MySQL

Question: Em um sistema de treinamentos tenho duas tabelas:
TABELA nota_locais:
id
id_local
nota

TABELA locais:
id
nome

A tabela nota_locais armazena as notas que cada usuário da para o local onde ocorreu o treinamento. Quando vou listar todos os locais de treinamentos, dou um select na tabela locais, mas queria além de listar o local, dizer qual a média de nota que esse local tem, ou seja, dar um select também na tabela nota_locais e já calculando a média, como fazer?
Comment: `SELECT l.nome, SUM(n.nota) / COUNT(n.nota) FROM locais as l INNER JOIN nota_locais as n ON n.id_local = locais.id` funciona? Se sim posto uma resposta explicando
Answer: Faça a seguinte operação de junção das chaves e aplicando as funções AVG e GROUP BY:
SELECT
   nome,
   AVG(nota) AS 'media'
FROM locais L
INNER JOIN notas_locais NL ON
L.id = NL.id_local
GROUP BY
   id,
   nome

Answer: Execute a query abaixo
select l.id, l.nome, AVG(nl.nota)
from nota_locais nl
inner join locais l  on  l.id = nl.id_local
group by l.id, l.nome

Comment: Valeu, só não dei a sua como certa pq responderam antes.
Comment: a resposta anterior não responde a sua pergunta, pois não está utilizando o group by
Comment: Verdade, testei e realmente não agrupou


"Output" com caracteres especiais

Question: Estou utilizando o compilador GCC versão 5.3 para o seguinte programa, redigido no Aptana:
#include <stdio.h>

void main(void)
{
    int positivo = 32767;
    int negativo = -32768;

    printf("%d +1 é %d\n", positivo, positivo+1);
    printf("%d -1 é %d\n", negativo, negativo-1);
}

Com a seguinte saída:

Já no bloco de texto, com a seguinte:

Como devo proceder para que os caracteres sejam exibido corretamente ?
Comment: Provávelmente problemas de compatibilidade de "encodings" dos terminal, editores de texto... (em que ambientes estás?)
Comment: Obrigado pela atenção, o que quer dizer com "ambientes" ? @JJoao
Comment: Em windows, linha de comando, podes definir qual o charset usado (exemplo windows-CP1252, unicode-utf8, etc) nos editores de texto podes selecionar o carser/encoding (exemplo ansi, etc). Em linux e mac as configurações têm valores iniciais mais razoáveis mas tambem são configuráveis. Sugestão ir comutanto tudo para o mesmo charset/encoding -- talvez unicode-uft8.
Answer: Provavelmente precisa usar um setlocale() para permitir o uso de caracteres acentuados.
#include <stdio.h>
#include <locale.h>

int main(void) {
    int positivo = 32767;
    int negativo = -32768;
    setlocale(LC_ALL,"Portuguese");
    printf("%d +1 é %d\n", positivo, positivo + 1);
    printf("%d -1 é %d\n", negativo, negativo - 1);
}

Veja funcionando no ideone. E no repl.it. Também coloquei no GitHub para referência futura.
Mas não é vantagem alguma porque eles já funcionavam sem o setlocale(). Não tive como testar nas suas condições para reproduzir o problema.
Comment: Essa solução funciona perfeitamente no bloco de notas, já no Aptana ele trocou por outro, também especial... E também notei, que trocastes `void` por `main` e ainda especificou um tipo, pode dizer por que ?
Comment: Provavelmente precisa configurar o editor de texto para gravar com outra codificação, um deles está gravando errado, mas aí já é outro problema.
Comment: E qual o motivo das trocas que fez ?
Comment: Acertar a codificação de caracteres, só que você tem que gravar todos seus fontes com a mesma, senão vira loucura, cada vez que salvar o fonte em um editor diferente, tem que mudar o código para lidar com isto, o que é um completo absurdo.
Comment: @bigown, parecia-me que se trata essencialmente de uma questão de encoding e da sua compatibilização entre editor e terminal.

"Output" com caracteres especiais


Exchange 2016 migration - MAPI/CDO tracing

Question: Microsoft has stated that Exchange MAPI over CDO will no longer be supported by Exchange 2016.  We are working on migrating our 2010 environment to either 2013 or 2016, and this is our primary concern.  We have numerous legacy applications that have simply been left to run, and many we have no idea how they connect to Exchange.
Is there a way that I can identify them in logs, or an application, or anything on the Exchange side?  I can't just go to all the applications to identify them (we have over 17k documented applications in our environment)...
Comment: I would answer the second part of the question - Yes, you can turn on logging for every Exchange protocol if desired.
http://blogs.technet.com/b/exchange/archive/2015/04/20/a-better-way-to-collect-logs-from-your-exchange-servers.aspx
Answer: As MAPI use the RPC's port, it would be easier to validate in your inventory tool the remote computers to check if Collaboration Data Objects, version 1.2.1 is installed or not. That is a pre-req to have your CDO working. I assume you have Office 2007 or later deployed. As any program that use CDO would need it.
Beginning in Exchange Server 2007 and Outlook 2007, CDO 1.2.1 will no longer be provided as a part of the install of the product. As a result, there is functionality missing that many applications depend upon. CDO 1.2.1 is a package providing access to Outlook-compatible objects through a COM-based API.
Check that chart for CDO version bundle with what; https://support.microsoft.com/en-ca/kb/171440


Let $f \in C^1(\mathbb{R})$ be such that both $f$ and $f'$ belong to $L^2(\mathbb{R})$. Show that $\hat{f} \in L^1(\mathbb{R})$

Question: The exercise is the following:

Let $f \in C^1(\mathbb{R})$ be such that both $f$ and $f'$ belong to $L^2(\mathbb{R})$. Show that $\hat{f} \in L^1(\mathbb{R})$.

What I want to prove is that $\int_\mathbb{R} |\hat{f}(\xi)|d\xi  < +\infty$ but I do not know how to proceed; I have started saying that $\int_\mathbb{R} |\hat{f}(\xi)|d\xi = \int_\mathbb{R} | \int_\mathbb{R} f(x)e^{-2\pi i \xi x} dx|d\xi $ and then the idea was to use the properties of Fourier transform and of its derivatives such as:

$f'(x) \hat{\to} 2\pi i \xi\hat{f}(\xi)$
$-2\pi i x f(x) \hat{\to}\hat{f'}(\xi) $

but these properties require $f$ to be in the Schwartz space, that is not our case.
Any suggestion is appreciated. Thanks in advance for the help.
Comment: Both $f$ and $xf$ are $L^2$, then use Cauchy Schwarz
Comment: The formula for the Fourier transform of $f'$ is wrong. The factor $\xi$ is missing.
Comment: @RyszardSzwarc thanks, corrected!
Comment: @reuns: why is $x\mapsto xf(x)\in L^2$? If $f$ had derivative in the sense of $L_2$ ($\frac{\tau_hf-f}{h}$ converges in $L_2$, then that would be the case. It might be the the hypothesis in the OP imply that $f'$ (in the usual sense) is also a derivative in the $L_2$ sense, but this is not clear.
Comment: @OliverDíaz $f'\in C(\mathbb{R}) \cap L^2,$ so its Fourier transform belongs to $L^2.$
Comment: I just misread, $\hat{f}$ and $\xi \hat{f}$ are in $L^2$
Comment: @OliverDíaz I understand that $f'$ is the usual derivative. By assumptions it is continuous and square integrable.
Comment: @OliverDíaz The link refers to another question. Here we need to show that $ \hat{f}$ is integrable. As suggested by reuns we can make use of the Cauchy-Schwarz inequality applied to the product  $$\hat{f}(\xi)=(1+\xi^2)^{-1/2} \cdot (1+\xi^2)^{1/2}  \hat{f}(\xi)$$
Comment: @RyszardSzwarc: from the link (or rather the answer by user "Desintegration by Parts" it follows that $f^2$ (and this $f$) is in $C_0$ and that $constant\cdot s\hat{f}(s)=\widehat{f'}(s)\in L_2$. In particular this also shows that $f$ is differentiable in the sense of $L_2$ ($f'$ being the $L_2$-derivative too). Maybe you want to write a solution to the OP.
Comment: @OliverDíaz This a standard exercise and I am afraid this question has been asked many times. See the answer formula (1) https://math.stackexchange.com/questions/1511897/does-the-fourier-transform-of-a-smooth-l2-function-decay-rapidly-at-infinity
Comment: @OliverDíaz I do not mind if you post an answer. By the way how do you attach a link in the comments section  ? I usually copy the entire link, but that's ugly.
Comment: @RyszardSzwarc: I am writing one now. to attach a link, enclose the word where the link will be anchored between square brackets, follows by the link enclosed by regular brackets, that is [Here](the link).
Comment: @OliverDíaz Thanks a lot.
Answer: From the exchanges among @reuns, @RyszardSzwarc and myself here is one possible solution:
The assumptions of the problem imply that

$f^2$ (and thus $f$) is in $C_0$. See for instance this posting.
$2\pi i s\hat{f}(s)=\widehat{f'}(s)\in L_2$. See for example this posting.

The conclusion follows as indicated by reuns and Ryszard:
$$\hat{f}(s)=(1+s^2)^{-1/2}(1+s^2)^{1/2}\hat{f}(s)$$
and so,
$$\int|\hat{f}|\leq\Big(\int(1+s^2)^{-1}\,ds\Big)^{1/2}\Big(\int(1+s^2)|\hat{f}(s)|^2\,ds\Big)^{1/2}<\infty$$
Comment: In formula $2.$ the first occurence of $s$ should be removed.
Comment: @RyszardSzwarc: I just removed the additional $s$. Now it has the correct form.
Answer: One observation: The assumption $f\in\mathcal{C}^1$ can be weakened to $f$ is differentiable in $\mathbb{R}$. To see this, define  $h=f^2$. Then $h\in L_1$ and $h'=2ff'\in L_1$. It follows that for any numbers $a<b$
\begin{align}
h(b)-h(a)=\int^b_a h'(t)\,dt
\end{align}
(See theorem 7.21 in Rudin, W., Real and Complex Analysis, Third edition, page 149, which does not make reference to absolute continuity). Then the limits $A=\lim_{x\rightarrow\infty}h(x)$ and $B=\lim_{x\rightarrow-\infty}h(x)$ exits. The integrbility of $h=f^2$ implies that $A=B=0$. Hence,  $f\in\mathcal{C}_0(\mathbb{R})$.
An application of Plancherel's theorem and Lebesgue integration by parts gives
$$\widehat{f'}(t)=\lim_{R\rightarrow\infty}\int^R_{-R}e^{-2\pi x t}f'(x)\,dx=\lim_{R\rightarrow\infty}2\pi i t\int^R_{-R}e^{-2\pi i tx}f(x)\,dx=2\pi it\widehat{f}(t)$$
The convergence on the  is in $L_2$, and pointwise along a subsequence $R_k\rightarrow\infty$. Therefore
$2\pi i t\widehat{f}(t)=\widehat{f'}(t)\in L_2$. The rest is as indicated in the Community wiki answer


Pyspark in Docker based on Hortonworks 2.6.1 is throwing error with EnableHiveSupport()

Question: I am trying to build an edge-node using docker with HDP2.6.1. Everything is available and running except Spark Support. I was able to install and run pyspark  but only when I comment enableHiveSupport(). I have copied over the hive-site.xml to /etc/spark2/conf as well from ambari and all the spark confs are matching with the cluster settings. But still get this error:
17/10/27 02:35:57 WARN conf.HiveConf: HiveConf of name hive.groupby.position.alias does not exist
17/10/27 02:35:57 WARN conf.HiveConf: HiveConf of name hive.mv.files.thread does not exist
Traceback (most recent call last):
  File "/usr/hdp/current/spark2-client/python/pyspark/shell.py", line 43, in <module>
    spark = SparkSession.builder\
  File "/usr/hdp/current/spark2-client/python/pyspark/sql/session.py", line 187, in getOrCreate
    session._jsparkSession.sessionState().conf().setConfString(key, value)
  File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/usr/hdp/current/spark2-client/python/pyspark/sql/utils.py", line 79, in deco
    raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':"
>>> spark.createDataFrame([(1,'a'), (2,'b')], ['id', 'nm'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'spark' is not defined

I have tried to search this error, but all the results that I get are possible windows errors related to permissions and hive-site.xml missing. But i am building it on centos:7.3.1611. And installing the following:
RUN wget http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/18.104.22.168/hdp.repo
RUN cp hdp.repo /etc/yum.repos.d
RUN yum -y install hadoop sqoop spark2_2_6_1_0_129-master spark2_2_6_1_0_129-python hive-hcatalog

Answer: So the solution to the above problem is that the hive-site.xml needs to only contain the property for hive.metastore.uris and NOTHING ELSE. (Reference: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_spark-component-guide/content/spark-config-hive.html). Once you take the other properties out, it works like a charm!


How to build up a C code for avr atmega32 microcontroller to capture light intensity through photodiode and transmit data through serial communication

Question: I am planning to construct a spectrometer. My aim is to rotate the grating and capture light intensity using a 3pin photodiode(S9055 series) and to plot a graph of Intensity Vs rotation angle using python. To construct the system I use avr ATMega32a microcontroller and I connect a MG995 servo motor to rotate the grating.
The proposed flow of the program as follows:

I use a white light source and a grating, and the photodiode captures the light comes diffracting through the grating.
I connect the MG995 motor to the microcontroller through OC1A pin and rotate it for small angles giving a delay.
I connect the photodiode through an op amp to the ADC0 pin (similar to the transimpedance amplifier) and transmit the data to the PuTTY terminal.
gather the data to a dataframe and plot the graph.

I am already okay with the rotating of the motor however there seems an error in the ADC part. The following is my code to capture light and convert the values.
<code>#define F_CPU 8000000UL
#include <avr/io.h>
#include <util/delay.h>
#include <stdlib.h>
#include <stdio.h>

//#define USART_BAUDRATE 9600
#define BAUD_PRESCALE (((F_CPU / (USART_BAUDRATE * 16UL))) - 1)

void UART_init(long USART_BAUDRATE)
{
    UCSRB |= (1 << RXEN) | (1 << TXEN);   /* Turn on the transmission and reception */
    UCSRC |= (1 << URSEL) | (1 << UCSZ0) | (1 << UCSZ1); /* Use 8-bit character sizes */

    UBRRL = BAUD_PRESCALE;                  /* Load lower 8-bits of the baud rate value */
    UBRRH = (BAUD_PRESCALE >> 8);           /*Load upper 8-bits*/
}

void UART_TxChar(char ch)
{
    while (! (UCSRA & (1<<UDRE))); /*Wait for empty transmit buffer*/
    UDR = ch ;
}

void UART_SendString(char *str)
{
    unsigned char j=0;

    while (str[j]!=0)   /*send string up to null */
    {
        UART_TxChar(str[j]);
        j++;
    }
}

void adc_init() // Initialize ADC
{
    ADMUX = (1<<REFS0); // AVCC as reference voltage
    ADCSRA = (1<<ADEN) | (1<<ADPS2) | (1<<ADPS1) | (1<<ADPS0); // Enable ADC and set prescaler to 128
}

// Function to read the ADC value
// Read ADC value from a channel
uint16_t adc_read(uint8_t channel)
{
    channel &= 0x07; // Select channel from 0 to 7
    ADMUX = (ADMUX & 0xF8) | channel; // Clear the lower 3 bits and set the channel
    ADCSRA |= (1<<ADSC); // Start conversion
    while(ADCSRA & (1<<ADSC)); // Wait for conversion to complete
    return ADC; // Return the ADC value
}

int main()
{
    uint16_t adc_value; // Variable to store the ADC value
    //DDRB = 0xFF; // Set PORTB as output for LED display
    adc_init(); // Initialize ADC

    char c="5";
    UART_init(9600);

    UART_SendString("\nInitializing....\n ");
    while(1)
    {
         adc_value = adc_read(0); // Read ADC value from channel 0 (connected to photodiode)

        // Print the ADC value to the Putty terminal
        printf("ADC value: %d\n", adc_value);
        UART_SendString(c );
        UART_SendString(adc_value );

        // Wait for a millisecond
        _delay_ms(1000);

    }
    return 0;
}
}
</code>
The output shows in the PuTTy terminal as:
click to view the image
I got the snip after I removed the serial communication cable.
So as you see, this cannot be definitely the output so can someone help me to figure out where I have gone wrong?
Comment: If you want people to help, always reduce your code sample to the smallest amount of code that still gives you a problem. For one thing, `char c = "5"` is incorrect; did you mean `char c[] = "5"`?
Answer: May be it would help
<code>int main()
{
    uint16_t adc_value; // Variable to store the ADC value
    adc_init(); // Initialize ADC

    char c[]="5";
    UART_init(9600);

    UART_SendString("\nInitializing....\n ");
    while(1)
    {
         adc_value = adc_read(0); // Read ADC value from channel 0 (connected to photodiode)

        // Print the ADC value to the Putty terminal
        char buf[20];

        sprintf(buf,"ADC value: %d\n", adc_value);
        UART_SendString(c );
        UART_SendString(buf);

        // Wait for a millisecond
        _delay_ms(1000);

    }
    return 0;
}
</code>


Sinusoidal signal with variable instantaneous frequency

Question: Below is a simple Matlab code that generates a sinusoidal signal with user specified no of cycles. How do I change this such that each cycle has a different instantaneous frequency? For example, if I have 10 cycles, then there should be 10 random frequencies in total where each cycle has a different frequency.
f1 = 120;
fs = f1/60; % sampling frequency depends on f1
no_of_cycles = 10;
f = no_of_cycles/60; % Frequency of sinusoid
duration = 1; % duration of the sinusoidal signal in minutes
A = 1;
t = 1/fs:1/fs:duration*no_of_cycles*1/f; % time index
y1 = A*sin(2*pi*f*t); % simulated sinusoidal signal

Comment: How should the transition be? One of the properties of the sinus function is that it is smooth (you can differentiate it an infinite number of times), changing the frequency instantaneously removes this property, it is actually not differentiable at the point where it changes.
Comment: Thank you. That should not be a problem for my current application because my analysis uses one cycle to generate features.
Answer: Say we have a vector of frequencies, e.g.
frequencies = [1,2,3,4,5];

then we want to make length(frequencies) plots on top of each other, each with a different frequency. In order to make plots on top of each other you can use the command hold on after the plot command.
Additionally we have to keep track of the time, as I suppose you want the next sine curve to continue, where the previous stopped. For this I use variables curTime and endTime to denote the time where the signal starts and ends. Now we have something like:
frequencies = [1,2,3,4,5];
curTime = 0;
for i = 1:length(frequencies)
    endTime = curTime + 2*pi/frequencies(i);
    t = curTime:0.1:endTime;
    plot(t,sin(frequencies(i)*t))
    curTime = endTime;
    hold on
end

However, as we chance the frequency then the y-value of the sine curve which ends is not the same of the one which starts. As I furhter suppose you want it to be continuous, we need to join them, by shifting the "next" sine curve, supposedly by changing $s$ in $sin(ft + s)$. As you are interested in whole cycles we can just chose $s$ such that$ft + s = 0$. Then we end out with:
frequencies = [1,2,3,4,5];
curTime = 0;
h = 0.01;
for i = 1:length(frequencies)
    endTime = curTime + 2*pi/frequencies(i);
    t = curTime:h:endTime;
    shift = frequencies(i)*t(1);
    plot(t,sin(frequencies(i)*t-shift))
    curTime = endTime;
    hold on
end

Comment: Thank you. Can you please let me know how i can incorporate this concept in the code i have posted?


Pass video file

Question: I create a REST server with Drupal 7, set everything and it work correctly, node created well, but I want to fill File field with REST.
my client side code is the following.
(function($){
  function getCsrfToken(callback) {
  jQuery.get(('http://example.com//services/session/token'))
    .done(function (data) {
      var csrfToken = data;
      callback(csrfToken);
    });
}

function postNode(csrfToken, node) {
  jQuery.ajax({
    url: 'http://example.com/rest/node',
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-CSRF-Token': csrfToken
    },
    data: JSON.stringify(node),
    success: function (node) {
      console.log(node);
    }
  });
}

var newNode = {
  type:'video',
  title: 'Example node title3',
  body: {
    und: {
    0: {
    value :"Body"}
       }
    },
  field_url:{
   und: {
     0: {
      value :"http://url.com"
    }
    }
    },
field_video:{
 ???????
 }
};

getCsrfToken(function (csrfToken) {
  postNode(csrfToken, newNode);
});
  })(jQuery);

How can I pass a file (or an image) to REST service? what should I write instead of ??????
Answer: You could upload the file first using the file:create_raw services resource, which will return a file id (fid). Then you create your node and pass the fid in the file field {fid: "x}:
var data = new FormData();
data.append('files[file_1]', file);

jQuery.ajax({
    url: 'http://example.com/rest/file/create_raw',
    data: data,
    async: false,
    cache: false,
    contentType: false,
    processData: false,
    type: 'POST',
    success: function(result){
        alert(result);
    }
});

Comment: can you provide more details or an example or useful example link?
Comment: I have added some sample code
Comment: thanks, what is data, base64decode of file?
Comment: @zhilevan, no `data`'s type is [FormData](https://developer.mozilla.org/en-US/docs/Web/API/FormData) which provides a way to easily construct a set of key/value pairs representing form fields. `files[file_1]` is the key and `file` contains the raw file, either from HTML form file input , or a Blob, s. [Using FormData Objects](https://developer.mozilla.org/en-US/docs/Web/API/FormData/Using_FormData_Objects)
Answer: You will want to base64 encode the image file before you submit your document to the REST endpoint.
You an use https://www.base64encode.org/ to get an sample encoded image.
Comment: I know it is base64 file, but how to pass to Drupal REST( with which format)?
provide your answer according to my question please.

Pass video file


How do I insert an encrypted data bag item value into a Chef recipe?

Question: I've created an encrypted data bag value that I'm trying to load into a chef recipe.
knife data bag show foo bar --secret_file secret.key
Encrypted data bag detected, decrypting with provided secret.
id:   bar
pass: p4ssw0rd

I'm trying to get the pass value to load up as a variable in a bash resource, and have the encrypted_data_bag_secret in /etc/chef on the client (hence no secret key show, reverting to default /etc/chef location):
dbag = Chef::EncryptedDataBagItem.load("foo", "bar")
foo_pass = dbag["pass"]

I've also tried using the recipe DSL instead of Chef::EncryptedDataBadItem method:
dbag = data_bag_item('foo', 'bar')
foo_pass = dbag["pass"]

And then loading it into a bash resource:
bash 'install_mysql' do
  code <<-EOC
    ...
    sudo mysqladmin -u root password {foo_pass}
    ...
  EOC
end

I had a few questions regarding this process.
i) Will Chef::EncryptedDataBagItem.load be deprecated and replaced with data_bag_item; should I use one over the other?
ii) Am I pulling the dbag["pass"] using the correct methods; how would I grab the 'pass' value from inside foo (data bag) => bar (item?)
iii) To call the foo_pass variable inside the bash resource, do I just encapsulate the variable in curly braces {}, or am I missing something here?
iv) Is there a better method than what I am trying out?
I've tried adding the following to see if I can see the variable value printed to screen when running the chef-client, but it's not showing me any of the text or values:
puts "foo_pass equals 1:{foo_pass} 2:#{foo_pass}'

I've been hammering away at this for half the day, and was hoping to get some more experienced responses as how to handle this.
Comment: "uh-huh uh-huh. he said `dbag` "  https://media1.giphy.com/media/TpFvrFrtm5BiU/giphy.gif
Answer:
Yes prefer data_bag_item in most cases, it is more correct.
Yes, that is correct.
You need #{foo_pass}, with the leading #.
Comment: coderanger, your responses have always been greatly appreciated! I used the hash character, and it appeared that I did miss an instance of it in my code. Thank you for clarifying!


Regression - mean squared error

Question: For $i=1,2,\dots,n$, let $Y_i = f(x_i) + \epsilon_i$, where $\epsilon_i$ are iid $N(0,\sigma^2)$. Let $f$ be estimated by $\hat{f}$. Let $x_0$ be a new value of $x$ (not any of $x_1,\dots,x_n$), so $Y_0 = f(x_0) + \epsilon_0$; we assume $\epsilon_0 \sim N(0,\sigma^2)$ and $\epsilon_0$ is independent of $\epsilon_1, \dots, \epsilon_n$. Show that \begin{equation*}E\left((Y_0-\hat{f}(x_0))^2\right) = \sigma^2 + \text{Var}(\hat{f}(x_0))+\left(\text{Bias}(\hat{f}(x_0))\right)^2\end{equation*}
By rewriting the left-hand side as $E\left(\left((Y_0-f(x_0))+(f(x_0)-\hat{f}(x_0))\right)^2\right)$, expanding, and making some manipulations, I have reduced the problem to showing that $E(\epsilon_0 \hat{f}(x_0)) = 0$. But I cannot figure out how to do that. Any assistance would be much appreciated.
Comment: Could you elaborate on how $\hat{f}$ is estimated? Is it a linear regression, and are the $x$'s real numbers or vectors?
Comment: The $x$s are real numbers. The only other information given about $\hat{f}$ is that it's an estimator for $f$ which depends on $(x_i,y_i)$ for $i=1,2,\dots,n$.
Comment: Thanks. Is $\epsilon_{0}$ not independent of all the other variables in the model? And so:  $E(\epsilon_0 \hat{f}(x_0)) = E(\epsilon_0) E(\hat{f}(x_0)) = 0$. Thus, the answer is immediate from you reduction? Since $\hat{f}$ is estimated without $x_{0}$, then $\hat{f}$ should be independent of $\epsilon_0$ regardless of how it is estimated.
Comment: Yes, I guess that's right, as discussed more fully in K. A. Buhr's answer below. I was trying to apply the formal definition of independence and couldn't see it that way.
Answer: We assume here that all the $x_i$ and $f$ are deterministic, and the only source of randomness are the $\epsilon_i$s.  The expression $\hat f(x_0)$ applies a random function to a non-random value $x_0$.  What makes $\hat f$ random?  It is the result of a regression of $Y_i$ on $x_i$ for $i=1,\dots,n$.  The only source of randomness in these observations are the $\epsilon_i, i=1,\dots,n$, so $\hat f$ is a deterministic function of $\epsilon_1,\dots,\epsilon_n$.
Thanks to the independence, of $\epsilon_0$ from $\{\epsilon_1,\dots,\epsilon_n\}$, we have $\epsilon_0$ independent from $\hat f$ and therefore from $\hat f(x_0)$, its application to the non-random value $x_0$.  So, your expectation is the expectation of a product of two indepedent random variables, which can be written as the product of the expectations, one of which is $E(\epsilon_0)=0$.  Therefore, the expectation is zero, as required.

Regression - mean squared error