How to remove rows with specific missing tags from a Pandas dataframe?

Question: In a Pandas dataframe, '-999' (as integer) is used as a tag to indicate the 'cells' with missing data. I am cleaning up the data by removing rows if a row contains '-999' in it. I tried this method:
flag = (dataframe != -999)
dataframe = dataframe[flag]

However, the resulting dataframe still has the same shape and the cells with -999 now became empty. Then I used another line:
dataframe.dropna(axis = 0, how = 'all', inplace = True)

But it did not remove those rows as I expected. Can someone help? Thanks!
Answer: you can use .any(axis=1) or .all(axis=1) for that:
In [92]: df
Out[92]:
     a    b    c
0    8    7    6
1    8    0 -999
2    8    9    9
3 -999    8    9
4    4    7    6
5    5    9    9
6    6    4    8
7    5 -999    9
8    5    0    5
9    0    6    5

In [93]: df.loc[~(df == -999).any(axis=1)]
Out[93]:
   a  b  c
0  8  7  6
2  8  9  9
4  4  7  6
5  5  9  9
6  6  4  8
8  5  0  5
9  0  6  5

or, alternatively, using .all(axis=1):
In [94]: df.loc[(df != -999).all(axis=1)]
Out[94]:
   a  b  c
0  8  7  6
2  8  9  9
4  4  7  6
5  5  9  9
6  6  4  8
8  5  0  5
9  0  6  5


Unfair die probability distribution

Question: Hello, I'm having some trouble wrapping my head around this one:
An unfair die looks like an ordinary 4-sided die but the probability of a face landing on 2 is three times the probability of landing on 1. Similarly, the probability of landing on 3 is two times that of landing on 1 and the probability of landing on 4 is four times that of landing on 1. Let Y denote the number of dots on the "up" face.

What is the probability distribution of Y?
How could I determine the probability of it having, say, three dots?
Comment: Please check http://stats.stackexchange.com/tags/self-study/info and add `[self-study]` tag accordingly.
Comment: I do not see this as a self-study question.
Answer: It simply follows a categorical distribution (or multinomial if you want to count the number of times it landed on each side, but that's not your question):
$$ \Pr(X = x) = \begin{cases}
\alpha   & \text{if }x=1, \\
3\alpha  & \text{if }x=2, \\
2\alpha  & \text{if }x=3, \\
4\alpha  & \text{if }x=4
\end{cases} $$
such that $\alpha + 3\alpha + 2\alpha + 4\alpha = 1$ since probabilities need to sum up to unity. Solving it needs just a basic algebra.
Comment: I think a more interesting question is how do you physically shave the die to make this work.  Shaving a die makes the opposite sides have the same surface area.  The sides with the largest surface area have higher probability of occurrence.  On a standard die 3 is opposite to 4, 2 to 5 and 1 to 6.  If you ignore the effect of the number of dots on the face.  we need P(1)=P(6), P(2)=P(5) and P(3)=P(4) and                require P(1) + P(2) + P(3) + P(4) + P(5) + P(6) =1.
Comment: The die is shaped this way in order to make it look similar to a cube.  I think it is important to note that a die has 6 sides! Tim has the concept right but the OP and Tim both fail to think about the physical shape of a cube.
Comment: @MichaelChernick there is a physical [four-side die](https://en.wikipedia.org/wiki/Four-sided_die). Moreover, this seems to be rather a basic probability/statistics homework question, then one asking about some real die and it's properties (interesting question for itself).
Comment: What does a three dimensional 4-sided die look like?  If it is a cube it has 6 sides.  I absolutely agree that assuming a 4-sided die exists you have given the right solution.  I actually gave you a plus for that.  Be aware that homework problems fall into the category of self-help whether the OP tags it that way or not.  So you are suppose to give hints rather than a complete answer.  With your reputation I would think you would know that.
Comment: @MichaelChernick it's a [tetrahedron](https://en.wikipedia.org/wiki/Tetrahedron)
Comment: @MichaelChernick There's several different designs for fair four-sided dice and several more that generate values between 1 and 4 but have more than four sides (e.g. 8-sided dice numbered 1-4 twice); I own several different kinds of four-sided dice.
Comment: I got my question answered @Glen_b.  I can imagine lots of 3 -D objects that can be called die and may physically exist.  Tim's example of a pyramid was satisfactory enough for me.  But we all know that the standard that gamblers use is the 6 sided cube.  There may be some intersting discussion on that with his new post.

Unfair die probability distribution


Como utilizar o alter table modificando um atributo para PK e FK?

Question: Eu tenho uma tabela Funcionário, onde sua PK é esse Cod_Func.
Depois eu criei essa tabela Vendedor, criei os os atributos manualmente, não coloquei o Cod_Func da Tabela Vendedor como PK, porque esse é um atributo que já existe na Tabela Funcionário.
Só que eu tenho um relacionamento entre as duas tabelas onde a Tabela Vendedor o Cod_Func é PK e FK.
Quando eu tenho usar o
ALTER TABLE [dbo].[VENDEDOR]
ADD CONSTRAINT PK_COD_FUN_VENDEDOR PRIMARY KEY(COD_FUN)

ele me gera um erro falando que não definir essa constraint nessa tabela, usando o foreign key ele funciona de boa.
Comment: Não seria ideal você ter um Cod_Vendedor nessa tabela (Vendedor) como PK ? Tambem não entendi o que você quer fazer, você quer adicionar o que na tabel vendedor? Você ja tem a FK não ?
Comment: Acredito que o que ele queira, é criar uma relação 1 para 1 entre funcionário e vendedor.
Answer: Primary Key (Chave Primária) -> Garante a Unicidade
Foreign Key (Chave Estrangeira) -> Garante a ligação de um ou vários registros a um campo de uma tabela...

Possível Solução:
Crie uma FK na tabela vendedor, e em seguida crie uma trigger para impedir que dois vendedores estejam ligados ao mesmo funcionário.

Oracle PL/SQL

CREATE TRIGGER tg_vendedor
BEFORE INSERT OR UPDATE
ON vendedor
FOR EACH ROW
REFERENCING NEW AS NEW
DECLARE cod_f number;

BEGIN
 SELECT COUNT(cod_func) INTO cod_f FROM vendedor WHERE cod_func = :new.cod_func;

 IF cod_f > 0 THEN
  raise_application_error( -20001, 'O funcionário já e um vendedor!' );
 END IF;

END;
/

Está trigger verifica antes de fazer Update ou Insert na tabela, se já existe um vendedor que possui uma FK ligada ao funcionário
Se caso já existir um vendedor que está usando a FK do funcionário já cadastrado, ele aborta o Insert ou Update
--
Sou estudante, creio que isso te ajude!
Comment: Não seria melhor criar uma relação 1 para 1, onde um vendedor é um funcionário? Nessa situação as duas tabelas estariam ligadas pela chaves primárias e não precisaria de trigger para resolver a questão de dois vendedores estarem ligados ao mesmo funcionário. Ou outra alternativa, seria também criar uma unique constraint, evitando assim também a criação da trigger.
Comment: Por que não se deve em hipótese alguma utilizar uma `FK` como `PK`? Ele pode usar uma chave composta com a chave de outra tabela.
Comment: André, agradeço.
Mas ainda não estou nesse estágio de criar Trigger. Ficou meio complexo para meu entendimento ai.
Mas vlw pela ajuda.
Consegui aqui. Vlw vlw
Answer:
Então, eu tenho uma tabela Funcionário, onde sua PK é esse Cod_Func.
Depois eu criei essa tabela Vendedor, criei os os atributos manualmente, não coloquei o Cod_Func da Tabela Vendedor como PK,
porque esse é um atributo que já existe na Tabela Funcionário.

O que está definido na tabela Funcionário vale somente para ela; não é herdado pela tabela Vendedor, mesmo que uma das colunas da tabela Vendedor referencie outra coluna na tabela Funcionário.

Para definir uma coluna existente como chave primária é necessário que ela não permita a ausência de valores; isto é, a coluna deve estar declarada como NOT NULL. A demonstração é simples.
Considerando-se relacionamento 1:1 entre Funcionário e Vendedor, e que já tenham sido definidas anteriormente as seguintes declarações de tabelas:
-- código #1
CREATE TABLE Funcionário (
     Cód_Func int,
     Nome_Func varchar(50),
     constraint I1_Funcionário primary key (Cód_Func)
);

e
-- código #2
CREATE TABLE Vendedor (
     Cód_Func int,
     Num_Percentual_Comissão numeric(3,1),
     constraint FK_Vend_Func foreign key (Cód_Func)
                             references Funcionário (Cód_Func)
);

ao executar código com instrução ALTER TABLE, para criar a chave primária da tabela Vendedor
-- código #3
ALTER TABLE Vendedor
     add constraint I1_Vendedor primary key (Cód_Func);

as seguintes mensagens de erro são exibidas:
Mensagem 8111, Nível 16, Estado 1, Linha 1
Não é possível definir a restrição PRIMARY KEY em coluna anulável na tabela 'Vendedor'.
Mensagem 1750, Nível 16, Estado 0, Linha 1
Não foi possível criar a restrição. Consulte os erros anteriores.

Ou seja, a coluna Cód_Func da tabela Vendedor deve ser antes definida como NOT NULL, o que é simples de executar:
-- código #4
ALTER TABLE Vendedor
     alter column Cód_Func int not null;

Feito isto, pode-se executar novamente o código #3 acima que a tabela Vendedor aceitará a coluna Cód_Func como chave primária.

O usual, ao se declarar as colunas de uma tabela, é definir como NOT NULL todas aquelas que sejam de preenchimento obrigatório, mesmo que não seja utilizada para compor a chave primária. No código #1, a coluna Cód_Func da tabela Funcionário propositalmente não foi declarada como NOT NULL. Mas na declaração da tabela consta a restrição I1_Funcionário como primary key, o que implicitamente torna a coluna Cód_Func como NOT FULL.
O ideal é definir explicitamente os atributos de cada objeto durante a declaração:
-- código #5 v2
CREATE TABLE Funcionário (
     Cód_Func int not null,
     Nome_Func varchar(50) not null,
     constraint I1_Funcionário primary key (Cód_Func)
);

CREATE TABLE Vendedor (
     Cód_Func int not null,
     Num_Percentual_Comissão numeric(3,1) not null,
     constraint I1_Vendedor primary key (Cód_Func),
     constraint FK_Vend_Func foreign key (Cód_Func)
                             references Funcionário (Cód_Func)
);

Comment: A agora eu entendi onde estava o erro, teve a manha de explicar, fiz aqui e deu certinho!!
Muito obrigado!!
Comment: @LuizGustavo Deu um pouco de trabalho escrever a resposta, mas valeu pelo resultado!
Answer: Coloquei aqui a tabela funcionário para ver se fica melhor de entender oq eu estou querendo.
Eu tenho a tabela funcionário e tenho essa tabela vendedor, o funcionário tem o Cod_Func que é a PK.
E o vendedor também é um funcionário, então ele "herda" esse COD_FUNC da tabela funcionário, então faço a chave composta, na tabala vendedor eu queria que ficasse como PK e FK.


If $D$ isn't a convex billiard table, then the billiard map $T$ is not continuous

Question: I am currently reading chapter 3 of "Geometry and Billiards" by Serge Tabachnikov, and I have some doubts about the need of using convex sets.
So, here's how the billiard map is defined:

To fix ideas, consider a plane billiard table $D$ whose boundary is
a smooth closed curve $\gamma$. Let $M$ be the space of unit tangent vectors
$(x, v)$ whose foot points $x$ are on $\gamma$ and which have inward directions.
A vector $(x, v)$ is an initial position of the billiard ball. The ball moves
freely and hits $\gamma$ at point $x_1$; let $v_1$ be the velocity vector reflected
off the boundary. The billiard ball map $T : M \to M$ takes $(x, v)$ to
$(x_1, v_1)$.

So far so good, the definition makes perfect sense to me. Then it is claimed

Note that if $D$ is not convex, then $T$ is not continuous: this
is due to the existence of billiard trajectories touching the boundary
from inside.

I am confused with this statement. For starters, are we talking with respect to which topology? Should I assume the subspace topology in $\mathbb{R}^2$? And why does not being convex affect continuity? Since the particle in a billiard trajectory starts in the table inside the curve, doesn't it always touch the boundary from the inside? Or am I misinterpreting the justification?
Answer: To begin with, note that the "unit tangent vectors $(x,v)$" are not tangent to the curve $\gamma$, but rathert tangent to the plane that $\gamma$ and $D$ live in. That is, $x$ is a point in the boundary set $\partial D$ whihc itself is a subset of $\mathbb R^2$ (and looks nice enough to be parametrized by a sufficiently nice function $\gamma\colon [0,1]\to\mathbb R^2$).
The vector $v$ also lives in $\mathbb R^2$ (though by it having unit length, it is restricted to the unit circle $S^1$).
Thus all $(x,v)$ are naturally elements of $\mathbb R^4$; hence it is understood that we consider $M$ as subspace of $\mathbb R^2$ and accordingly use the subspace topology.
Note that $M$ itself is twodimensional (because $x$ is on the onedimensional $\partial D$ boundary and $v$ on the onedimensional $S^1$), but cannot be viewed as a subspace of §\mathbb R^2$ globally. (It is however homeomorphic to a subspace of the torus).
Once it is clear what topology we consider on $M$, the rest of the questions perhaps becomes clearer.
Loosely speaking, if $D$ is not convex, then we can pick a suitable point $x_0\in\partial D$ and let $v$ "swipe"  continuously over its allowed values, and it may happen that the value of $T(x_0,v)$ "jumps". Namely, we find points $x_0$ and $x_1$ on $\partial M$ such that the line segment $x_0x_1$ is not completely inside $D$. But we can Also find $x_2$  - e.g., very close to $x_0$ - such that $x_0x_2$ is completely in $D$ (or can we?). Then are are unit vectors $v_1=\frac{x_1-x_0}{|x_1-x_0|}$ and $v_2=\frac{x_2-x_0}{|x_2-x_0|}$ such that $v_i$ points to $x_i$ from $x_0$ (but do the $(x_0,v_i)$ really belong to $M$? I.e., do the $v_i$ necessarily point to the interior from $x_0$?). When $v$ changes continuously from $v_1$ to $v_2$ (why can it? I.e., might there be gaps where $v$ is not pointing to the interior?), then $T(x_0,v)$ moves from $T(x,v_1)$, which is some point $(y_1,w_1)\in M$ such that by construction certainly $y_1\ne x_1$,  to $T(x,v_2)$, which is of the form $(x_2,w_2)$. That is, the first component of $T(x_0,v)$ changes from $y_1$ to $x_2$ without ever passing through $x_1$. This shows (how?) that $T$ is not continuous if $D$ is not convex. Or actually, this is just the idea and a very rough sketch of a proof. For a formal proof it leaves a lot of questions open (and I highlighted only very few of them for you to think about).


Using chrome.tabs.executeScript to check a box, not working

Question: I am not able to get the checkbox in this page to be checked.

    <form name="f">
      <input type="checkbox" name="vehicle" value="Bike" /> 1

    </form>

My popup.js is below:
function click(e) {
  chrome.tabs.executeScript(null,
      {code:"document.body.f.vehicle.checked='true'"});
  window.close();
}

document.addEventListener('DOMContentLoaded', function () {
  var divs = document.querySelectorAll('div');
  for (var i = 0; i < divs.length; i++) {
    divs[i].addEventListener('click', click);
  }
});

In the console log I keep getting the error: Uncaught TypeError: Cannot read property 'vehicle' of undefined.
My popup.html is below:
<!doctype html>

    <title>Set Page Color Popup</title>

1

Any help would be appreciated.
Answer: It should be like this,
document.f.vehicle.checked='true';

http://jsfiddle.net/PnXsE/
Comment: I tried that (document.f.vehicle.checked='true';) even. I still get the exact same error.


JSF 2.0 rapid web design tool

Question: I'm learning about JSF 2.0, I want to know if there is a tool for rapid web design.
I looked around and found RichFaces but it seems not to be compatible with Glassfish v3 (it's the server I use by the way).
I'll apreciate your advice.
Thanks in advance.
Answer: PrimeFaces
http://www.primefaces.org/
Answer: Primefaces is the best lightweight rapid web designing tool.

JSF 2.0 rapid web design tool


IBM ServeRAID bad stripes and synchronize

Question: After one of the drives in our IBM RAID gone, we bought a new one and replaced it.
The rebuilding went fine, but now we see "Bad stripes" reported for the logical disk.
Moreover, some data can not be read from the disk, and ServeRAID reports bad blocks.
CHKDSK just hangs on 22% of reading, closing it and re-running jumps to the 22% again.
Question - is it possible to resolve this situation by synchronizing the logical drive?
Or it better leave the CHKDSK running until it hopefully completes?
Also, what the synchronization actually does and can it heal bad stripes?
The server model is x3400, RAID is ServeRAID 8k-l.
Thanks!
Comment: I don't have a solid answer for you, but my advice would be to contact IBM; support is one of the reasons you pay a premium for good server hardware. Also - You haven't said what model of server and RAID controller you have, that will probably make a difference for the answer as well.
Comment: Correct - question updated.
Answer: Are you sure the new hard disk isn't "DOA"?
Comment: RAID1 actually, but that was hell indeed :).
Comment: The disk is fine, the rebuild completed successfully. According to IBM, disks can be re-built and still considered with bad stripes - i.e. contain bad blocks.
Comment: I think you should get some answers from IBM. Maybe there's more than one faulty hard disk, or your RAID card is causing corruption. Your trouble wasn't caused by a single faulty disk.
Comment: It seems BOTH disks became faulty at same time - just our luck it seems...
Comment: Welcome to RAID5 hell :-)


Can a trend stationary series be modeled with ARIMA?

Question: I have a question / confusion about stationary series required for modeling with ARIMA(X). I am thinking of this more in terms of inference (effect of an intervention), but would like to know if forecasting versus inference makes any difference in the response.
Question:
All the introductory resources I have read state that the series needs to be stationary, which makes sense to me and that is where the "I" in arima comes in (differencing).
What confuses me is the use of trends and drifts in ARIMA(X) and implications (if any) for stationary requirements.
Does the use of either a constant/drift term and/or trend variable as an exogenous variable (i.e. adding 't' as a regressor) negate the requirement of the series being stationary? Is the answer different depending on if the series has a unit root (e.g. adf test) or has a deterministic trend but no unit root?
OR
Does a series always have to be stationary, made so via differencing and/or detrending before using ARIMA(X)?
Answer: Looking at the comments it seems that we didn't address the question about how to choose between a deterministic or stochastic trend. That is, how to proceed in practice rather than the consequences or properties of each case.
One way to proceed is the following: Start by applying the ADF test.

If the null of a unit root is rejected we are done. The trend (if any) can be represented by a deterministic linear trend.
If the null of the ADF test is not rejected then we apply the KPSS test
(where the null hypothesis is the opposite, stationarity or stationarity around a linear trend).
o If the null of the KPSS test is rejected then we conclude that there is a unit root and work with the first differences of the data. Upon the first differences of the series we can test the significance of other regressors
or choose an ARMA model.
o If the null of the KPSS test is not rejected then we would have to say that the data are not much informative because we weren't able to reject none the of the null hypotheses. In this case it may be safer to work with the first differences of the series.

As mentioned in a previous answer, remember that these tests may be affected by the presence of outliers (e.g. an outlier at a single time point due to an error when recording the data or a level shift due for example to a policy change that affects the series from a given time point on). Thus, it is advisable to check these issues as well and repeat the previous analysis after including regressors for some potential outliers.
Comment: @pidosaurus lack of agreement between and ADF and KPSS tests may be due to small sample size, presence of outliers, non-linear trend,...
if after further inspection still not clear which one is more appropriate,
it may be safer to consider the presence of a unit root. A quick view to your data suggested to me the presence of a non-linear trend, a quadratic trend of the form $a_1 t + a_2 t^2$ may be appropriate.
Comment: Awesome! Was my comment question above correct that if we see what looks like a trend, we use the ADF test that includes a trend (#3 option in the link I posted)?
Comment: Last question - what do you make of the situation where you fit an ARIMA, say ARIMA(0,1,1) to a series and the mean of the difference is non-zero? This means I believe that you add a constant to the model - which also represents a linear trend in the original series. What does this case signify? Is the trend in the original series deterministic because differencing the series did not remove the trend?
Comment: @B_Miner Regarding your first comment, I would start by including only an intercept. If the trend looks somewhat exponential you can also add the slope parameter of the linear trend and see if it is significant. In general, it is better to start with a model with few parameters and if the diagnostic of the residuals is not satisfactory then consider adding other elements.
Comment: @B_Miner Taking first differences removes both a deterministic and a stochastic trend. If you see a trend in the differenced series in a model with intercept, then you should consider taking differences again (i.e. test for a second unit root).
Comment: What if ADF and PP reject $H_0$ of unit root, but KPSS (level and trend) reject $H_0$ of stationarity? My time-series: https://pastebin.com/raw/VF7XzxVP
Answer: As empirically shown below, using a trend variable in ARIMAX  negated the need for differencing and makes the series trend stationary. Here is the logic I used to verify.

Simulated an AR process
Added a deterministic trend
Using ARIMAX modeled with trend as exogenous variable the above series without differencing.
Checked the residuals for white noise and it is purely random

Below is the R code and plots:
<code>    set.seed(3215)

    ##Simulate an AR process
    x <- arima.sim(n = 63, list(ar = c(0.7)));
    plot(x)

    ## Add Deterministic Trend to AR
    t <- seq(1, 63)
    beta <- 0.8
    t_beta <- ts(t*beta, frequency=1)
    ar_det <- x+t_beta
    plot(ar_det)

    ## Check with arima

    ar_model <- arima(ar_det, order=c(1, 0, 0), xreg=t, include.mean=FALSE)

    ## Check whether residuals of fitted model is random

    pacf(ar_model$residuals)
</code>
AR(1) Simulated Plot

AR(1) with deterministic trend

ARIMAX Residual PACF with trend as exogenous. Residuals are random, with no pattern left

As can be seen above, modeling deterministic trend as an exogenous variable in the ARIMAX model negates the need for differencing. At least in the deterministic case it worked. I wonder how this would behave with stochastic trend which is very hard to predict or model.
To answer your second question, YES all ARIMA including ARIMAX have to be made stationary. At least that's what text books say.
In addition, as commented, see this article. Very clear explanation on Deterministic Trend vs. Stochastic trend and how to remove them to make it trend stationary and also very nice literature survey on this topic. They use it in the neural network context, but it is useful for general time series problem. Their final recommendation is when it  is clearly identified as deterministic trend, the do linear detrending, else apply differencing to make the time series stationary. The jury is still out there, but most researchers cited in this article recommend differencing  as opposed to linear detrending.
Edit:
Below is random walk with drift stochastic process, using exogenous variable and difference arima. Both appear to give same answer and in essence they are same.
<code>    library(Hmisc)

    set.seed(3215)

    ## ADD Stochastic Trend to simulated Arima this is AR(1) with unit root with non zero mean

    y = rep(NA, 63)
    y[[1]] <- 2

    for (i in 2:63)  {
    y[i] <- 3+1*y[i-1] + rnorm(1, mean = 0, sd = 1)
    }

    plot(y, type="l")

    y_ts <- ts(y, frequency=1)

    ## Lag to create Xreg

    y_1 <- Lag(y, shift=1)

    ## Start from 2 value to avoid NA and make it equal length with xreg

    y <- window(y_ts, start =2, end=63)
    xreg1 <- y_1[-1]

    ## Check the values with ARIMA and xreg

    g <- arima(y, order=c(0, 0, 0), xreg=xreg1)

    pacf(g$residuals)

    ## Check the values with ARIM

    g1 <- arima(y, order=c(0, 1, 0))

    pacf(g1$residuals)

    ##

    ARIMA(0, 0, 0) with non-zero mean

    Coefficients:
          intercept   xreg1
             3.1304  0.9976
    s.e.     0.2664  0.0025
</code>
Comment: The link to the article is now dead. It is a good idea to provide some info (e.g. article title) instead of a mere uninformative link, so that even if the link goes dead we can search for it (now we can't :( ).
Comment: I am interested in others views too - I am not sure, does the residual being white noise meet the requirement for the series being stationary - i.e. if you achieve white noise can you be satisfied? Or, is the inclusion of the exogenous variable actually acting to "de-trend" and make this series stationary? I wonder if a check on this later question is if you get the same model (ar1 coefficient, etc) if you de-trend with linear regression and then fit the arima(1,0,0)...I did this and the results are close. So maybe adding the exogenous variable is the same as detrending.
Comment: Yes it is the same, few months back I came across this in a neural network forecasting article. I'll provide reference if I find it.
Comment: Any idea about the case when there is a unit root or when there is a constant in the model?
Comment: I have updated my answer with the article I was referencing.
Comment: I think you may have not saved it.
Comment: @forecaster If I undesrstood correctly your code, you simulate an autoregressive model with coefficient $0.86$. This is a stationary process that, as you find, does not require differencing. The term stochastic trend refers to a process that contains a unit root; a random walk, where the shocks have an everlasting effect and generate a long term trend. In a stationary process the
effect of shocks disappears after a few observations and there is no trend effect or long term cycle. Maybe you would like to try some experiments with a random walk, which can be simulated as `cumsum(rnorm(n=100))`.
Comment: @javlacalle, the original code forecaster posted correctly spoke to one of my sub-questions: "what if a series has a deterministic trend but no unit root?" Correct?
Comment: @B_Miner I have modified the second version of the code to include unit root with constant.
Comment: Please note that the second example is equivalent to 1st order differencing
Answer: Remember that there are different kinds of non-stationarity and different ways on how to deal with them. Four common ones are:
1) Deterministic trends or trend stationarity. If your series is of this kind de-trend it or include a time trend in the regression/model. You might want to check out the Frisch–Waugh–Lovell theorem on this one.
2) Level shifts and structural breaks. If this is the case you should include a dummy variable for each break or if your sample is long enough model each regimé separately.
3) Changing variance. Either model the samples separately or model the changing variance using the ARCH or GARCH modelling class.
4) If your series contain a unit root. In general you should then check for cointegrating relationships between the variables but since you are concerned with univariate forecasting you shoud difference it once or twice depending on the order of integration.
In order to model a time series using the ARIMA modelling class the following steps should be appropriate:
1) Look at the ACF and PACF together with a time series plot to see wheter or not the series is stationary or non-stationary.
2) Test the series for a unit root. This can be done with a wide range of tests, some of the most common being the ADF test, the Phillips-Perron (PP) test, the KPSS test which has the null of stationarity or the DF-GLS test which is the most efficient of the aforementioned tests. NOTE! That in case your series contain a structural break these tests are biased towards not rejecting the null of a unit root. In case you want to test the robustness of these tests and if you suspect one or more structural breaks you should use endogenous structural break tests. Two common ones are the Zivot-Andrews test which allows for one endogenous structural break and the Clemente-Montañés-Reyes which allows for two structural breaks. The latter allows for two different models. An additive outlier model which accounts for sudden changes in the slope of the series and an innovative outlier model which takes gradual changes into account and allows a break in the intercept and slope.
3) If there is a unit root in the series then you should difference the series. Afterwards you should run look at the ACF, PACF and the time series plot and probably check for a second unit root to be on the safe side. The ACF and PACF will help you decide on how many AR and MA terms you should be including.
4) If the series does not contain a unit root but the time series plot and the ACF show that the series has a deterministic trend you should add a trend when fitting the model. Some people argue that it is completely valid to just difference the series when it contains a deterministic trend although information may be lost in the process. Never the less its a good idea to difference it in order to see have many AR and/or MA terms you will need to include. But a time trend is valid.
5) Fit the different models and do the usual diagnostic checking, you might want to use an information criterion or the MSE in order to select the best model given the sample you fit it on.
6) Do in sample forecasting on the best fitted models and calculate loss functions such as MSE, MAPE, MAD to see which of them actually perform best when using them to forecast because that is what we want to do!
7) Do your out of sample forecasting like a boss and be pleased with your results!
Comment: Yes exactly. Often the Ljung-Box Q test is used to test the residuals in order to see whether or not they are random. In case they are then the model is an adequate representation of the data. Something which I forgot to mention in the above answer is also that after you have fitted the model you can look at the ACF and time series plot of the residuals of the fitted series. This will give you a good indication whether or not your residuals are white noise or not (the ACF should not have any significant lags). Anyways, most statistical packages have a command for the Ljung-Box Q test.
Comment: And to answer your question quickly. Yes it can.
Comment: I wasn't allowed to comment on the above since I dont have enough reputation but I want to point out that a white noise process is stationary. Its mean and variance does not change over time so it is stationary!
Comment: Dan, great answer! Regarding your last comment here, are you saying that if you end up with white noise residual, which is stationary....then it tells you that you have modeled the series adequately (i.e. dealt with issues of stationarity properly / adequately)?
Comment: Here is the wiki link for the Ljung-Box Q test: http://en.wikipedia.org/wiki/Ljung%E2%80%93Box_test
Answer: Determining whether the trend (or other component such as seasonality) is deterministic or stochastic is part of the puzzle in time series analysis. I will add a couple of points to what has been said.
1) The distinction between deterministic and stochastic trendsis important because if a unit root is present in the data (e.g. a random walk) then test statistics used for inference do not follow the traditional distribution. See this post for some details and references.
We can simulate a random walk (stochastic trend where first differences should be taken), test for the significance of deterministic trend and see the percentage of cases in which the null of deterministic trend is rejected. In R, we can do:
<code>require(lmtest)
iter <- 10000
cval <- 0.05
n <- 120
rejections <- 0
set.seed(123)
for (i in seq.int(iter))
{
  x <- cumsum(rnorm(n)) # random walk
  fit <- lm(x ~ seq(n))
  if (coeftest(fit)[2,"Pr(>|t|)"] < cval)
    rejections <- rejections + 1
}
100 * rejections / iter
#[1] 88.67
</code>
At the 5% significance level, we would expect to reject the null in the 95% of cases, however, in this experiment it was rejected only in ~89% of cases out of 10,000 simulated random walks.
We can apply unit root tests to test whether a unit root is present. But we must be aware that a linear trend may in turn lead to failure to reject the null of a unit root. In order to deal with this, the KPSS test considers the null of stationarity around a linear trend.
2) Another issue is the interpretation of the deterministic components in a process in levels or first differences. The effect of an intercept is not the same in a model with a linear trend as in a random walk. See this post for illustration.
Analytically, let's take a random walk with drift:
$$
y_t = \mu + y_{t-1} + \epsilon_t \,,\quad \epsilon_t \sim NID(0, \sigma^2) \,.
$$
If we substitute repeatedly $y_{t-i}$ by lagged versions of $y_t$:
\begin{eqnarray*}
y_t &=& \mu + \underbrace{y_{t-1}}_{\mu + y_{t-2} + \epsilon_{t-1}} + \epsilon_t \\
&=& 2\mu + \underbrace{y_{t-2}}_{\mu + y_{t-3} + \epsilon_{t-2}} + \epsilon_{t-1} + \epsilon_t \\
&=& 3\mu + y_{t-3} + \epsilon_{t-2} + \epsilon_{t-1} + \epsilon_t \\
&...&
\end{eqnarray*}
We arrive to:
$$
y_t = y_0 + \mu t + \sum_{i=1}^t \epsilon_i
$$
where $y_0$ is some arbitrary initial value. Thus, we see that the accumulation of shocks and the long memory of the random walk makes the intercept $\mu$ to have the effect of a linear trend with slope $\mu$ (in this case the constant term $\mu$ is called a drift).
If the graphical representation of a series shows a relatively clear linear trend, we cannot be sure whether it is due to the presence of a deterministic linear trend or to a drift in a random walk process. Complementary graphics and tests statistics should be applied.
There are some caveats to bear in mind since an analysis based on unit root
and other test statistics is not foolproof. Some of these tests may be affected by the presence of outlying observations or level shifts and require the selection of a lag order which is not always straightforward.
As a workaround to this puzzle, I think that the common practice is to take differences of the data until the series looks stationary (for example looking at the autocorrelation function, which should go to zero fast) and then choose an ARMA model.
Comment: Greta post- you are clearly a great asset to this site! I was curious looking here and your other posts - is it possible to use a ADF or KPSS test to determine if what looks like a trend in the series in levels is deterministic or stochastic? I found this : http://faculty.smu.edu/tfomby/eco6375/BJ%20Notes/ADF%20Notes.pdf which makes it look like if you see a trend visually in the series, use option #3 of the test and if you dont reject the null, you have evidence for a deterministic trend.
Comment: Based on Dan's comment, I guess if you get white noise residuals, none of this matters :)


How to add custom login logout links in Bootstrap theme?

Question: I am new to drupal, but I had a requirement, my current site using bootstrap theme. Previously there was no login and logout buttons on header. Today I got html login and logout strip from desinger. For this I have created a new block and selected bootstrap region as Navigation. Now this custom header perfectly sits at the top of the page. with static Myaccout, Logout , contactus  which are sent by desinger.

      <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
        Toggle navigation

      </button>

Navigator

Contact Us

 My Account

 Log Out

But my question how to make that header dynamic like when user with out login need to show Login , and after login need to show myaccount and logout. Above code is the static html bootstrap header sent by desinger. Please suggest me how to do this?
Comment: Hi, you could render or place the user menu block in the navigation regions
Answer: It can be done this way

<?php
global $user;
 if($user->uid):?>

 My Account

 Log Out

<?php else: ?>

 Log in

<?php endif; ?>

Hope it will help
Answer: Edit the code between your ul class="nav navbar-nav navbar-right"> to make it look like this:
<?php if(global $user):?>

 My Account

 Log Out

<?php else: ?>

 Log in

<?php endif; ?>


Rails error: No HTTP_REFERER was set in the request to this action

Question: I am developing a new rails app. but found a error:

ActionController::RedirectBackError in UsersController#show
No HTTP_REFERER was set in the request to this action, so redirect_to :back could not be called successfully. If this is a test, make sure to specify request.env["HTTP_REFERER"].

my code:
rescue_from CanCan::AccessDenied do |exception|
     redirect_to :back
end

I have some question for this error:

what is HTTP_REFERER?
why redirect to back will trigger this error?

Anyone has good idea?
Answer: what is HTTP_REFERER?
The HTTP referer is an HTTP header field that identifies the address of the webpage (i.e. the URI or IRI) that linked to the resource being requested. This is set by ActionController::Request object.
why redirect to back will trigger this error?
This usually happens when request.env["HTTP_REFERER"] is not set. (I also wonder to know in which case it is set and not set)
You could refer this answer to fix your issue.
(OR) I would highly prefer to define a custom page for access denied and redirect to it instead of redirecting :back (which I think bad idea)
For example from cancan gem docs,
rescue_from CanCan::AccessDenied do |exception|
    render :file => "#{Rails.root}/public/403.html", :status => 403, :layout => false
    ## to avoid deprecation warnings with Rails 3.2.x (and incidentally using Ruby 1.9.3 hash syntax)
    ## this render call should be:
    # render file: "#{Rails.root}/public/403", formats: [:html], status: 403, layout: false
end

Hope this helps!
Comment: Thanks! I guess I know why I trigger the error, because I modify the URL manually, so the HTTP_REFERER is not exists.


how to retrieve the data from all the pages in python by selenium

Question: I want to retrieve the data from all pages in the link bellow. I retrieve just the first page data, but I couldn't get the rest of the pages. Could you complete my code?
url: https://www150.statcan.gc.ca/n1/pub/71-607-x/2021004/exp-eng.htm?r1=(1)&r2=0&r3=0&r4=12&r5=0&r7=0&r8=2022-02-01&r9=2022-02-01
<code>from selenium import webdriver
import time

url = "https://www150.statcan.gc.ca/n1/pub/71-607-x/2021004/exp-eng.htm?r1=(1)&r2=0&r3=0&r4=12&r5=0&r7=0&r8=2022-02-01&r9=2022-02-01"

driver = webdriver.Chrome("C:\Program Files\Python310\chromedriver.exe")
driver.get(url)
table = driver.find_element_by_id('report_table')
body = table.find_element_by_tag_name('tbody')
cells = body.find_elements_by_tag_name('td')

for cell in cells:
    print(cell.text)
</code>
Answer: You can use Selenium click() element method to click on the Next button, and using a for loop, you can iterate through all the pages.
For example,
<code># run the loop 26 times
for i in range(26):

    # your code
    table = driver.find_element_by_id('report_table')
    body = table.find_element_by_tag_name('tbody')
    cells = body.find_elements_by_tag_name('td')
    for cell in cells:
        print(cell.text)

    # click on the Next button
    driver.find_element_by_xpath('//*[@id="report_results_next"]').click()
</code>


Is there a non-verbose way to notate finger sustained notes in LilyPond?

Question: I recently started learning LilyPond and am trying to notate finger sustained notes as described in Is there piano notation for finger sustained notes (as opposed to pedal sustain)?. I would be fine with either one of the notations from the top two answers.
I was able to notate what's described in the second answer like so:
\relative f' { f4~ <f c'>~ <f c' f>2 }

I find the code I wrote very verbose because I have to repeat all of the previous notes whenever I add a new one. Ideally I would only need to specify the next note and that I'd like the previous notes to continue and tie over. Is there a less verbose way to express this or what's described in the top answer of the question I linked?
Answer: I don't know whether it's possible to shorten the code you have provided (I doubt it but perhaps someone will know a way). However you could use \set tieWaitForNote = ##t to a good effect here. When it is set to ##t, each note with tie waits for the next occurrence of the same note and ties it with that. So your snippet could be rewritten like this:
\relative f' { \set tieWaitForNote = ##t f4~ c'~ <f c' f>2 }

which would result in the first f tying right to the half-note f, without the need for the second f in between.
You can play with it in a lilybin that I put together for that.


How can I apply the results of groupby command to all rows?

Question: I have a dataframe as follows:

I want to have the max of the 'Sell_price' column according to 'Date' and 'Product_id' columns, without losting the dimension of dataframe as:

Since my data is very big, without a doubt using 'for' command is not logical.
Answer: I think you are looking for transform:
df['Sell_price'] = df.groupby(['Date', 'Product_id'])['Sell_price'].transform('max')


Samba name mangling too mangled to be practical

Question: Switched recently from W2K3 to Samba4.0.9/CentOS6.4 for our fileshare for WinXP clients.
Have an ancient (1995!) piece of software that uses 8.3 filename format.  After the switch, long filenames became useless in the context of the File->Open dialog box.  Instead of the first few characters, we get maybe 1 character the same if we're lucky, which in a directory of thousands makes it impossible to find.  For example, instead of "S:\Air conditioning control system" becoming "S:\AIRCON~1" like it would before, it's displayed in this program as "S:\A51FHG~S".
In our directory of client identifiers with their contact names appended, formerly directory mangling would leave enough characters intact that client identifiers could still be used.  Not anymore.
None of the settings in the docs seem to talk about this exact problem.  In fact, they seem to show it the way we were used to.  Our smb.conf doesn't use any of the settings because the defaults seem to be what we want, according to the docs.  Any hints?
Answer: If you only look at the section NAME MANGLING, you'll miss these handy parameters:
<code># sets it to the old Samba 2 way of doing things, closer to what you're used to with Windows
mangling method = hash
</code>
or, in our case, we wanted the first 6 characters to be seen, so we ended up going with:
<code># only with hash2 method, but lets you set the number of chars to preserve (1-6)
mangle prefix = 6
</code>


install sql server driver on ubuntu

Question: I want to connect to sql server database which is placed on a remote server while my php code has been installed on an ubuntu server. I found that it needs to install sqlsrv driver or mssql driver before connecting. How can I install the requirement(s) in advanced?
Comment: What's the interface your PHP application uses for connection? PDO, ODBC, *mssql_\*()* functions directly, any other?
Comment: I'm using PDO driver
Answer: Install PHP extension in OS: apt-get install php5-sybase, restart Apache or PHP-FPM (depending on your SAPI). Make sure the newly installed PDO driver is loaded (should be enabled automatically, check for output line php5_invoke: Enable module pdo_dblib for [your SAPI here]). This will also install all dependencies. Now you can use mssql: DSNs in PDO, as presented in PHP manual.
There's another PDO driver for MSSQL, but it only works under Windows.
You can also use PDO with ODBC.

install sql server driver on ubuntu


Perform JS function on all ID's on a page

Question:    var hslider_slider_id = jQuery('.hslider_misc_holder').attr('id').slice(8);
   activate_width_checker(1080, 1080, 'responsive', hslider_slider_id);

This code is overriding a conflict between fullpage.js and WP Hero Slider I have two pages with two sliders. This code is being called successfully on the first slider, but not the second. I need to refactor the code in order to make sure the JS function is called on the second slider, which has the same ID .hslider_misc_holder, not just the first. Do i need to run it through a loop in order to achieve this??
Or use .map
The conflict is causing the Hero Slider not to load full size.
Ant
Comment: Call it again when your second page is loaded.
Comment: Sorry if I wasn't clear there are two slider on each page, the second slide on each page isn't being affected by the function. Just need to run the function through a loop to call it on all instantiations of the ID. Not a great Programmer that's why I'm asking!!!
Comment: Have you tried wrapping your code inside an IIFE to avoid naming conflicts (e.g. `(function(){}())`) ?
Answer: First, the . prefix signifies a class and not an ID. The # prefix should be used for selecting an element by id.
Update your question to include an HTML snippet so I can understand the structure.
It sounds like you want to select multiple elements by class and then loop though these and pass the id (or a direct reference to the object) into the function.
    var sliders = jQuery('.hslider_misc_holder');
    $.each(sliders, function(idx, slider) {
            activate_width_checker(1080, 1080, 'responsive', $(slider).attr('id'));
    });

Comment: Sorry your right it is a class as the code indicates!!
Comment: I just want the second slider to be affected by the function at the moment it is not being called the first slider is. Just need to loop through all of the instances of this class on the page.
Comment: The class is part of the parent element of the slider
Comment: Yes I added the .each() function to loop the code and that then called all the instances of that class on the page. Appreciate all the help.
Answer: --- UPDATED
When you call .attr("id") the jQuery will return only the ID of the first element in the set. You will have to perform a look trought the set:
$(".hslider_misc_holder").each(function () {
  activate_width_checker(1080, 1080, 'responsive', $(this).attr("id").slice(8));
});

Answer: You can use a REGEX in selector. For example, on onclick event.
On two elements in html:

DIV1

DIV2

Now in JavaScript:
`
$('[id ^= DIV').on('click', function(){alert();});

`
REGEX selector allow find all id in html with same pattern.
I hope help you.


Telling Typedoc Not to Load Jasmine Annotations

Question: I'm getting this error in an Angular library project:
Using TypeScript 3.7.5 from /home/ole/Github/validatorjs-workspace/projects/validatorjs/node_modules/typescript/lib
Error: /home/ole/Github/validatorjs-workspace/node_modules/@types/jasmine/ts3.1/index.d.ts(20)
 Definitions of the following identifiers conflict with those in another file: beforeAll, beforeEach, afterAll, afterEach, describe, fdescribe, xdescribe, it, fit, xit, expect, DEFAULT_TIMEOUT_INTERVAL, CustomMatcherFactory, CustomEqualityTester
Error: /home/ole/Github/validatorjs-workspace/node_modules/@types/jasmine/ts3.1/index.d.ts(308)
 All declarations of 'ArrayContaining' must have identical type parameters.
Error: /home/ole/Github/validatorjs-workspace/node_modules/@types/jasmine/ts3.1/index.d.ts(312)
 All declarations of 'ObjectContaining' must have identical type parameters.
Error: /home/ole/Github/validatorjs-workspace/node_modules/@types/jasmine/ts3.1/index.d.ts(359)
 All declarations of 'message' must have identical modifiers.

Can we configure typedoc not to look at the Jasmine annotations?
Answer: Going up to the workspace project and removing @types/jasmine from package.json and running npm i fixes the problem.


How to interpret this population growth model?

Question: I was working on this equation $$\frac{d P}{d t}=\frac{v P}{K+P}-d P$$
where $P$ is population density and $v,d,K$ is constant, the stedy-state of this equation is $P_1=0$ or $P_2=\dfrac{v}{d}-K$.
If we let $\frac{d P}{d t}=f(P)$, and take derivative wrt $P$, we get $\frac{df}{dP}=\frac{v}{P+K}-\frac{v P}{(P+K)^{2}}-d$, plug the stedy-state, we have $\dfrac{df(P_1)}{dP}=\dfrac{v}{K}-d$ and $\dfrac{df(P_2)}{dP}=\dfrac{Kd^2}{v}-d$, what is the biological meaning of the two steay-state if one of them is stable in terms of the three constant.
And how to interpret what this equation saying, I mean, I don't know the meaning of the three constant and the derivation of this equation.
Comment: Just a complementary to Harry49's answer. $v$ is the maximal growth rate and $K$ is the half-saturation constant. This means when $P = K$, then the growth rate is $v/2$.
Comment: It is not quite exponential growth, see https://en.wikipedia.org/wiki/Michaelis%E2%80%93Menten_kinetics
Comment: Thank you, but I really don't know what half-saturation constant is, can I regard the first term of the equation as an exponential growth?
Answer: If we linearize the ODE in the vicinity of the origin, we have
$$
\frac{d P}{ dt} = v\, \frac{P/K}{1+P/K} - d P \underset{(P\to 0)}{\simeq} \left(\frac{v}{K} - d\right) P .
$$
The coefficient $r = v/K - d$ is the growth rate at origin, where $v/K$ may be viewed as a birth rate and $d$ as a mortality rate. Interpreting the coefficient $d>0$ as a mortality rate is common in various dynamical systems (such as the SIR epidemiology model, for instance).
One notes that $dP/dt$ vanishes at the origin $P=0$ and at $P=rK/d$, which are the steady states. Thus, by analogy with Verhulst's logistic growth model, we may rename the steady state $\kappa = rK/d$ as the carrying capacity. The dynamical system rewrites as
$$
\frac{d P}{ dt} = r P\, \frac{1 - P/\kappa}{1+ P/K} \, .
$$
One recognizes a modified logistic growth model, where the point of maximum growth is $P = \frac{\kappa}{1 + (1 + \kappa/K)^{1/2}}$. The corresponding instantaneous rates $dP/dt$ are represented below with respect to $P$ for several values of $K$ (and $r=1$, $\kappa = 1$). One notes that the curves are non-symmetric with respect to the maximum. Qualitatively, this model is similar to the $\theta$-logistic model.

The parameters $r$, $\kappa$ defined here are related to the original parameters $v$, $K$, $d$ through
$$
\begin{aligned}
r &= v/K - d \\
\kappa &= v/d - K
\end{aligned}
\qquad\text{i.e.}
\qquad
\begin{aligned}
v &= (\kappa + K)\, K\tfrac{r}{\kappa} \\
d &= K \tfrac{r}{\kappa}
\end{aligned}
$$

EDIT: Thanks to @Paichu for the comment mentioning Michaelis-Menten kinetics. Thus, the first term ${vP}/(K+P)$ of the growth rate is of Michaelis-Menten type (maximal growth rate $v$, half saturation $K$), and the second term $-dP$ is an additional mortality term. Similar terms are found in the Jacob-Monod predator-prey model.
Comment: The interpretation is fantastic! But do $v,d,K$ have any biological meaning?
Comment: @Cloud the parameter $d$ really acts as a *mortality rate*. However, it's hard to say what $v$ and $K$ mean, except that $v/K$ represents a birth rate... Maybe fertility and/or level of competition?


Checking if string exists in List C#

Question: I've been coding a program that stores student data (name and age) in a .txt file. I am now doing the delete method. However, when the user enters a string, I want it to compare the input to the strings in my List<string>, that's full of names. Code:
    string tempFileName;
    string inputSel; // Selection string for delete
    Console.WriteLine(" -- Deleting Grade {0} -- ", grade);
    Console.WriteLine("- Enter a student name to delete: ");
    foreach (string file in fileNames)
    {
        tempFileName = file.Replace(".txt", "");
        studentNames.Add(tempFileName);
    }
    foreach (string name in studentNames)
    {
        Console.Write("{0}\n", name);
    }
    Console.WriteLine();
    Console.Write("> ");
    inputSel = Console.ReadLine();
    string input = inputSel.ToLower();
    string tempString;
    bool foundString = false;
    foreach (string file in studentNames)
    {
        tempString = file.ToLower();
        if (inputSel == tempString)
        {
            foundString = true;
        }
    }
    if (!foundString)
    {
        Console.WriteLine("Wrong name entered!");
        Console.WriteLine("Returning to grades menu..");
        Console.WriteLine("Press any key to continue.");
        Console.ReadKey();
        return;
    }

As you can see, the program stores inputSel into input (ToLower()), then compares each string in studentNames List<string>, and if it finds a match it flips the foundString bool, but even if I enter a name that matches (for example, it says JacobMusterson, I enter JacobMusterson, it should skip the "student not found", but it doesn't.
Answer: you should use input not inputSel
if (input == tempString)
{
    foundString = true;
}

because of line:
string input = inputSel.ToLower();

where you are assinging to input lower version of inputSel
I suggest you to use IngonreCase in string.Compare for not making ToLower()
var b =  string.Compare("a","A",StringComparison.OrdinalIgnoreCase);

it will return 0 if equal see here
EDIT:
personally I would use:
var exists = studentNames.Any(x=>string.Compare(x,inputSel,StringComparison.OrdinalIgnoreCase)==0);

Comment: Wow, I can't believe I missed that. Thanks a lot! (Why can't I accept answers?)
Comment: @Ilan321 you can, but you have to wait several minutes, popup says that
Answer: you can simply do it this way:
Boolean foundString = studentNames.Exists(name => name.ToLower().Equals(input));

Answer: Why not simply use,
if(list.Contains(s)){
    //found
}else{
    //not found
}

Where list is List<String> and s is a String
Answer: It will be more readable and effective if you will use Contains method of List:
foreach (string file in studentNames)
    {
        tempString = file.ToLower();
        if (inputSel == tempString)
        {
            foundString = true;
        }
    }
if (!foundString)
    {
        Console.WriteLine("Wrong name entered!");
        Console.WriteLine("Returning to grades menu..");
        Console.WriteLine("Press any key to continue.");
        Console.ReadKey();
        return;
    }

can be rewrite:
if(!studentNames.Contains(inputSel, StringComparer.Create(CultureInfo.InvariantCulture, true)))
{
            Console.WriteLine("Wrong name entered!");
            Console.WriteLine("Returning to grades menu..");
            Console.WriteLine("Press any key to continue.");
            Console.ReadKey();
            return;
}

Answer: Just a small comment on your foreach-loop. You always iterating through all entries in the loop also if you already found out that the string is in the collection.
You can improve the performance of your code by replacing the last foreach.
Try:
bool foundString = studentNames.Select(file => file.ToLower()).Any(tempString => inputSel == tempString);

Checking if string exists in List C#


Is my evaluation scheme correct? model selection and algorithm evaluation

Question: I have a machine learning algorithm with some hyperparameters.
First, I split the data to 70% (A-set) and 30% (B-set).
Then, I used 5-fold cross-validation on the A-set to find the best hyperparameters.
Finally, I used 10-fold cross-validation on all data for reporting the performance of the algorithm.
Was my approach correct?
If yes, is there any reference for it?
Is my approach biased?
Thanks in advance.
Comment: Possible duplicate of [How to split the dataset for cross validation, learning curve, and final evaluation?](https://stats.stackexchange.com/questions/95797/how-to-split-the-dataset-for-cross-validation-learning-curve-and-final-evaluat)
Answer: This approach is not correct - hyperparameters are tuned on the data that is later used for evaluation.  A better approach would be to only test on the set B using 10 fold.
Comment: Or, instead of a single split into A and B sets, go for nested cross validation, i.e. wrap the 10-fold CV around the optimized model training with the 5-fold-CV inside.


Hello i want to know how can i use both java +kotlin both in flutter actually i want to use both languages to to create my own app in flutter

Question: public class My Question
{    public void main()
<code>    {
     System.out.println( "sorry but i could not post my question due to this body section problem"):
    }
</code>
}
Answer: Flutter apps are written with dart lang.
You can use Java/Kotlin and Obj-C/Swift for platform-specific code using Platfom Channel. But, your Java/Ktl code will run only on Android.
https://flutter.dev/docs/development/platform-integration/platform-channels
Flutter app code: dart.
Platform specific code Android: Java/Kotlin.
Platofrm specific code iOS: Obj-C/Swift.
You can't use Java/Kotlin for flutter app main code (.dart files).
Comment: so nice of you sir thanks  for solving my problem  you have saved my time


CloudFront "Compress Objects Automatically" and REST API

Question: I have a cloudfront distribution in front of a REST API server.  Just for now, I need to disable cache, so I'm using the Managed-CachingDisabled policy.
I can't get any responses to return as gzipped. I'm using cURL passing --compressed and the response has a Content-Length header and is between 1,000 and 10,000,000 bytes and is application/json.
Per the docs

Make sure the TTL values in the cache policy are set to a value higher than zero. When you set the TTL values to zero, caching is disabled and CloudFront doesn't compress content.

Do you really need to have caching enabled for "Compress Objects Automatically" to work?
Can I just use a policy that sets cache default to 0 and max 24h then have my origin (REST API) return a Cache-Control: max-age=0
Answer: I've confirmed this does work.  This is a showstopper for me however:

CloudFront is busy:
In rare cases, when a CloudFront edge location is unusually busy, some files might not be compressed.

In my testing I saw CF not compressing quite frequently


How to Insert a stored procedure into sql developer

Question: I'm trying to create an INSERT stored procedure in my database using Oracle SQL Developer but I can't seem to figure out what's the problem with this code;
create or replace procedure insert_order
as
BEGIN
    INSERT INTO ORDERS (ORDER_NUM, ORDER_DATE, CONDITION, STATUS, CUSTOMER_CUSTOMER_NUM, CUSTOMER_EMPLOYEE_NUM)
    VALUES(1, '30/OCT/2007', 'BRANDNEW', 'ORDERD', 103, 1);
end insert_order;

Can you help please?
Comment: And table definition is? Could you try using TO_DATE('30/OCT/2007')
Comment: What's the error it was throwing??
Comment: Error(4,3): PL/SQL: SQL Statement ignored, and
Error(5,23): PL/SQL: ORA-00984: column not allowed here
Answer: The error message PL/SQL: SQL Statement ignored, and Error(5,23): PL/SQL: ORA-00984: column not allowed here indicates that the error is with the value '30/OCT/2007' (line 5, column 23), since you are not using the standard format for dates.
You can try TO_DATE('30/OCT/2007') or '2007-10-30'.
Comment: Posting the solution that were added in the comments. Tagging it as community wiki following this recommendation: [link](http://meta.stackoverflow.com/questions/251597/question-with-no-answers-but-issue-solved-in-the-comments/251598#251598).


Get Record according to year range

Question: I want to select a result from student table-
query -
if I pass any year like 2013 then select the students join start date and end date or he is currently there
in between these year
like if any student start date year is 2010 and end date year is 2013
and I select 2011 year then show this student record also .
Table structure is-
StudentID   Age startDate   EndDate

1           14  5/05/2013   7/05/2013
4           17  4/04/2012   8/10/2012

and I'm trying this-
select * from tblstudent
Where  DATEPART(year, StartDate) BETWEEN @year-1 AND @year

Thanks in advance.
Comment: i do not understand your question. please phrase your question more clearly.
Answer: Not sure if I understood you, but this should do the job:
select *
from tblStudent s
where @year >= DATEPART(year, StartDate) and @year <= DATEPART(year, EndDate)

Answer: Actually, you almost there.
You just need to compare year part of your StartDate is smaller or equal to your @year and year part of your EndDate is bigger or equal to your @year.
Try like this;
select *
from tblStudent
where DATEPART(year, startDate) <= @year and DATEPART(year, EndDate) >= @year

For @year = 2013, output will be;
| STUDENTID | AGE |                  STARTDATE |                     ENDDATE |
------------------------------------------------------------------------------
|         1 |  14 | May, 05 2013 03:00:00+0000 | July, 05 2013 03:00:00+0000 |

Here SQL Fiddle Demo.

Get Record according to year range


.is(":animated") vs stop()

Question: So this code does not do what I expect it to do, the stops do nothing and it keeps animating giving strange results, (because the calculation of newpos is wrong.
if (direction =='left') {
    newpos = parseInt(currentpos) - 300;
    $('.slider').stop(true, true).animate({"margin-left": newpos}, 1000, "easeOutQuart");
}
else if (direction =='right') {
    newpos = parseInt(currentpos) + 300;
    $('.slider').stop(true, true).animate({"margin-left": newpos}, 1000, "easeOutQuart");
}

But this works
if (direction =='left' && !$('.slider').is(':animated')) {
    newpos = parseInt(currentpos) - 300;
    $('.slider').stop(true, true).animate({"margin-left": newpos}, 1000, "easeOutQuart");
}
else if (direction =='right' && !$('.slider').is(':animated')) {
    newpos = parseInt(currentpos) + 300;
    $('.slider').stop(true, true).animate({"margin-left": newpos}, 1000, "easeOutQuart");
}

The question is how can I do it without the is(':animated') in the conditional? Should stop() do the job in this case?
Comment: If you can create a jsfiddle that would be great
Answer: if (direction =='left') {
    newpos = parseInt(currentpos) - 300;
}
else if (direction =='right') {
    newpos = parseInt(currentpos) + 300;
}
$('.slider').animate({"margin-left": newpos}, 1000, "easeOutQuart");

Comment: you don't need to stop the animation, you need the animation to revert back from it's state so that you can do the reverse of the animation so i think you don't need the stop

.is(":animated") vs stop()


ASP Styling isn't working properly

Question: I'm working on an ASP site for the company I work at. It's about adding money saving or earning activities to an overview that one of the directors can look at.
Currently though I have some styling issues, using bootstrap here.

I believe I used to have this issue once but how I solved it escapes me. I can't center all the elements and I can't make the input boxes connect with the addons on the right side as is seen in the picture.

Abstract

                        FYE
                        <select class="form-control required" id="new-activity-fye-dropdown">
                            <option value="1">On Track</option>
                            <option value="2">Issue</option>
                            <option value="3">Behind</option>
                        </select>

                            ?

                        Activity Name
                        <input type="text" class="form-control required" id="new-activity-modal-name-field"
                            aria-describedby="new-activity-modal-name-field" />

                            ?

                        Responsible
                        <input type="text" class="form-control required" id="new-activity-modal-responsible-field"
                            aria-describedby="new-activity-modal-responsible-field" disabled />

                            ?

                        Department
                        <select class="form-control required" id="new-activity-modal-department-dropdown">

                        </select>

                            ?

                        Start Time
                        <input class="form-control" type="text" id="new-activity-modal-datepicker-start" />

                            ?

                        End Time
                        <input class="form-control" type="text" id="new-activity-modal-datepicker-end" />

                            ?

Description

                        Upload a thorough description explaining the Activity.

Estimation

                        AX Account Number(s)
                        <input type="text" class="form-control required" id="new-activity-modal-ax-account-numbers-field"
                            aria-describedby="new-activity-modal-ax-account-numbers-field" />

                            ?

                        Upload the estimation calculations for the Activity.

I feel that my HTML is fairly normal but I have a hard time applying styles from CSS files in ASP in the first place.
How would I make the addons on the right side connect with the input fields?
UPDATE
It appears that the problem was inside a default generated file called Site.css. It had the following in it:
input,
select,
textarea {
    max-width: 280px;
}

Commenting that out solves the problem.
Comment: This code displays correctly for me. You could try using ctrl+f5 to force a refresh in the browser, it may be caching old styles. If that doesn't work I would use the browser inspect element tool and look to see if you have overridden the bootstrap style anywhere.
Answer: Since I don't know how exactly ASP's css structure is, here is how i would approach this situation:
use your browser's inspector to check exactly what type of css rule is making the "?" button be all the way to the right, i can only assume it either has a float, or the form has a huge margin on it's right.
once you have found the issue, you could use your browser's inspector further to test out some edits, if it's a margin thing, see how much margin it will actually need, if it's a float issue, see how it will look with float:none
as soon as you've found the solution to the issue, you will need to apply it, to do this add a custom .css file AFTER ASPs css files, this way whatever you write in it will overwrite any existing rules.
add the fixes to this new file and you should be done.
Comment: I figured it out, though I'll comment up your answer for trying :)
Answer: There must be something elsewhere on the page that is breaking the style of the select boxes, because when I view your snippet in codepen, in both Chrome and IE11, the select boxes do stretch to the question mark boxes at the right:
codepen.io/anon/pen/pgVoQg
btw, inline styles should usually be avoided where possible
Comment: Yeah I usually don't apply inline styling but ASP styling is just weird to me. Directly applying css in separate files have varying results. I will try and put some more info up.
Comment: Okay I found the problem. Inside the default "Site.css" file is this line present: `input,
select,
textarea {
    max-width: 280px;
}` So commenting that out fixed it.
Comment: If you have css declared in multiple files the last file that is loaded in your references will take priority. If you want to make sure a certain style is used you can use the !important tag on a style.

ASP Styling isn't working properly


General linear hypothesis test statistic: equivalence of two expressions

Question: Assume a general linear model $y = X \beta + \epsilon$ with observations in an $n$-vector $y$, a $(n \times p)$-design matrix $X$ of rank $p$ for $p$ parameters in a $p$-vector $\beta$. A general linear hypothesis (GLH) about $q$ of these parameters ($q < p$) can be written as $\psi = C \beta$, where $C$ is a $(q \times p)$ matrix. An example for a GLH is the one-way ANOVA hypothesis where $C \beta = 0$ under the null.
The GLH-test uses a restricted model with design matrix $X_{r}$ where the $q$ parameters are set to 0, and the corresponding $q$ columns of $X$ are removed. The unrestricted model with design matrix $X_{u}$ makes no restrictions, and thus contains $q$ free parameters more - its parameters are a superset of those from the restricted model, and the columns of $X_{u}$ are a superset of those from $X_{r}$.
$P_{u} = X_{u}'(X_{u}'X_{u})^{-1} X'$ is the orthogonal projection onto subspace $V_{u}$ spanned by $X_{u}$, and analogously $P_{r}$ onto $V_{r}$. Then $V_{r} \subset V_{u}$. The parameter estimates of a model are $\hat{\beta} = X^{+} y = (X'X)^{-1} X' y$, the predictions are $\hat{y} = P y$, the residuals are $(I-P)y$, the sum of squared residuals SSE is $||e||^{2} = e'e = y'(I-P)y$, and the estimate for $\psi$ is $\hat{\psi} = C \hat{\beta}$. The difference $SSE_{r} - SSE_{u}$ is $y'(P_{u}-P_{r})y$. Now the univariate $F$ test statistic for a GLH that is familiar (and understandable) to me is:
$$
F = \frac{(SSE_{r} - SSE_{u}) / q}{\hat{\sigma}^{2}} = \frac{y' (P_{u} - P_{r}) y / q}{y^{t} (I - P_{u}) y / (n - p)}
$$
There's an equivalent form that I don't yet understand:
$$
F = \frac{(C \hat{\beta})' (C(X'X)^{-1}C')^{-1} (C \hat{\beta}) / q}{\hat{\sigma}^{2}}
$$
As a start
$$
\begin{array}{rcl}
(C \hat{\beta})' (C(X'X)^{-1}C')^{-1} (C \hat{\beta}) &=& (C (X'X)^{-1} X' y)' (C(X'X)^{-1}C')^{-1} (C (X'X)^{-1} X' y) \\
~ &=& y' X (X'X)^{-1} C' (C(X'X)^{-1}C')^{-1} C (X'X)^{-1} X' y
\end{array}
$$

How do I see that $P_{u} - P_{r} = X (X'X)^{-1} C' (C(X'X)^{-1}C')^{-1} C (X'X)^{-1} X'$?
What is the explanation for / motivation behind the numerator of the 2nd test statistic? - I can see that $C(X'X)^{-1}C'$ is $V(C \hat{\beta}) / \sigma^{2} = (\sigma^{2} C(X'X)^{-1}C') / \sigma^{2}$, but I can't put these pieces together.
Answer: For your second question, you have $\mathbf{y}\sim N(\mathbf{X}\boldsymbol{\beta},\sigma^2 \mathbf{I})$ and suppose you're testing $\mathbf{C}\boldsymbol{\beta}=\mathbf{0}$.  So, we have that (the following is all shown through matrix algebra and properties of the normal distribution -- I'm happy to walk through any of these details)
$
\mathbf{C}\hat{\boldsymbol{\beta}}\sim N(\mathbf{0}, \sigma^2 \mathbf{C(X'X)^{-1}C'}).
$
And so,
$
\textrm{Cov}(\mathbf{C}\hat{\boldsymbol{\beta}})=\sigma^2 \mathbf{C(X'X)^{-1}C}.
$
which leads to noting that
$
F_1 = \frac{(\mathbf{C}\hat{\boldsymbol{\beta}})'[\mathbf{C(X'X)^{-1}C'}]^{-1}\mathbf{C}\hat{\boldsymbol{\beta}}}{\sigma^2}\sim \chi^2 \left(q\right).
$
You get the above result because $F_1$ is a quadratic form and by invoking a certain theorem.  This theorem states that if $\mathbf{x}\sim N(\boldsymbol{\mu}, \boldsymbol{\Sigma})$, then $\mathbf{x'Ax}\sim \chi^2 (r,p)$, where $r=\textrm{rank}(A)$ and $p=\frac{1}{2}\boldsymbol{\mu}'\mathbf{A}\boldsymbol{\mu}$, iff $\mathbf{A}\boldsymbol{\Sigma}$ is idempotent.  [The proof of this theorem is a bit long and tedious, but it's doable. Hint: use the moment generating function of $\mathbf{x'Ax}$].
So, since $\mathbf{C}\hat{\boldsymbol{\beta}}$ is normally distributed, and the numerator of $F_1$ is a quadratic form involving $\mathbf{C}\hat{\boldsymbol{\beta}}$, we can use the above theorem (after proving the idempotent part).
Then,
$
F_2 = \frac{\mathbf{y}'[\mathbf{I} - \mathbf{X(X'X)^{-1}X'}]\mathbf{y}}{\sigma^2}\sim \chi^2(n-p-1)
$
Through some tedious details, you can show that $F_1$ and $F_2$ are independent.  And from there you should be able to justify your second $F$ statistic.
Comment: @caracal Sure.  I edited my response to add in some details.
Comment: Thanks for your fast reply! Could you please explain the "which leads to noting that" step to $F_{1}$ a little bit further? That's the one I'm not getting...
Comment: I'll accept this answer, but I'd still be very happy about an answer to my first question as well - literature tips are of course welcome!
Comment: Related: https://stats.stackexchange.com/q/188626/119261.
Answer: Since nobody has done so yet, I will address your first question. I also could not find a reference for [a proof of] this result anywhere, so if anyone knows a reference please let us know.
The most general test that this $F$-test can handle is $H_0 : C \beta = \psi$ for some $q \times p$ matrix $C$ and $q$-vector $\psi$. This allows you to test hypotheses like $H_0 : \beta_1 + \beta_2 = \beta_3 + 4$.
However, it seems you are focusing on testing hypotheses like $H_0 : \beta_2 = \beta_4 = \beta_5 = 0$, which is a special case with $\psi=0$ and $C$ being a matrix with one $1$ in each row, and all other entries being $0$. This allows you to more concretely view the smaller model as obtained by simply dropping some columns of your design matrix (i.e. going from $X_u$ to $X_r$), but in the end the result you are seeking is in terms of an abstract $C$ anyway.
Since it happens to be true that the formula $(C\hat{\beta})' (C (X'X)^{-1} C') (C \hat{\beta})$ works for arbitrary $C$ and $\psi=0$, I will prove it in that level of generality. Then you can consider your situation as a special case, as described in the previous paragraph.
If $\psi \ne 0$, the formula needs to be modified to
$(C\hat{\beta} - \psi)' (C (X'X)^{-1} C') (C \hat{\beta} - \psi)$,
which I also prove at the end of this post.

First I consider the case $\psi=0$.
I will try to keep some of your notation. Let $V_u = \text{colspace}(X) = \{X\beta : \beta \in \mathbb{R}^p\}$.
Let $V_r := \{X\beta : C\beta=0\}$.
(This would be the column space of your $X_r$ in your special case.)
Let $P_u$ and $P_r$ be the projections on these two subspaces.
As you noted, $P_u y$ and $P_r y$ are the predictions under the full model and the null model respectively. Moreover, you can show $\|(P_u - P_r) y\|^2$ is the difference in the sum of squares of residuals.
Let $V_l$ be the orthogonal complement of $V_r$ when viewed as a subspace of $V_u$. (In your special case, $V_l$ would be the span of the columns of the removed columns of $X_u$.)
Then $V_r \oplus V_l = V_u$, and moreover,
In particular, if $P_l$ is the projection onto $V_l$, then $P_u = P_r + P_l$.
Thus, the difference in the sum of squares of residuals is
$$\|P_l y\|^2.$$
If $\tilde{X}$ is a matrix whose columns span $V_l$, then
$P_l = \tilde{X} (\tilde{X}'\tilde{X})^{-1} \tilde{X}'$ and thus
$$\|P_l y\|^2 = y'\tilde{X} (\tilde{X}'\tilde{X})^{-1} \tilde{X}' y.$$
In view of your attempt at the bottom of your post,
all we have to do is show that choosing $\tilde{X} := X(X'X)^{-1} C'$ works, i.e., that $V_l$ is the span of this matrix's columns.
Then that will conclude the proof.

It is clear that $\text{colspace}(\tilde{X}) \subseteq \text{colspace}(X)=V_u$.
Moreover, if $v \in V_r$ then it is of the form $v=X\beta$ with $C\beta=0$,
and thus $v' \tilde{X} = \beta' X' X (X'X)^{-1} C' = (C \beta)' = 0$, which shows $\text{colspace}(\tilde{X})$ is in the orthogonal complement of $V_r$, i.e. $\text{colspace}(\tilde{X}) \subseteq V_l$.
Finally, suppose  $X\beta \in V_l$. Then $(X\beta)'(X\beta_0)=0$ for any $\beta_0 \in \text{nullspace}(C)$. This implies $X'X\beta \in \text{nullspace}(C)^\perp = \text{colspace}(C')$, so $X'X\beta=C'v$ for some $v$. Then, $X(X'X)^{-1} C' v = X\beta$, which shows $V_l \subseteq \text{colspace}(\tilde{X})$.

The more general case $\psi \ne 0$ can be obtained by slight modifications to the above proof.
The fit of the restricted model would just be the projection $\tilde{P}_r$
onto the affine space $\tilde{V}_r = \{X \beta : C \beta = \psi\}$, instead of the projection $P_r$ onto the subspace $V_r =\{X\beta : C \beta = 0\}$. The two are quite related however, as one can write $\tilde{V}_r = V_r + \{X \beta_1\}$, where $\beta_1$ is an arbitrarily chosen vector satisfying $C \beta_1 = \psi$, and thus
$$\tilde{P}_r y = P_r(y - X\beta_1) + X \beta_1.$$
Then, using the fact that $P_u X \beta_1 = X \beta_1$,
we have $$(P_u - \tilde{P}_r) y = P_u y - P_r(y - X \beta_1) - X \beta_1 = (P_u - P_r)(y - X\beta_1) = P_l(y - X \beta_1).$$
Recalling $P_l = \tilde{X} (\tilde{X}'\tilde{X})^{-1} \tilde{X}'$ with $\tilde{X} = X(X'X)^{-1} C'$,
the difference in sum of squares of residuals can be shown to be
$$(y - X \beta_1)' P_l (y - X \beta_1) = (C \hat{\beta} - \psi)'(C (X'X)^{-1} C') (C\hat{\beta} - \psi).$$


model overfitting vs applicability

Question: Consider two models I built:
Model A

I use a Neural Network to build a classification model and get a
model that over fits , lets say the FPR in test set in 2 times that
in train set.
I am comfortable with the FPR on test set for my business purpose
and I also check that the model test performance is reasonably stable over many other random samples of train vs Test

Model B

I build a L1 Logistic model with a lot of features and spend a lot of
time feature engineering to capture variable interactions.
The train and test FPR are within 20% of each other for some optimal parameter, BUT the FPR here is not as good as the test FPR in Model A.

Note that the same Train and test samples are used for both models.
How much weight do I need to give to overfitting Vs prediction accuracy?
Answer: It is a common misconception that a model is overfit when the training and hold out error metrics are divergent.  This does tend to happen when models become overfit, but it is not a sufficient condition.  A better definition of overfitting is the following

A model is overfit when a (small) decrease in model complexity results in an increase in hold out performance.

Models with quite different training and hold out performance can be underfit (a small increase in complexity improves hold out performance).
So for your case, where you want all and only the model with the best hold out performance, you should focus on this aim.  If you are confident in your estimate of the hold out performance of your neural network, and there are no barriers to using it in your problem domain, go with that.

model overfitting vs applicability


Sax practice routine for jazz

Question: Hi I've been playing tenor for around 6m now, former pianist to semi-pro standard so I have a strong ear and a decent idea of what I want from my sound.
My current practice routine consists of:

Long tones (10 mins)
Technical exercises (octaves, 1-2 articulation, groups of 3 etc, 5 mins)
Scales/ arpeggios, slurred and tounged (10 mins, rotating through keys each day)
Play patterns from several books (20 mins, working through a few of the Chad LB books as well as Jay Metcalfe's "pentatonic patterns for improvisation")
Improvise over some 2-5-1s (15 mins)
Play along over some (non-bebop) tune on the radio (recently alot of african music and soca, 5-10 mins)
Work on a transcription or maybe play from the omnibook (whatever time I have left)

Any thoughts would be appreciated. Have been amazed at how much my sound has come along in such a short time, but I guess that's daily practice for you!
Comment: Sounds like you're doing fine, man
Answer: It's great that you are practising, and welcome to the saxophone club! But I'm not sure what exactly your question is. Oh, and don't get surprised if you reach a plateau in your sound development in the next 6 months. But you should be familiar with that from playing piano.
If you haven't upgraded your reed strength, now is probably the time. If you had started out with 1.5-2, like almost everyone else, you should move on to 2.5-3 now. If you still have any unused soft reeds left, you can donate them to a school or to someone who is starting out on the saxophone.
Your entire routine amounts to 70+ minutes, and you don't specify how you break that up (if at all). I've had 2 saxophone teachers (and most of the information from this post is going to come from what they have taught me), and both recommended not practising for more than 25-30 minutes without at least a 5-minute break. The last teacher insisted on it so much that he asked me to use a timer for each practice session (that's what he does himself). Once the timer rings/beeps, put down your instrument and move to something else (ideally, not related to music). If you love the instrument, it gets all too easy to get stuck in working on a particular concept or piece. I personally intersperse my saxophone and flute practice with physical exercise (push-ups, pull-ups and so on), but anything works, really.
Do you keep a practice diary, or do you just go through your exercises randomly? Do you start and finish practising around the same time each day? Those things are important too. Especially as you move along and more exercises and scales get added to your repertoire.
How do you practise long tones? It's a boring exercise that is best combined with something else (in general, at least the more you practise, it is best to combine practising 2 or more things within the same bit). I usually practise long tones by going through major triads (the pattern is: C-A-F-A-C [C is the 5th]; C-G#-D#-G#-C [C is the 3rd]; C-G-E-G-C [C is the root]; you can do the same with minor triads).
By the looks of it, you focus almost exclusively on jazz, and I don't think that's enough. But it's probably a personal choice.
Another thing that seems to be missing from your practice routine is overtone exercises, which are important for developing a consistent sound and eventually being able to play in the altissimo register. The way I was taught to do them was to play overtones on the 4 low notes (C#, C, B and B♭; if you have a baritone, you can add the low A to the list) and to try to match the sound of regular fingerings to the sound of those overtones as much as possible (not the other way 'round!). So, for example, your regular middle B♭ fingering should sound like the 1st overtone on low B♭. But check that your low end is actually in tune before doing that. Most saxophone models (especially "student" ones) are notorious for how ridiculously bad and out-of-tune the low end is.
Yet another exercise recommended to me by my second teacher is note-bending. The idea is to start from the top of the regular range (high F, even if you have a high F# key) and try to progress as far as you can downwards without changing the fingering. So, for example, finger a high F and try to play a high E without changing the fingering, then high E♭ and so on.
You don't mention metronome or legato tonguing either. Do you practise your scales at different subdivisions of the beat with a metronome? Legato tonguing is a necessity in jazz saxophone playing as well (and a somewhat difficult thing to describe). Basically, the idea is to slightly tongue every second note without fully stopping the airflow. The reason for that is that you can't really play "jazz triplets" on higher bpm settings, and pretty much the only way to give notes a "jazz feel" is to slightly tongue the note that'd normally be a triplet-8th-long following a 2-triplet-8th-long note.
You might also want to add some "advanced" techniques to your practice at a later stage, like growling, flutter-tongue and multiphonics.
You've also tagged your question with bebop. Are you familiar with the common bebop scales, how they are normally played and the music theory behind it? For example, the bebop major scale is normally played ascending, and the bebop dominant scale - descending.
Comment: First of all, thanks a lot for such a detailed answer, and for excellent suggestions. Apologies for the lack of clarity on a few points. I believe what you refer to as legato tounging is what I call 1-2 articulation. Yes, I am pretty religious about practising with a metronome and also mix it up between 4 beats, 1&3,  2&4, just 4 etc. Overtones is something my teacher has just got me working on, as well as vibrato, and I do tend to break up my routine to fit around my day. I'm very curious to know what you meant by 'jazz is not enough'? What else do you think I should look at? Cheers!
Comment: Also should mention that I do use a tuner for the long tones - I recently upgraded from a beginner horn as well and noticed a massive difference.
Comment: @Michaelborn, I hope it was helpful. What you refer to as 1-2 articulation is not exactly the same (provided I understand what you mean by it). Playing the 1st note legato and the 2nd tongued is slightly different compared to playing all notes legato and tonguing every 2nd note. With the latter, there are no stops in the airflow (except for breathing). When you tongue a note normally, you fully push the reed against the mouthpiece, closing the air vent. With "legato tonguing" you just slightly close it (like a door left slightly ajar).
Comment: @Michaelborn, if you intend to play jazz (or anything besides classical, really), you will eventually want to improvise, and your phrasing will be a lot more interesting if you also invest some time into classical music (very counterintuitive, I know) and rock. I'm personally a huge fan of involving woodwinds in metal music as well. Diminished scales sound surprisingly sexy in some jazz. The difference between a new horn and an older one could also be the state of the pads, but I hope your new instrument will continue working better for you!

Sax practice routine for jazz


RoR: Controller Submit with params

Question: I'm writing an entry submission controller by this tutorial: http://www.communityguides.eu/articles/1
When I try to submit an entry, I get Users can't be blank...I don't want to pass the user_id in as a hidden field, right? So how should I change this to automatically get the user's id?
I'm using devise for authentication. :) And I'm a complete rails novice.
This is my controller:
    def submit
      @entry = current_user.articles.find(params[:id])

      # submit only, if article is currently in draft or rejected-state
      if (@entry.state == 0) or (@article.state == 2)
        @entry.state = 1
        @entry.submitted = Time.now

        if @entry.save
          flash[:notice] = 'Your article was successfully submitted for approval.'
        else
          flash[:error] = 'There was an error while submitting your article.'
        end
      else
        flash[:error] = 'This article can not be submitted.'
      end

      respond_to do |format|
        format.html { redirect_to(:action => 'myarticles') }
        format.xml  { head :ok }
      end
    end
  # GET /entries/1/edit
  def edit
    @entry = Entry.find(params[:id])
  end

  # POST /entries
  # POST /entries.xml
  def create
    @entry = Entry.new(params[:entry])

    respond_to do |format|
      if @entry.save
        format.html { redirect_to(@entry, :notice => 'Entry was successfully created.') }
        format.xml  { render :xml => @entry, :status => :created, :location => @entry }
      else
        format.html { render :action => "new" }
        format.xml  { render :xml => @entry.errors, :status => :unprocessable_entity }
      end
    end
  end

  # PUT /entries/1
  # PUT /entries/1.xml
  def update
    @entry = current_user.entries.find(params[:id])

        #if the entry has been approved, the user cannot change the title or URL.
        if @entry.state > 2
            params[:entry].delete(:title)
            params[:entry].delete(:url)
        end
    respond_to do |format|
      if @entry.update_attributes(params[:entry])
        format.html { redirect_to(@entry, :notice => 'Entry was successfully updated.') }
        format.xml  { head :ok }
      else
        format.html { render :action => "edit" }
        format.xml  { render :xml => @entry.errors, :status => :unprocessable_entity }
      end
    end
  end

Answer: From your description it seems that you would encounter this error when trying to create a new Entry.
  def create
    @entry = current_user.entries.new(params[:entry]) # <-- Scope to the current user

    respond_to do |format|
      if @entry.save
        format.html { redirect_to(@entry, :notice => 'Entry was successfully created.') }
        format.xml  { render :xml => @entry, :status => :created, :location => @entry }
      else
        format.html { render :action => "new" }
        format.xml  { render :xml => @entry.errors, :status => :unprocessable_entity }
      end
    end
  end

Comment: `Entry.find(params[:id])` can find any Entry record in your database,  but quite often you want users to only access records that belong to them. In your edit action above any user could edit any Entry if they knew (or guessed) the id.  It is common practice to prevent this problem by limiting the find to only return Entries that belong to the current user.  `current_user.entries.find(params[:id])` does just that by adding a WHERE condition to the SQL query like `WHERE entries.user_id=123`
Comment: Also, you'll want to do the same thing for the edit action.  `@entry = current_user.entries.find(params[:id])`  So that a user cannot edit another user's entries.
Comment: Alrightie! Thanks. :)  Would you give a little more explanation for me?  Like I said, I'm new--I don't exactly understand what you mean by `scope to`...

RoR: Controller Submit with params


fit a line of the form $y=ax$ through those points

Question: Suppose that some experiment produces the following data points:
$$(1,5),(2,5),(3,10),(4,13),(5,9)$$
We want to fit a line of the form $y=ax$ through those points. The error  (which of course depends on the choice of $a$ ) that our fit makes at every input location $x$, is measured by the following error criterion: $$((ax−y)^2)⋅\log(((ax−y)^2)+1),$$
where obviously the $\log$ is the natural logarithm.
Now, let the function $E:a↦E(a)$ be the total error overall points, depending on the parameter $a$.
Find the minimum value that $E$  attains, to three decimals accuracy behind the decimal point:
Comment: Looks like homework. What have you tried?
Comment: i am not sure where to start with it, maybe take the derivative of the function and substitute the points into it?
Answer: If you use the normal least square fit, you minimize
$$F(a)=\sum_{i=1}^5 (ax_i-y_i)^2$$ which corresponds to the linear regression with no intercept. So, for this case, the formula is
$$a=\frac{\sum_{i=1}^5 x_i y_i}{\sum_{i=1}^5 x_i^2}=\frac{142}{55}\approx 2.58182$$ and this is more than likely a good estimate for the minimization of
$$G(a)=\sum_{i=1}^5 (ax_i-y_i)^2\log\big((ax_i-y_i)^2+1 \big)$$ Since you want to minimize $G$, you need its derivative equal to zero. Differentiating, we have
$$H(a)=\frac {dG(a)}{da}=\sum_{i=1}^5 \Big( \frac{2 x_i (a x_i-y_i)^3}{(a x_i-y_i)^2+1}+2 x_i (a x_i-y_i) \log \left((a x_i-y_i)^2+1\right)\Big)$$ for which you need a numerical method such as Newton.
The iterates being given by
$$a_{n+1}=a_n-\frac{H(a_n)}{H'(a_n)}$$ If you are patient you could establish the formula for $H'(a)$ but this would be quite tedious. So, use numerical derivatives and write
$$a_{n+1}=a_n-\frac{2\,\epsilon\, H(a_n)}{H(a_n+\epsilon)-H(a_n-\epsilon)}$$ with, for example $\epsilon=10^{-3}a_n$.
Using $a_0=\frac{142}{55}$, the first iterate will be $a_1=2.52038$ and continuing
$$\left(
\begin{array}{cc}
 n & a_n \\
 0 & 2.58182 \\
 1 & 2.52038 \\
 2 & 2.52035
\end{array}
\right)$$ which clearly shows that, for the resuired accuracy, one single iteration is more that sufficient.


Proving that the multiplicative group modulo $2^r$ are cyclic iff $r<3$

Question: No idea how to start, can someone give me a hint?
Show that $(\mathbb{Z}/2^r\mathbb{Z})^*$ is a cyclic group if and only if $r<3$.
Answer: Hint: It is easy to verify directly that the group is cyclic if $r=0$, $1$, or $2$.
To show the group is not cyclic if $r\ge 3$, note that in a cyclic group the equation $x^2=1$ has at most $2$ solutions. But if $r\ge 3$, then the congruence $x^2\equiv 1\pmod{2^r}$ has, beside the two obvious solutions, solutions $x\equiv 2^{r-1}\pm 1\pmod{2^r}$.
Added: We show that if a finite group $G$ is cyclic, then the equation $x^2=1$, where $1$ is the identity element, has at most two solutions.
Let $G$ have $n$ elements, and let $g$ be a generator of $G$. So every element of $G$ is of the shape $g^k$, where $1\le k\le n$, and $g^n=1$.
Suppose that $x^2=1$. Then $x=g^t$ for some $r$ between $1$ and $n$, and $g^{2t}=1$. It follows that $2t$ divides $n$. If $t\ne n$, then $2\le 2t\lt 2n$. Thus we must have $2t=n$. We conclude that there is at most one solution of $x^2=1$ other than the trivial solution $x=1$.


^= operator, what does it do?

Question: I just saw a code where operator ^= was used in it.
What does it do?
It was used like this:
number^=arr[m];
Wher both number and arr[m] are integers.
I would search for it, but it looks impossible to search for ^= .
Comment: I'm voting to close this question as off-topic because there's enough information in the official documentation
Comment: Take a look at the [^ operator](https://msdn.microsoft.com/en-us/library/zkacc7k1.aspx)
Comment: You can read the documentation for the [`^=` operator](https://msdn.microsoft.com/en-us/library/0zbsw2z6.aspx) on MSDN.
Comment: @farshad -- you may be a beginner to coding, but don't you know how to google?
Comment: Even beginners are expected to spend some effort searching the documentation.  As D Stanley noted, the first result is a list of all operators.  It wasn't hard to find.
Comment: @farshad don't take it so hard. 2 upvotes on a question can make up for 5 downvotes.  Plus the point is for you to get the answer, not just to get points, right? :)
Comment: @farshad Don't sweat it then.  Looks like you got your answer and a net of 4 points so far :)
Comment: I understand, but there was no need to down vote. I just couldn't search for ^ on the internet so I asked it here.
Comment: You can browse the documentation and look for the list of operators
Comment: Down voted due to lack of research effort.
Comment: @farshad No but if you _do_ search for "C# ^= operator" the first result is the MSDN page that lists ALL operators, including `^=`.  It shouldn't have been that hard to find,
Comment: OK. You guys are so tough to beginners here. I'm simply not used to search document. learnt about it just now.
Comment: well, I agree I had to search more. Just because ^ wasn't in the preview I didn't open the google results. Can anyone delete this post?
Comment: @farshad -- if you hover your cursor over the down vote button, the very first reason listed is "this question does not show any research effort"
Comment: @farshad There's should be a "delete" link under the post if you want to delete it yourself; otherwise it takes a moderator to delete it.
Comment: @DStanley now it has an answer, it won't let me to delete is.
Comment: Yeah, You're write. Sorry to bother you guys. I'll promise to search harder next time.
Answer: x ^= y

is the short version of
x = x ^ y

and is performing bitwise XOR operation.
Comment: @Amit.  For integral arguments it is.
Comment: It's not necessarily **bitwise**.
Comment: @DStanley - is there anything in this entire post that says it's about integral types?
Comment: @Amit yes. It says "number and arr[m] are integers."
Comment: @Amit I know, i just hoped he doesn't ask about bool parameters. At least he stated so.
Comment: @DStanley - ha! I didn't notice that. Sorry.

^= operator, what does it do?