AR Forecasting one day ahead

Question: I am doing a project where I want to forecast prices for the next day based on some lagged values of the price. When I read about how to implement this (in Python) what I find it that they use a training data set, fit the model, and then use these parameters to forecast a test data set.
For example:
#first until n-5 observation
df_train = df['Temp'].iloc[:-5]
#n-5 until n observation
df_test = df['Temp'].iloc[-5:]

from statsmodels.tsa.ar_model import AutoReg
model = AutoReg(df_train, lags=22).fit()

forecasts = model.forecast(5).tolist()
test_values = df_test.tolist()
for index in range(len(forecasts)):
  difference = forecasts[index] - test_values[index]
  print(forecasts[index], test_values[index], difference)

Source
First, I see that long training sets are generally used, but then the model uses for example 22 lags to predict the test set, like here. In this case, wouldn't it be sufficient to use this training set:
df_train = df['Temp'].iloc[-27:-5]

Which would give 22 observations?
Secondly, in my case, I don't want to use a fixed training set to forecast a set number of days in the future, but I want to use my lagged data to predict the next 24 hours.
Say if I use a 7 days lag and the first forecasting day is n. Then, to predict n, I want to use observations n-8 until n-1. For the next day, n+1, I want to use n-7 until n to predict.
So:

Do I even need a training and test split for this forecast?
Are you aware of some sources where I can read about this procedure?

All the best,
Simon
Comment: Sidenote (not related to answering the question): Arima can model a correlation structure based on which you make predictions, but does it also make sense for your case? Does the timeseries seem to behave like a more or less constant arima process or can there be peculiar fluctuations and variations (related to particular events in the market)? With a neural network you might pick up on more details than just extrapolating values from the previous 22 days.
Comment: Thank you @SextusEmpiricus. I realize the code I attached was confusing, I am not using 22 lags in my model, that was just an example on how AR forecasting are described in the articles I have found. The benchmark I am using use 1, 2 and 7 lags, plus min/max value from the previous day and daily dummies. I will try to beat this benchmark later with a neural network model, among others.
Answer: +1 to Richard's answer. A few additions:
First off, as Richard notes, you are using the full training set to estimate the autoregression coefficients, which is what AutoReg does. In forecasting, you would only use the last 22 (or whatever your order is) observations together with the estimated coefficients.
This is completely standard autoregressive modeling, which is a subset of the more general autoregressive integrated moving average (ARIMA) class of models. You can read all about this in this excellent online textbook.
Importantly, if your goal is actual forecasting, it makes a lot more sense to use an automated ARIMA modeler. This will decide on the optimal AR order, but also on the number of differences and the MA order, and estimate the coefficients.
Any decent auto-ARIMA modeler will likely end up with far fewer than 22 lags. (Seasonality may come in, depending on what prices you are modeling. If this makes sense to you, be sure to allow seasonality.) See this thread and this one.
Comment: @StephanKolassa the example from the link simply looked at the ACF plot and decided on 22 lags because the correlation with a lag of 23 was the first to get below a significant level. But possibly an AR(1) model might be sufficient, that also can give a significant correlation for a lag of 22. The author took as guideline, 'significant autocorrelation means that the coefficient must be non-zero'.
Comment: @SextusEmpiricus: thanks, good find. For the benefit of others, the argument "my last lag with an autocorrelation above the significance threshold is 22, *therefore* I will fit an AR(22) model" is false. See the bottom of [this page](https://otexts.com/fpp3/non-seasonal-arima.html) on how to select AR/MA orders based on ACF/PACF plots, which is the ancient Box-Jenkins methodology and which we keep on getting questions on here. Honestly, why anyone uses this approach rather than information criteria in 2022 is beyond me. [A question on this here.](https://stats.stackexchange.com/q/595150/1352)
Comment: Thank you for the additions. I will check that book out as I think I may not understand the essence of the estimation of coefficients in the AR, as mentioned in the comment on Richard's answer. In this case, I am estimating a benchmark model so I have a set number of lags that I have to incorporate. But thank you for those sources, I will use them when I will develop my own model later.
Comment: Hm. If you are actually working on hourly data, then a seasonal model with a seasonal period of length 24 may be indicated, and make much more sense than a model with 22 lags - I have never come across a model with this lag structure before. If your benchmark uses 22 lags (but possibly no seasonality), then I would seriously question the appropriateness of that benchmark.
Comment: To clarify, this example is nothing that I am using (nor the benchmark), it is just an example of how the AR forecasting models that I have found have been described. That is, estimating coefficients once, and then forecasting. Compared to what (I think/guess) I want to do, re-estimating the coefficients every day.
Comment: Great points here!
Answer:
First, I see that long training sets are generally used, but then the model uses for example 22 lags to predict the test set, like here. In this case, wouldn't it be sufficient to use this training set: df_train = df['Temp'].iloc[-27:-5] which would give 22 observations?

What you are suggesting is to use 22 data points* for estimating a model with 22 slope coefficients and an intercept. That is guaranteed to yield extreme overfitting. You would not even be able to use OLS as there are more free parameters than data points. You need a much larger training subsample.

I want to use my lagged data to predict the next 24 hours

If you have daily data, you cannot make hourly predictions. On the other hand, you probably meant forecasting 1 day ahead? That would work.

Say if I use a 7 days lag and the first forecasting day is n. Then, to predict n, I want to use observations n-8 until n-1. For the next day, n+1, I want to use n-7 until n to predict. So: Do I even need a training and test split for this forecast?

You need a training sample to estimate the model's coefficients. Having done that, you only need the most recent $p$ data points to predict the next one. (Here, $p$ is the order of the AR model you are using.)
*I am not proficient in Python, but does [-27:-5] not yield 23 data points?
Comment: @SimonRydstedt, thanks!
Comment: - My mistake, should've been [-26-5]
- It may be misleading as the example uses daily observations, but I do have hourly observations in my set. - Do you know a source where I can understand the estimation of coefficients because, in my head, a p-order AR model only uses the last p observations to predict data. And therefore older data is not necessary.
Comment: @SimonRydstedt, a model that has already been estimated does not require more than the last $p$ data points to be able to output predictions of future values. However, you need to estimate the model's coefficients first. For that, you need a larger (training) sample.

AR Forecasting one day ahead


Swap-Idiom of abstract class with private data members

Question: Assume we have this two classes, with implemented swap-idioms. The copy constructor and assignment operator of the base class are deleted, as it makes no sense. However the swap-method is implemented, as it holds a member.
namespace std
{
    template<>
    void swap<Base>(Base& a, Base& b)
    {
        a.swap(b);
    }

    template<>
    void swap<Derived>(Derived& a, Derived& b)
    {
        a.swap(b);
    }
}

class Base
{
public:
    Base(int ID) : ID_(ID) {}
    virtual std::string getString()=0;

    Base(const Base&)=delete;
    operator=(const Base&)=delete;

    void swap(Base& rhs)
    {
        std:swap(ID_, rhs.ID_);
    }
private:
    int ID_;
}

class Derived : public Base
{
public:
    Derived(int ID, bool Value) : Base(ID), Value_(Value) {}
    virtual ~Derived() {}
    Derived(Derived& rhs)
    {
        std::swap(*this, rhs);
    }

    virtual std::string getString() {return Value_ ? "True" : "False"}
    void swap(Derived& lhs, Derived& rhs)
    {
        std::swap(static_cast<Base&>(lhs), static_cast<Base&>(rhs);
        std::swap(lhs.Value_, rhs.Value_);
    }
private:
    bool Value_;
}

As seen in many examples, this would be the standard way to do it I suppose.
However, I see a problem with the public Base::swap, as it should not be possible to swap only the abstract base-class!
Wouldn't it be better to remove the template for the Base class and make the Base::swap Method protected:
class Base
{
    ...
protected:
    void swap(Base& rhs, Base &lhs);
}

class Derived : public Base
{
    ...
public:
    void swap(Derived& lhs, Derived& rhs)
    {
        Base::swap(static_cast<Base&>(lhs), static_cast<Base&>(rhs);
        std::swap(lhs.Value_, rhs.Value_);
    }
}

Assuming, there is another class derived from base, with the first implementation it would be possible to swap the ID, however the data members of the derived objects would stay the same.
So, am I right thinking that swapping of a abstract class should not be possible from outside?
Comment: Note: If you define a swap function for your class, you should declare it as an overload in the namespace in which your class is declared, not a specialization of `std::swap`.
Comment: See http://stackoverflow.com/questions/11562/how-to-overload-stdswap
Comment: You are right in that private datamembers of a class should only be accessed by a swap that is allowed to access those data members...
Answer:
The copy constructor and assignment operator of the base class are deleted, as it makes no sense.

Actually, this is terrible. Now you made Derived uncopyable! Why? There is no reason to add this restriction. The default copy constructor and assignment operator of Base are perfectly reasonable in the context of copying the most-base class of the hierarchy.
Once you undo that, there's no need to do anything else as std::swap(derived1, derived2) would already do the right thing. The default move construction/operation is correct. It's always good to let the compiler do things for you.
But if you want to override swap anyway, the correct way to do that would be as a non-member friend:
class Base {
   ...
   friend void swap(Base& lhs, Base& rhs) {
       using std::swap;
       swap(lhs.ID_, rhs.ID_);
   }
};

class Derived : public Base {
   ...
   friend void swap(Derived& lhs, Derived& rhs) {
       using std::swap;
       swap(static_cast<Base&>(lhs), static_cast<Base&>(rhs));
       swap(lhs.Value_, rhs.Value_);
   }
};

Also, your Derived copy constructor makes no sense. Remove it as per the first paragraph of my answer.
Comment: Obviously you are right about the non-member friend, thanks.
However, think about another derived class, holding a int instead of a bool, and the ID somehow mapped to a database table holding generic values, it would certainly _not_ be reasonable to be able to modify the Base information without information about the derived class.
Comment: @Vallant I don't understand what you're objecting to.
Comment: Useful to note that `static_cast` only works in the free standing swap function and not in `Derived::swap(Derived& rhs)`


Neural language model training - stochastic vs batch

Question: Dealing with a very basic neural language model: 3 words of context, vector size 100, one hidden layer size 200, vocabulary size 1000, predicting the next word with a softmax output layer.
Previously I have trained similar models with a stochastic (online) gradient descent - updating the parameters after each training example. This time, I wanted to try batch learning to speed up the matrix operations. However, the system is getting a big drop in accuracy (a rise in perplexity) going from batch size 1 to 2, and it gets even worse with bigger batches.
Training on 200K words for 10 epochs, the perplexities are as follows:
<code>Batch size    Perplexity
1             61.13
2             189.69
10            566.53
</code>
Batch sizes 1 and 2 converge to these values, batch size 10 actually diverges. Also tried halving the learning rate after each epoch - this gave 415.92 with batch size 10.
Initially, I was hoping that I just have a bug somewhere, but as far as I can tell that's not the case. I even created batches that are full of the same example, dividing the learning rate by batch size, and the system gave the same result as a stochastic training process.
I expected batch training to get a small increase in perplexity due to parameters being updated less frequently. But also thought that this would be offset by the benefit of averaging out parameter updates before applying them.
Has anyone compared batch vs stochastic training on a feedforward neural network LM? Does such a result make sense on this task? Is there perhaps something that should be handled differently when batch training? Or do you think I still have a sneaky bug somewhere? Many thanks for suggestions!
Answer: Whenever you change the model configuration, you'll probably have to change the learning rate, any momentum parameters, and the number of epochs you use for training. I wouldn't be surprised if the best number of epochs, best momentum parameters and the best learning rate were different for each batch size.
Additionally, if your objective function is a sum of losses instead of a mean of losses, increasing the batch size has the implicit effect of increasing the learning rate. See: Mean or sum of gradients for weight updates in SGD
Answer: Theoretically, that should not be the case. if you have 100 patterns for example, why dont you try your batch code with batch size of 1 and repeat for 100 times. Will you get the same results as with the online training? If yes than your code is sound.
Comment: The results I showed above are given with the same code actually. The first experiment uses batch size of 1, which is the same as online training. I have an alternative LM implementation that I used to check that this result is valid, but I don't have an alternative batch learning implementation unfortunately.
Comment: At least it's not obvious that the code is unsound in that case.


Erro em ordenar números positivos e negativos Jasvascript

Question: Tenho um array com vários números, esses números são positivos, negativos e com decimal também:
var array = [365, -1.304, -17.803, -3.529, -3.602, -2.942, -2.074, -115]

Preciso ordenar esse array de forma do MENOR para o MAIOR. O resultado que era de se esperar é:
[-17.803, -3.602, -3.529, -2.942, -2.074, -1.304, -115, 365]

Mas ao invés disso está saindo isso:
[-115, -17.803, -3.602, -3.529, -2.942, -2.074, -1.304, 365]

O código que estou usando para ordenar:
GraphOne[Regional].sort(function(a,b){return a > b});

Comment: Cara, você tá usando `.`. Isso é separador decimal
Comment: @leofontes, mas -17.803 é 17 mil. Então 17 mil está mais longe do 0 do que -115
Comment: @AlissonAcioli sim eu tinha lido na convenção inglesa então fazia sentido meu comentário, mas você tá pensando em português.
Answer: Você precisa tirar o . dos números, o Javascript usa a convenção inglesa no qual o . é a nossa ,
No caso, pra eles 1.100 = um virgula um, enquanto pra nós seria mil e cem.
var array = [365, -1304, -17803, -3529, -3602, -2942, -2074, -115];

console.log(array);

array.sort(function(a,b){return a > b});

console.log(array);

Dá o resultado correto. Espero que tenha ficado clara a confusão que estava ocorrendo.
Answer: Você terá que remover os pontos decimais de todos os valores do array para poder considerá-lo como um inteiro. Para isso voce pode usar o Array#map, dentro da função convertendo cada número em string, trocar . por vazio, e então converter para número novamente.

var array = [365, -1.304, -17.803, -3.529, -3.602, -2.942, -2.074, -115].map(function(x) {
  x += '';
  return Number(x.replace('.', ''));
}).sort(function(a, b) {
  return a - b;
});

console.log(array);


How do you get a random decimal number (positive and negative) in PHP

Question: I wish to randomly position markers on a Google map so hoped to randomly generate the longitude and latitude of the markers using PHP and load them in via AJAX. The problem I have is not only are coordinates decimals but some are negative. For example I need the longitude to be between -2.07137437719725 and -1.92779606909178 and the latitude to be between 50.71603387939352 and 50.793906977546456. The only random functions I can find only work for positive integers so aren't feasible. I did try multiplying the numbers by a billion to remove the decimals and then would later divide the resulting random number by the same amount to return to using decimals but unfortunately PHP can't handle such large numbers.
I hope you can help.
Thanks
Paul
Comment: See this answer: http://stackoverflow.com/a/1504655/782609
Comment: Sounds like you want to do something similar to a `Geo guessing` thing?
Comment: Fred - Nope just need random markers on the map.
Comment: Kamituel - Thanks but that is limited to 14 digits and doesn't do decimals or negatives.
Comment: @AdrenalineJunky SO tip: Add the `@` symbol in front of the person's name you are addressing, that way the person in question will be notified ;-)
Comment: @AdrenalineJunky Personally, I don't know how to do it. However with a bit of searching on SO or Google and a bit of time, am sure I'd come up with something, or a mix of different functions to achieve what you wish to perform. You'll need a few functions to do this, one that will randomly select from a positive or negative function, then the number itself with a pre-defined set of accepted numbers and decimals.
Comment: @Fred I should of realised that @ is used. Thanks. I tried various approaches from Google & SO but non seemed to work properly, the main issue to overcome is variable type conversion and limitations such as integers are limited so can't do big numbers.
Comment: @AdrenalineJunky `dougBTV's` answer below seems like what you need. Food for thought: You might want to try and omit regions such as Africa and others that have not been scanned by Google; those coordinates could very well come up.... eventually. ;-)
Answer: Here's a function that I brewed up to do just this. It takes a number of decimal points as an argument and uses a check of a random number even/odd to make the positivity/negativity randomized.
<code>    $latlon = randomLatLon(4);
    print_r($latlon);

    function randomLatLon($granularity) {

            // $granularity = Number of decimal spaces.
            $power = pow(10,$granularity); // Extended 10 to the power of $granularity.

            // Generate the lat & lon (as absolutes) according to desired granularity.
            $lat = rand(0,90 * $power) / $power;
            $lon = rand(0,180 * $power) / $power;

            // Check if a random number is even to randomly make the lat/lon negative.
            if (rand(0,100) % 2 == 0) {
                    $lat = $lat * -1;
            }

            // Same for lon...
            if (rand(0,100) % 2 == 0) {
                    $lon = $lon * -1;
            }

            return array(
                    "lat" => $lat,
                    "lon" => $lon,
            );

    }
</code>
Answer: You could use something like this:
<code>float Latitudes = (float)((rand(0, 180000000) - 90000000)) / 1000000);
float Latitudes = (float)((rand(0, 360000000) - 180000000)) / 1000000);
</code>
Personally I think the accuracy up to 0.001 would be good enough for the locations. If you do not believe me, try it on google map (-32.833, 151.893) & (-32.832, 151.895), see how much further apart they are.
Comment: rand() is crap, use mt_rand() instead.
Comment: @yiz Thanks but that's quite low level or accuracy and it doesn't allow specification of a maps range which is important in my current project as the map is split into regions so it must be within one of the regions
Comment: @GordonM Thanks but even mt_rand has issues as it is still uses only integers so no positive, negative or decimals.


Definir Valor Default @Html.DropDownListFor via Json - ASP.NET MVC5

Question: Estou fazendo uma página de marcação de horário utilizando ASP.NET MVC5. O usuário digita os dados da marcação conforme o print abaixo:

Porém, caso ocorra algum erro no cadastro da marcação, o controller retorna a model para view com os dados inseridos pelo usuário, utilizando o conceito de model binding, e já traz os campos preenchidos pelo usuário e a mensagem de erro. Conforme o print abaixo:

Reparem que trouxe todos os campos preenchidos, porém menos o campo funcionário. Esse DropDownList eu preencho via Json, ele traz o dropdown preenchido com os funcionários, porém ele sempre vem selecionado o primeiro item da lista, e não o que usuário selecionou. Minha pergunta é: Como faço para depois que preencher a @Html.DropDownListFor eu setar um valor específico que será exibido ao usuário.
O controller que retorna o JSon com a lista de Funcionários é esse:
  FuncionariosFiltro = (from ord in Funcionarios
                                                     join detail in listaTurnoFuncionario on ord.Id equals detail.TCAD_PLAN_TRAB_FUNCIONARIO.TCAD_FUNCIONARIOId
                                                     where detail.TCAD_PLAN_TRAB_FUNCIONARIO.DATA_INICIO <= DateTime.Now.Date && detail.TCAD_PLAN_TRAB_FUNCIONARIO.DATA_FIM >= Convert.ToDateTime(Session["DataAgenda"]).Date
                                                     && ord.CODIGO_STATUS.Id == 1 && (listaAusencia.Where(l => l.FUNCIONARIOId == ord.Id).Count()) == 0
                                                     select ord).ToList();

return Json(new SelectList(FuncionariosFiltro, "Id", "NOME"));

No código abaixo, estão o dois DropDownList, o primeiro de serviços, que quando selecionado chama o controller acima que retorna a lista de funcionários, que preenche o segundo DropDownList, que é do Funcionários.

                @Html.DropDownListFor(p => p.CodigoServico, new SelectList(ViewBag.Servico, "CODIGO", "DESCRICAO"), "Serviço", new Dictionary<string, object>
                                                                                {
                                                                                {"data-toggle", "tooltip"},
                                                                                {"title","Informe o serviço desejado"},
                                                                                {"class","form-control" },
                                                                                {"id","CategoryType" },
                                                                                {"name","CategoryType" }
                                                                                })

                @Html.DropDownListFor(p => p.CodigoFuncionario, new SelectList(string.Empty, "Id", "NOME"),  new Dictionary<string, object>
                                                                                {
                                                                                {"data-toggle", "tooltip"},
                                                                                {"title","Informe o Funcionário que irá realizar o serviço"},
                                                                                {"class","form-control" },
                                                                                {"id","SubCategory" },
                                                                                {"name","SubCategory" }
                                                                                })

Quando é selecionado o serviço pelo usuário ele dispara o Java Script Abaixo, que chama o controller que retorna a lista de funcionários e preenche o dropDownList. Acredito que seja nessa função java script que eu deva mexer pra pré-selecionar o valor do dropDownList, porém não sei de qual forma proceder.
$(document).ready(function () {

    $("#CategoryType").change(function () {

        $("#SubCategory").empty();

        $.ajax({

            type: 'POST',

            url: '@Url.Action("CarregarFuncionarioVSservico", "ManutencaoAgenda")',
            dataType: 'json',
            data: { id: $("#CategoryType").val() },
            success: function (subcategories) {

                $.each(subcategories, function (i, subcategory) {

                    $("#SubCategory").append('<option value="'
                        + subcategory.Value + '">' +
                        subcategory.Text + '</option>');
                });

            },
            error: function (ex) {
                alert('Failed to retrieve Sub Categories : ' + ex);
            }
        });
        return false;
    })
    });

Answer: Para setar o DropDownList você tem que informar que o registro está selecionado. No fonte abaixo, troca o regraParaSelecionar pela sua logica para validar o funcionário que estava selecionado.
$.each(subcategories, function (i, subcategory) {
    if (regraParaSelecionar) {
        $("#SubCategory").append('<option selected="selected" value="' + subcategory.Value + '">' + subcategory.Text + '</option>');
    }
    else {
        $("#SubCategory").append('<option value="' + subcategory.Value + '">' + subcategory.Text + '</option>');
    }
});

Comment: Utilizei o código enviado, e era exatamente o que eu precisava. Muito obrigado pela ajuda.
Comment: @WagnerBarbosaDoNascimento se conseguir marque a resposta como sendo a resposta para a pergunta


Fracanapa tango meter and how to write it

Question: This is the first measure of the accordion part of Fracanapa, a Tango from Astor Piazzolla (you can easily find it on YouTube).

In the first 19 measures the meter of the accordion part is 3,3,2 and you can understand it by the accents and the slurs.
I was wondering if (at least today) wouldn't be better writing it like this:

Would that be correct? Wouldn't that allow you to better grasp the meter of the piece?
Thanks, g.
Comment: When I read the question, the first rhythm was completely clear to me; the second I initially misread as triplets. I would without question prefer the first notation.
Comment: Compare , for fun, with the rhythm of  Blue Rondo Alla Turk, which is in 9/8 but played (for a while, anyway) as 2-2-2-3 .  https://www.musicnotes.com/sheetmusic/mtd.asp?ppn=MN0049792
Comment: @Aaron I had the same comment from a composer friend.
Comment: The accents and articulation are there to contrast against the duple meter of the piece. Barring it as triplets/duplets, implies a change to the actual metric composition.
Comment: All depends, it seems. The notation in your second picture is precisely the beaming notation I've found in the Royal Conservatory of Music printing of "Spring Celebration" by Stephen Chatman I have (yes, this has influenced my beaming). It is also beaming notation that updaters at NinSheetMusic, a piano transcription website I frequent and have contributed to, continually tell me to avoid.
Comment: @Aaron To avoid the misreading I also tried to remove the slurs... but I think this would remove a useful information, so at the end I decided to keep them.
Answer: Except for older vocal practice (where the beaming follows the syllables), beams are strictly following the indicated meter and are an important performance indication.  Indeed, the beaming you suggest would strongly suggest playing triplets.
It is particularly bad, actually, for notating a Tango Nuevo rhythm that lives from the tension of its strong syncopation.  Your suggested notation makes no difference between first and second group of three and thus detracts from the Tango frame being 4-2-2 with the second note coming as a surprise early attack, giving the Argentinian Tango dancer a more cat-like predatory gait than the stiffer stop-and-go of the European ballroom Tango.
Comment: I agree, +1, I have very little experience playing tango but did do a few concerts of Piazzolla music on bass a few years ago. The second attack has a syncopation feel which makes regular 4/4 a better choice over 3-3-2 beaming. The other issue is when other rhythms that are simple to read in 4/4 are introduced they will sometimes look very odd with the 3-3-2 grouping.
Answer: I'd say that's a bit borderline choice.
I don't completely agree with those who say that the second example suggests triplets: while implicit triplets are considered normal writing practice, they rarely exist since the beginning of the piece (or the section), and at least the first group(s) or bar explicitly write it.
Yes, from a first glance they can look like triplets, but after seeing the first full bar it's clear that they are not (also, the "Tempo di Tango" is an important hint).
Also, when writing, it's important to consider the "target audience" of the score (as in the reader, not the listener). A musician accustomed to that repertoire would never have a doubt about that.
Then, the example referenced by Carl Witthoft is an interesting one to consider.
In that case the rhythm is clearly played for both the melody and the accompaniment, meaning that the actual metre is not 3+3+3 but 2+2+2+3.
This tango style is not that different, though: the whole rhythm section plays 3+3+2, and the melody usually follows a very similar pattern.
Considering this, using the same writing pattern would be coherent (and correct).
The problem comes when any rhythm stops following that pattern.
Let's take a basic rhythm:

Grouping should always follow the meter structure, so it means that then you should theoretically do this:

That is clearly terrible. Any musician would want to throw something at you for that.
The fact is: for meters clearly based on 4/4 (no matter their "groove"), it's usually unnecessary to use those subdivisions. They usually make sense only for complex pieces in which there are lots of meter changes with additive rhythms, so when it comes an actual 3+3+2 you really should write it in that way (even with the difficult writing above).
Then again, it depends on who's reading. For a rhythm section (bass, percussions, etc), it totally makes sense to have a 3+3+2 grouping. As a percussionist, I'd prefer it for this kind of music, since it really makes it clear what is the feeling I've to transmit in my playing, especially if no multiple rests are used: since most percussion instruments don't have a real sustain, having lots of rests is just distracting.
A typical and often preferred writing would be like this (note the staccato dots):
Comment: *"Grouping should always follow the meter structure, so it means that then you should theoretically do this:"* The example below that sentence, I don't quite see how the notation below follows from the rhythm above it. Why split the first half note into a dotted quarter and an eighth note? The half note already shows the middle of the measure perfectly fine, and if you absolutely had to split that half-note, why not tie two quarter notes together instead, which would at least show where each beat lies?
Comment: @user45266 that's how the grouping of the rhythm of the first example could theoretically be shown in a 3+3+2 pattern grouping.
Answer: The excerpt provided suggests that the time signature is actually 8/8, and not 4/4 at all.
This breaks down into 3+3+2/8 (just like 9/8 but with the last quaver cut off).
Sww Sww Sw
123 456 78
Beaming in this time signature will be as you have suggested – three quavers, three quavers, two quavers.
Comment: Yes, absolutely. In this sense my question could be reworded as "Would be ok write this tango as 8/8 etc. etc."


looking for specific errors in a log within the past hour

Question: I have a log that I need to audit every hour going back an hour. I currently run a command and just keep track of what I have accounted for in it's output but having an hourly audit would make things a lot easier. I've left off the additional grep filters for this output since I do not believe they are necessary.
How can I make this command output lines from the last hour up to now.
cat error.log | grep `date -u "+%Y-%m-%d"`| grep -i dangling*

Answer: You can use the -d option to adjust the current time.
cat error.log | grep `date -d '-1 hour' -u "+%Y-%m-%dT%H"` | grep -i dangling*

See https://unix.stackexchange.com/questions/49053/linux-add-x-days-to-date-and-get-new-virtual-date
Comment: @DomJangles if this or any answer has solved your question please consider [accepting it](https://meta.stackexchange.com/q/5234/179419) by clicking the check-mark. This indicates to the wider community that you've found a solution and gives some reputation to both the answerer and yourself. There is no obligation to do this.


Prove $f(x)\geq e^x$

Question:
Let $f:\mathbb{R}\rightarrow \mathbb{R}$ be a differentiable function, such that $f^{\ '}\geq f$ and $f(0)=1.$ Prove $f(x)\geq e^x, \forall x\geq0$.

So I started by defining a function $g(x)=f(x)-e^x$, and my intuition tells me that $f^{\ '}\geq f$ means $f$ is increasing, but I'm not sure how to continue.
Any help appreciated.
Comment: @BAYMAX The negation of $\forall x,\ f(x)\geq e^{x}$ is **not** $\forall x,\ f(x) < e^{x}$; but your contradiction relies on the latter.
Comment: (What you wrote in your comment, after editing, basically. You initially relied on taking $x=0$ in the contradiction; and indeed, you cannot, as you realized.)
Comment: I was thinking about this , please correct me if I am wrong - i proceed through contradiction suppose $f(x) < e^{x}$ , then $f(0) < 1$ but $f(0) = 1$ , hence a contradiction thus $f(x) \geq e^{x}$, but here i never used the fact of $f^{'} \geq f$ ?.
Comment: Baymax, your idea is wrong, as you have to prove that for all $x\ge 0, f \ge e^x$.
Comment: Still i don't get why am I wrong @Lelouch
Comment: Baymax, you assumed that FOR ALL x\ge 0, f(x) \le e^x, and then made a contradiction. This shows that for NOT ALL x\ge 0, f(x) \le e^x. But you have to prove that FOR ALL x, f(x) is \ge e^x. Think carefully. You will understand.
Comment: I think negation is $\exists x < 0 s.t f(x) < e^{x}$  is it correct @ClementC.so I cannot take $x =0 $
Comment: Yes, and this is not what you relied on, making your proposed proof flawed. You have no guarantee at all that $f(0)<1$, when proceeding by contradiction.
Comment: "this is not what you relied on" ? @ClementC.
Answer: Consider $g(x)=f(x)e^{-x}$,note that $g$ is increasing and $g(0)=1$ hence $f(x) \geq e^x$ QED
Comment: Your proof is good, but this dosen't accurately answer the OP's question regarding his approach. +1for the proof anyway
Comment: @ClementC.: I guess that the answerer should explain that.
Comment: How do you know that $g$ is increasing ?
Comment: @YvesDaoust Differentiating, as in the accepted answer to the nigh-identical duplicate http://math.stackexchange.com/questions/2129435/if-fx-is-continuous-and-differentiable-function-prove-that-fx-geq-e3 , I reckon.
Comment: @YvesDaoust Oh, surely. I'm with you on that.
Comment: Dear @YvesDaoust: That was left for OP to figure out.Regards,
Comment: @ArpitKansal: sorry, I beat you out.
Comment: Dear @YvesDaoust: Yeah thats fine,cheers!
Answer: I have a semi formal proof to help you along your line of thought.
Let $g(x) = f(x) - e^x$ (as you defined.
Then $g'(x) = f'(x) - e^x$
Clearly $\forall x\ge 0, g'(x) \ge g(x)$ since $f' \ge f$.
Let us look at the zeroes of $g(x)$. Let $a$ be a zero. That is
$g(a) = 0, a\gt 0$ (if there exists no such a, then our proof is already done - justify yourself why)
Then,$g'(a) \ge 0$ but, we know that $g(x)$ is positive in the neighbourhood of x = 0(again justify why)
So, for $g(x)$ to be negative in the neighbourhood of $a$, the derivative must be negative(justify why). But clearly,that is not so. You can also justify from the last inequality that at all further roots, the derivative is only zero at those point(touchig the axis). Hence your proof is done, as $g(x)$ is non negative. Hope this helped.
Answer: The crux of the proof is the transformation
$$f'(x)\ge f(x)\implies f'(x)e^{-x}-f(x)e^{-x}\ge 0\implies(f(x)e^{-x})'\ge 0.$$
Then $f(x)e^{-x}$ is a non-decreasing function such that $f(0)=1$ and $f(x)e^{-x}\ge1$ for positive $x$.

Prove $f(x)\geq e^x$


kendo grid required fields

Question: I've a Kendo grid with editable = incell and more than one required text field. The default values are empty strings. When I click on "Add new record" the first required field is marked as error because it's empty. I can only leave the field after I've done some input. Although the other required fields are still empty I can leave the row.
How can I prevent that a row can't be left without filling all required fields?
Comment: Can you show you Kendo UI Grid definition?
Answer: Try this,
<code> $('.k-grid-save-changes').on("click", function () {
            var grid = $("#grid").data("kendoGrid");
            if (grid.editable && !grid.editable.validatable.validate()) {
                //What ever you want to do
                e.preventDefault();
                return false;
            }
       });
</code>

kendo grid required fields


Track a person by his clothes color using the Kinect sensor

Question: I'm new to the image processing, and I'm working on a simple project to recognize people by their clothes color. I'm not sure what the best way is to do that. Since I'm using the Kinect (with Kinect-SDK), it is easy to detect people using the depth stream, and by mapping the depth data to the color data, I can get the color pixels of the people. I tried to build a color histogram for each person to recognize the person color. I'm not sure if this right or not!
What I'm doing is:
1- Get the depth data from the Kinect device.
2- Ensure if a pixel is a player pixel or not by using the Player Index.
3- Map player pixels to color pixels.
4- Build a color histogram for the player.
I have a problem dealing with step 4. This is how I'm trying to build the histogram (32 bins):
<code>color = ColorPixelData[colorPixelIndex];
B_Values[color / 8]++;

color = ColorPixelData[colorPixelIndex + 1];
G_Values[color / 8]++;

color = ColorPixelData[colorPixelIndex + 2];
R_Values[color / 8]++;
</code>
I think I'm doing it in a wrong way. The colors values look very different every time I run the program on the same scene.
Could anyone give me some points?
Any help will be appreciated.
Comment: Is this affected by the amount of ambient light in the room?
Answer: Color histogram wont help you. Back in the day when i was doing some face recognition tool, color histogram would give different values for the pictures that look almost alike. So it's not a way to go. Instead of building a color histogram you could see how much for example red color is present on the scene, if someone of your subjects is wearing a red jacket.


IIS7 - Change physical path of all sites in server config

Question: I've been trying to migrate an IIS7 webserver (rather large one), and so far am appalled by the state of msdeploy and using it for what I need to do. Enough on that, here's my current problem:
The server I'm migrating from has all the sites on a D:\ drive. It isn't possible for us to have a D:\ drive in the new environment, so I need to change all the site paths to C:\. I obviously do not want to do this for each site manually, so I thought to edit the paths in applicationHost.config. However, IIS doesn't seem to care one bit. All the paths are C:\, yet IIS still tries to reference D:\. When I look at site settings, it still says D:\. Where in the world is this value coming from?
It's on a 64-bit server, and I checked the OTHER applicationHost.config in \syswow64\, and that has magically changed to C:\ as well.
How can I do this?
Comment: "Fixed" it by copying applicationhost.config to a 2003 box, editing it there and copying back. Apparently 2008 "pretends" to save your file (i.e. it shows my changes when I open it in notepad) but read them from, either some cached version, or from another file elsewhere.
Answer: There should be a relatively simple using the correct syntax in MSDeploy.
Better than trying to explain all of the ins-and-outs of the process here, I'd recommend reading this thread at iis.net
Make sure you go through the whole thread before starting a migration.
Comment: Hey. I'd read that already. Sadly the person who posted there had a very different setup. He was running IIS 6 with sub-apps. We're running IIS 7 with individual apps. I will however play with the migrate verb a bit now. Any idea where the actual physicalPath values are stored for the sites though?
Comment: Sorry, the only physicalPath I know of off the top of my head is the one that's located in applicationHost.config -- There must be another "global" location to store this, but I don't recall running into it.
Answer: "Fixed" it by copying applicationhost.config to a 2003 box, editing it there and copying back. Apparently 2008 "pretends" to save your file (i.e. it shows my changes when I open it in notepad) but read them from, either some cached version, or from another file elsewher


NoClassDefFoundError with klio-mobile-dataservice on Android

Question: I'm trying to implement klio's library, but I keep on getting this exception, even following the tutorial.
Here's the logcat:
<code>    11-12 23:34:30.979: E/AndroidRuntime(27299): FATAL EXCEPTION: main
    11-12 23:34:30.979: E/AndroidRuntime(27299): java.lang.NoClassDefFoundError: flex.messaging.io.amf.client.AMFConnection
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at tiwiz.connection.test.ConnectionActivity.retrieveNetData(ConnectionActivity.java:72)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at tiwiz.connection.test.ConnectionActivity.onCreate(ConnectionActivity.java:32)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at android.app.Activity.performCreate(Activity.java:5008)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1079)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2139)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2210)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at android.app.ActivityThread.access$600(ActivityThread.java:142)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1208)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at android.os.Handler.dispatchMessage(Handler.java:99)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at android.os.Looper.loop(Looper.java:137)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at android.app.ActivityThread.main(ActivityThread.java:4931)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at java.lang.reflect.Method.invokeNative(Native Method)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at java.lang.reflect.Method.invoke(Method.java:511)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:791)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:558)
    11-12 23:34:30.979: E/AndroidRuntime(27299):    at dalvik.system.NativeStart.main(Native Method)
</code>
Compiling goes well, so it looks like it finds classes during compile process...
Can someone help me? :)
Comment: if you are trying to run it on device, you may be using it to compile but not deploying the resource.
Comment: Have you ensured classpath runtime config correctly?
Comment: What do you mean? I added it as reference from Project -> Properties -> Project References and signed it for export in Order and Export tab, shall I do something else I'm not aware of? Thanks
Comment: Did you ever get klio-mobile-dataservice to work?  I am currently have an issue where MyService.MyMethod call isn't working.  Also trying to not sure how to add my own custom classes to be serialized and unserialized
Answer: The problem is due to a change in how libraries are managed in ADT, starting from version 17.
This is the solution, essentially:

remove the KlioMobileDS project from your Java Build Path
export it into .jar
put that .jar into your project libs folder
Comment: Same as above, but now I can't see it under libraries, it's under Android Dependencies.
Comment: Tested again and now it works, I forgot to clean everything in the project. Many Thanks!
Answer: Check whether there is a libs folder inside your project. If not create it and add all your external libraries to it. Then import those libraries from this libs folder to your buildpath. That's it. If the problem remains go to Order and Export tab in Build Path and move all libraries you imported to the Top.


Mapping a part of Memory mapped file in Windows CE

Question: Where exactly does the address space get blocked for the MMF in the process of creating and mapping the MMF in Windows CE. Is it during the creation of MMF or when the view is getting mapped? Will the creation succeed only if there is a contiguous space of x (specified during creation) in virtual address space?
Secondly, is it possible to map a part of file in win CE like we do in win32. If I create a MMF with 0x8000 bytes and map a view of the file giving an offset of 0x5000 and size of 0x7000, the mapping is failed. It says INVALID_PARAMETER. Although the mapping is allowed when sum of offset and size of the view is less than 0x8000. Does it always get mapped from beginning of the file?
Answer: In Windows Mobile 6.x (CE 5.0), Memory mapped files are stored in the Large Memory Area above 0x42000000 in the virtual address space.
Yes, you need contiguous space to successfully allocate a MMF.
0x5000 + 0x7000 = 0xC000. That's larger than the allocated space of 0x8000 bytes.
-PaulH
Edit
Also, the offset must be a multiple of your system's allocation granularity. Use GetSystemInfo to find that value. On one device I have, for instance, it is 0xa11 bytes.
Comment: But the instruction is supposed to read 0x7000 bytes starting from offset 0x5000 in file and read and store the following 0x7000 in the space allocated for MMF which is 0x8000 right? In that case, since 0x7000 is way lesser than 0x8000, why should it fail?
Comment: By "where", I meant the stage of creation of MMF. Not the location. Or it should have been "When". Does it get allocated during CreateMMF call or MapViewOfFile call?
Comment: To answer your other question, it is mapped during the MapViewOfFile call: "Creating a file-mapping object creates the potential for mapping a view of the file but does not map the view. MapViewOfFile maps a view of a file into the address space of a process."
Comment: That offset is within the file being mapped. So, if your offset is 0, then you're at the beginning of the file. If your offset is 0x400, then your 1KB in to the file and the maximum size you can map is 1KB less than your allocation.


7 - Can't update page titles

Question: I'm currently working on a project on which I'm trying to dynamically shorten the pages titles as following:
"This is very long text blahblahblah | Site Name" becomes "This is very ... | Site Name" but the rendered page doesn't take this into account.
Here's my problem: I tried several hooks to implement this behaviour, using hook_menu_alter(), hook_preprocess_html(), hook_node_preprocess(), etc. and I followed the indications given to update the pages titles when using these hooks, but it always falls back to using the original long title.
Actually, when I set the arrays accurate keys/values, would it be $vars['head_title'], $vars['title'], etc. I can var_dump the array and verify the needed variable is updated, but the title is untouched. Using drupal_set_title($new_title) didn't help either.
I'm quite puzzled; would this be because I'm using Custom Field, Metatag, or a specific module? Or would this be because I'm not updating the page at the correct moment during its generation? Drupal is somehow new to me!
Thanks a lot :)
~Steph
Comment: I want know how you have created page by (view,panel or something else)
Answer: OK! After going back and forth between var_dump's, installed modules list and hooks, I discovered that the ONLY hook that will update the page's tags is hook_metatag_metatags_view_alter(&$output), in which something like this needs to be done:
$output['title']['#attached']['metatag_set_preprocess_variable'][0][2] = "Custom title";

My n00b diagnosis is that this hook is called right after the last Metatag module's function that rebuilds the metatags from scratch, and therefore it's the only one that matters.
Hope this will help others! :)
~Steph
Answer: Did you try template_preprocess_page? You can modify the $variables['title'] value there.
Inside your theme's template.php file,
function YOUR-THEME_preprocess_page(&$variables) {
  $variables['title'] = YOUR-TRUNCATE-FUNCTION($variables['title']) . ' | ' . $variables['site_name'];
}

7 - Can't update page titles


Proving that a line segment joining the midpoints of two sides of a triangle equals half of third side.

Question: Is this proof correct? I know it can be proven using the Midpoint Theorem. I believe the proof is correct, in that case, please explain how I should prove it with the same approach without a graph. If it is incorrect, please explain why.

What I am basically trying to do involves interpreting:

(source)
on a graph such that CA is on the x-axis.
As point D lies in the centre of slope BA, y position for D should equal (y position for B + y position for A)/2
=(y position for B)/2
y position for D = ED
y position for B = BC
Therefore, ED = 1/2 BC
Would appreciate if you could stick to layman terms.
EDIT1: Changed how I was expressing my doubt.
Answer:
"Since the length of A is 0 (A point has no dimensions)"

Many people would object to this. Strictly speaking, length of a point is not what you are thinking it like to be.

"DE is halfway between A and BC "

Again, this is not what a mathematician would accept. What do you mean by the midpoint of a point (here $A$) and a line (here $BC$). It could mean the point that it halfway between the line passing through $A$ and intersecting $BC$ perpendicularly. But,it may not mean this also. There ambiguity. So its not acceptable.

"DE is halfway between A and BC "

You should be careful using mathematical logic. The therefore has great power of conclusion to a situation. And with great power comes great responsibility. Your previous statement and the therefore statement is not complying. Its difficult to see why you make such a conclusion. So that's again not acceptable.
Answer: Alternative proof:
Since $\angle BAC = \angle DAE$ and ${BA\over DA} = {CA\over EA} =2$ triangles $ABC$ and $ADE$ are similar with ratio of similarity 2, so $BC = 2 DE$.
Answer: This is not quite right. Firstly, as an aside, we don't talk about "lengths of points". As such, your first statement about the length of $A$ doesn't quite make sense.
You then make a statement about the line $DE$. You say that it's "half-way between" something and something else. What is the meaning of this? What does it mean for a whole line to be halfway between something and something else? If $DE$ was not a straight line, would this statement still stand? What if $DE$ was "rotated" a little?
Certainly your final assertion isn't right. What is the logical step between $DE$ being halfway between $A$ and $BC$, and $DE$ having half the length of $BC$? This does not follow in any simple way.
As such, your proof has a few holes, and I suspect is not quite going to be fruitful enough to prove your statement. I hope this helps.


What exactly is meant by "constant autocorrelation structure" in the definition of stationarity of a time series?

Question: I have come across the term "constant autocorrelation structure" in every definition of stationarity but I have not been able to find an explanation of this phrase. I know that autocorrelation refers to present values being dependent on past values but can anyone please elaborate on what a "constant" autocorrelation structure looks like?
Secondly, can I say that seasonality and trends are all conceptually subsets of autocorrelation? Can there be a situation where there is a trend and seasonality but zero autocorrelation?
Comment: So it is either subset or zero correlation?
Answer: "Constant" refers to the requirement that the pattern of autocorrelation does not change over time. So the strength of the dependence of, say, 2020 and 2014 is the same as that between 1990 and 1984. When you for example have an AR(1) process
$$
y_t=\phi y_{t-1}+e_t,$$
the autocorrelation coefficient of $y_t$ and $y_{t-j}$ is well-known to be $\phi^j$. This depends on the lag $j$, but not on when we look at that lag. If $\phi$ were to change over time, as in $\phi=\phi_t$, the condition would be violated.
For the second question, consider $y_t=\delta t + u_t$, a trending process of time $t$ with slope $\delta$ and an error term $u_t$ which we assume to be iid. The autocorrelation function does not show any autocorrelation, as
\begin{eqnarray*}
Cov(y_t,y_{t-j})&=&E[(y_t-E(y_{t}))(y_{t-j}-E(y_{t-j}))]\\
&=&E[(\delta t + u_t-\delta t)(\delta (t-j) + u_{t-j}-\delta (t-j))]\\
&=&E[u_tu_{t-j}]\\
&=&0
\end{eqnarray*}
You could run the following code to get such a "trend-stationary" process.
<code>T_end <- 100
t <- 1:T_end
delta <- 2
u <- rnorm(T_end)
y <- delta*t+u
plot(t, y, type="l")
</code>
Comment: Thanks for the comment. I got my answer for the 1st question. For the 2nd one, can you please elaborate a bit and mention what does the two terms on right hand side mean? Is is possible if you can add a visual graph that exhibits this behaviour?
Comment: @ShahzebNaveed, I made an edit


Removendo elementos duplicados em uma lista com python

Question: Tenho que fazer uma função e não estou conseguindo desenvolver a lógica. É o seguinte:

Escreva a função remove_repetidos que recebe como parâmetro uma lista com números inteiros, verifica se tal lista possui elementos repetidos e os remove. A função deve devolver uma lista correspondente à primeira lista, sem elementos repetidos. A lista devolvida deve estar ordenada.

Dica: Você pode usar lista.sort() ou sorted(lista).
Comment: amigo esse era um desafio da USP em Python correto? tente se esforçar mais e fazer perguntas referente ao seu código...não pedir respostas
Comment: Essa pergunta é constantemente  feita na comunidade, até uma simples busca no google retornaria posts relacionados com tudo espero que tenha entendido sobre o conteúdo e antes de refazer a pergunta poderia ter pesquisado sobre no campo chamado "PESQUISA"  logo acima.
Answer: Uma função pythonica para resolver este problema seria:
def remove_repetidos(lista):
    l = []
    for i in lista:
        if i not in l:
            l.append(i)
    l.sort()
    return l

lista = [1, 1, 2, 1, 3, 4, 3, 6, 7, 6, 7, 8, 10 ,9]

lista = remove_repetidos(lista)
print (lista)

Saída:
>>>[1, 2, 3, 4, 6, 7, 8, 9, 10]

É bem fácil perceber como a função funciona, por isso vou omitir mais explicações.
Comment: Me deparei com esse desafio da USP hoje e não tava conseguindo fazer, obrigado. Eu estava tentando usar o `del` pra deletar os elementos repetidos da lista, mas isso bagunçava os índices, fazendo com que o `for` não funcionasse corretamente.
Comment: Outra resposta tem uma versão realmente "pythonica", simplesmente usar `sorted(set(lista))`. É mais limpo, mais fácil de entender e mais rápido.
Esta resposta funciona bem, só estou deixando este comentário pois a resposta está bem votada e pode criar uma visão errada do que é considerado "pythonico" pela comunidade.
Answer: A lista deve estar ordenada, porém na mesma ordem da original ou pode-se ordenar depois?
Digo isso pois set() é usado justamente pra isso e é bastante eficiente inclusive.
De forma geral você deve evitar de reprogramar aquilo que já está na biblioteca padrão.
>>> lista = [1, 1, 2, 1, 3, 4, 3, 6, 7, 6, 7, 8, 10 ,9]
>>> sorted(set(lista))
[1, 2, 3, 4, 6, 7, 8, 9, 10]

Já se você necessita manter a mesma ordem da lista original veja essa resposta: https://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-in-whilst-preserving-order
Ou então crie um OrderedSet que seria o equivalente a um collections.OrderedDict só que para listas. Funciona bem também.
Answer: TL;DR
Se a ordem original da lista NÃO importa:
def remove_dups(lista):
    """Remove os itens duplicados mas não mantém a ordem original."""
    return list(set(lista))

Se a ordem original da lista importa (Python3.7+)
# CPython3.6+
# Pypy 2.7+
# Outras implementações Python3.7+
def remove_dups(lista):
    """Remove os itens duplicados mantendo a ordem original."""
    return list(dict.fromkeys(lista))

Se a ordem original da lista importa (Versões mais antigas do Python)
from collections import OrderedDict

# Versões anteriores do Python
def remove_dups(lista):
    """Remove os itens duplicados mantendo a ordem original."""
    return list(OrderedDict.fromkeys(lista))

Mantendo a ordem da lista original
Como não vi postarem aqui ainda, depois que a linguagem começou a garantir a ordem de inserção nas chaves do dict (ver notas de versão do Python 3.7), você pode usar o método dict.fromkeys para criar um dicionário recebendo apenas um iterável com as chaves (ou seja, sem OrderedDict). E depois basta converter de volta para lista.
Demonstração:
lista = [1, 2, 10, 4, 5, 1, 1]

dict_aux = dict.fromkeys(lista)
# {1: None, 2: None, 10: None, 4: None, 5: None}

resultado = list(dict_aux)
# [1, 2, 10, 4, 5]

PS: Lembrando que esse comportamento já existe na implementação do CPython 3.6.

Ignorando a ordem da lista original
Já se você não se importa com a ordem basta converter para um set e depois para uma lista novamente, como já mencionado em outras respostas.
lista = [1, 2, 10, 4, 5, 1, 1]

set_aux = set(lista)
# {1, 2, 4, 5, 10}

resultado = list(set_aux)
# [1, 2, 4, 5, 10]

Performance
A motivação desta sessão foi esse tweet do Raymond Hettinger (um dos core developers do CPython), onde ele afirma que a maneira mais rápida de remover itens duplicados mantendo a ordem da lista original é usando dict.fromkeys.
Então estou testando os seguintes algoritmos:

Usando set mas perdendo a ordem da lista original (para comparar com as outras soluções):
def remove_dups_set(lista):
    return list(set(lista))

Usando dict.fromkeys:
def remove_dups_fromkeys(lista):
    return list(dict.fromkeys(lista))

Usando OrderedDict como já proposto em outra resposta:
def remove_dups_ordered_dict(lista):
    return list(OrderedDict((v, None) for v in lista))

Alterando a versão anterior para usar o método OrderedDict.fromkeys:
def remove_dups_fromkeys_ordered_dict(lista):
    return list(OrderedDict.fromkeys(lista))

Usando um for normalmente como mencionado em outra resposta:
def remove_dups_for(lista):
    l = []
    for i in lista:
        if i not in l:
            l.append(i)
    return l

Alterando a versão anterior para que use um set checar se o valor já está na lista ou não:
def remove_dups_for_and_set(lista):
    l = []
    visited = set()  # guarda os itens que já estão em `l`
    for i in lista:
        if i not in visited:  # sets são otimizados para operações do tipo "in"
            l.append(i)
            visited.add(i)
    return l

Resultados
Resultados obtidos neste Repl.it usando o módulo timeit para medir o tempo de execução:
                  remove_dups_set:  1.2999 s
             remove_dups_fromkeys:  2.7048 s
         remove_dups_ordered_dict: 25.0388 s
remove_dups_fromkeys_ordered_dict:  5.4657 s
                  remove_dups_for:  7.6752 s
          remove_dups_for_and_set:  5.2055 s

PS: a máquina que roda o Repl.it não é muito forte, se você for executar os testes na sua máquina seria legal aumentar o número de iterações para um número maior (no Repl.it está number=100_000). E também aumentar o tamanho da lista para que se torne mais evidente a diferença das soluções de acordo com o tamanho da lista (no Repl.it está * 50).
Answer: Outra forma interessante para resolver esta questão é utilizar o método OrderedDict da biblioteca collections. Neste caso podemos montar o seguinte código:
from collections import OrderedDict

def remove_repetidos(li):
    return sorted(OrderedDict((i, None) for i in li))

lista = [1, 1, 2, 1, 3, 4, 3, 6, 7, 6, 7, 8, 10, 9]

print(remove_repetidos(lista))

O resultado deste algoritmo é:
[1, 2, 3, 4, 6, 7, 8, 9, 10]

Podemos obter o mesmo resultado utilizando a expressão lambda associada com a função map(), tal como listado no código abaixo:
lista = [1, 1, 2, 1, 3, 4, 3, 6, 7, 6, 7, 8, 10, 9]
unicos = sorted(map(lambda x: x[1], filter(lambda x: x[1] not in
                lista[:x[0]], enumerate(lista))))

print(unicos)

Com este algoritmo obtemos o mesmo resultado que é:
[1, 2, 3, 4, 6, 7, 8, 9, 10]

Outra forma interessante é utilizar o seguinte algoritmo:
def remove_repetidos(li):
    return sorted(dict(zip(li, li)).keys())

lista = [1, 1, 2, 1, 3, 4, 3, 6, 7, 6, 7, 8, 10, 9]

print(remove_repetidos(lista))

Obtendo desta forma o mesmo resultado:
[1, 2, 3, 4, 6, 7, 8, 9, 10]

Outra forma válida de responder esta questão pode ser:
lista = [1, 1, 2, 1, 3, 4, 3, 6, 7, 6, 7, 8, 10, 9]
r = []
[r.append(i) for i in lista if not r.count(i)]

print(sorted(r))

Resultando na seguinte saída:
[1, 2, 3, 4, 6, 7, 8, 9, 10]

Answer: Acho que dá pra fazer utilizando a função del, da seguinte forma:
def remove_repetidos(l):

    lista = l
    i = 0
    while i < len(lista):
        j = i + 1
        while j < len(lista):
            if lista[j] == lista[i]:
                del(lista[j])
            else:
                j = j + 1
        i = i + 1

    return sorted(lista)```

Answer: tive problemas utilizando lista.sort() ou sorted(lista), (se for uma lista txt, é colocado um "\n" toda vez que apertamos "enter", então fiz de outra forma)
Código:
def tira_nome(lista):
    listaP = []
    num = 0
    for lis in lista:
        lis = lis.rstrip()
        for letter in listaP:
            if letter == lis:
                num = 1
        if num == 0:
            listaP.append(lis)
            num = 0
        else:
            num = 0
    return listaP

Answer: numeros = input("Digite uma lista com números inteiros: ")

print("Você digitou os seguintes números: ")
print(numeros)

#Cria a função remove_repetidos que cria e ordena um set a #partir de uma lista
remove_repetidos = lambda l: list(sorted(set(l)))

'''remove_repetidos.remove(" ")'''

print("Sua lista filtrada e ordenada: ")
print(remove_repetidos(numeros))

Answer: new_df = list(set(self.df['col']))
Comment: Poderia explica melhor sua resposta?


Three ways to harmonize a note with a triad?

Question: For simplicity assume we're in C major. Let's say I'm playing a song, and I want to play a basic triad chord to harmonize whatever current melody note is played. In this example say the melody is a D note that I would play with my right hand. Then for my left hand, I think there's three possible chord combinations that I can play (I usually pick the one that sounds best):

The note is on the left of the triad
The note is in the middle of the triad
The note is on the right of triad.

Does this sound about right?
Edit: In this question I (incorrectly?) assumed the triad has to include the melody note. I haven't ever seen it not including the melody note, but maybe that's just due to my inexperience.
Comment: This is a related video that I found to be quite interesting: https://www.youtube.com/watch?v=JXfQsHT5c30&t=528s
Comment: You mean with no accidentals or alterations, right?
Comment: yep no accidentals completely diatonic
Comment: Have you considered inverting your triads?
Comment: inversions don't count (for this question)
Comment: Using only the parameters given in your question, Laurence's answer is accurate. While Todd's has many more votes, I don't think he's answered the question you pose. Any chance of some clarity, as right now, it's a vaguely written question. Unless it's exactly what you want to ask - triads only, only containing C key diatonic notes, and all containing a D note. In which case, it's a simple 'yes', there are only those three.
Comment: @Tim I edited the question at the bottom. feel free to edit it the title or answer any way you want though if you think it's not clear.
Comment: It's better now. Before, it gave the impression that the triad had to include note D. Hence Todd's answer, which suggests other notes. But strictly speaking a triad will contain a root, third and fifth. Hence, for example, D, G and A, (sus4) isn't a triad, it just has three notes.
Comment: @Tim I think part of the confusion might be whether or not the question separates the triad from the melody or not. In other words, is the question about a melody note and also a separate triad for a total of four notes at once, or is the question merely about a single triad and only three notes? It seems that foreyez has confirmed that the former interpretation is what was intended.
Comment: @ToddWilcox - yes, you're right. Something that never needed to happen had the question been posed more clearly. It was ambiguous, but my answer wasn't appropriate anyway. Sometimes one has to read between the lines, and you did.
Comment: @Tim In defense of foreyez, it took me forever to understand what you found unclear about the question. It seemed completely clear to me when I read it, but I was pre-disposed to that interpretation because I've been collaborating with a melody writer who is also very picky about the chords I try to put under the melodies, so I've been living in "how to harmonize a melody" land for about 18 months now. When I sell my collaborator on a chord that doesn't have the melody note as a chord tone, it's a small victory for me.
Comment: You just asked *another* question! Yes, probably more than 99%. 'Laura' is one that springs to mind with melody notes outside the written chords - although instead of them being m7, they *could* have been m9.
Answer: There are actually more ways, many more ways.
One fairly straightforward way is to make the melody note the seventh of the chord. So in this case the chord would be E G B.
From there you could make the melody note an add 9 by using the chord C E G, but an octave lower, add 11 with A C E, etc.
Depending on how long you are playing the D and on what beat and whether it is emphasized or not, you could "harmonize" it with almost any chord. Each choice has a different flavor.
You could also borrow triads that contain D from related keys. And of course you could branch out beyond triads.
Comment: @Tim I read it as asking how many ways are there to harmonize a melody note using a triad. He proposed three possible triads, a D minor triad, a B diminished triad, and a G major triad, and then asks if there are any others. I suggest that there are many other triads that can harmonize with that note, depending on the desired overall sound.
Comment: @Tim The diagram doesn't exactly show what the text relates. Note the discussion of right hand playing a D and the left hand playing a triad. I think the diagram is just there to show that he assumes that the melody must always be a chord tone, not that he would only be playing three total notes in his hypothetical situation. Rather, he would be playing one note with his right hand and a triad with his left for a total of four notes.
Comment: @Tim You're looking at the question title. Todd is answering the question in the text,  "I want to play a basic triad chord to harmonize whatever current melody note is played." The answer being, "Don't limit yourself to the melody always being a chord tone."
Comment: The OP asks about notes *with* a triad, then gives examples of *those* triads. Should the question read *within* rather than *with*, as your answer alludes to the latter, while mine leans to the former? Unclear question, or unclear translation thereof?
Comment: The OP harmonised D with *two other notes*, thus making his triad. There wasn't really mention of any more than *two other notes*, otherwise, it's not a triad. You're right, in that D could be mixed with a myriad of other notes, diatonic or not, but I wonder whether the triad part (including D) is the crux. Not happy with the wording of the question. Seems to allow too much interpretation.
Comment: I'm still concerned with the vagueness of the question. A triad is a note, its 3rd and its 5th. Three notes. And apparently diatonic, which seemingly brings it all down to the three shown. So that D can be root, 3rd or 5th of a triad, but the other two notes must belong to key C. So the correct answer is 'yes'. What am I missing?
Comment: @Tim Todd's answer is valid. the question was just about how to harmonize using any triad within the diatonic scale. my experience was just that I see 99% of the time the melody note tends to be one of the chord tones. (note that the 99% could be due to my lack of experience, if its not that high let me know!)
Answer: If by "C major," you are assuming the key of C, with, as you explained in a comment, no alterations and no accidentals, and using the note "D" as a reference point, and if you are assuming a triad to mean only the adjacent diatonic intervals of 3rds (based on your graphic), then yes, what you have built is accurate. It is pretty fundamental to music theory.
Of course, this applies to any note in the scale, not just D. In C major, start on C, and you can have C be the root (C major), the third (A minor), or the fifth (F major); go to D, and the same applies; go to E, and the same applies; etc.
Your question more particularly seems to assume the constraint that the melody be part of the triad itself or be a note that is in common with a note in the triad. This need not be the case; indeed, it often is not the case. Melodies can be tricky, fluid things when you are considering harmonization.
Take the well-known "Auld Lang Syne," for instance:

LilyBin link, with MIDI audio
In this very basic harmonization, even in the first measure, we encounter a melody note that does not "fit" into the triad of F major. In almost every measure, there is a melody note that does not "fit" into the triad that is harmonized with it.
On the other hand, if you harmonized every note in its own triad according to the rules in your question, it might arguably sound worse:

LilyBin link, with MIDI audio
The combination of melody notes that are part of the triad and that are not part of the triad is often what provides musical interest.
This is to say nothing, of course, of extending the harmony beyond triads—suspensions, extensions, and borrowed, oh my! But, given the constraints you provided in your question, yes, notes can be harmonized with any of the three notes in a diatonic triad of which that note is a part—but when harmonizing a melody, I think it is often advisable to know how best to produce good music. To only look for triads that "fit" the melody in the way you describe seems like an inadvisable course to stay on, but it is a good starting point.
Comment: Note that the asker has stated elsewhere that he refuses to learn to read music (don't ask). Of course, the score in this answer will be helpful for others who are curious about the same thing.
Comment: Do I need to separate you two?
Comment: @ToddWilcox Anyway, many of others probably can't hear the second version in their heads, and don't have access to an instrument right now, so a link to a recording or a MIDI would help both OP and other readers.
Comment: @ToddWilcox I said I refuse to learn a song by reading sheet music. it's different when we're talking theory and analyzing then I'm for it. and I think this answer is great. I wish lilybin would use html audio to play the thing instead of me having to download it. also I can't play the midi file with quicktime for some reason so I'm having to drag the midi files to garageband. I'm on a mac.
Comment: @foreyez Sorry, I must have misunderstood when you wrote: "*I'm illiterate in music by design. I don't read or write music and will never do so as I prefer everything by ear.*"
Comment: @ToddWilcox sigh. you know what I mean. funny I actually found a video the other day of another guy that feels the same way. https://www.youtube.com/watch?v=9HoIJK6XW80
Comment: @foreyez All evidence suggests that I didn't actually know what you meant.
Answer: It's a very narrow question.  But, within its terms, yes you're correct.
But that information isn't terribly useful.  Because there are triads other than diatonic ones.  And there are chords other than triads.  And there are 'non-chord tones'.  And all these are VERY commonly used, even in quite simple music.
But if you INSIST on "I want to play a basic triad chord" I can't argue with your logic.
Answer: That is right if you want an easy guaranteed harmony for a single note. When you consider music is a series of notes you also need to consider the sequence of chords you have and their relation to one another. It works playing several notes with the same chord, some that are in the chord and some that are not. Music that only uses "safe" harmonies tends to sound static and boring. Try holding down a chord and seeing how each note of the scale (or out of it)sounds against that chord. Some cry out to you to move them back or forward to a more harmonic note. If the next note of a melody does this it gives a feeling of tension then resolution giving a sense of dynamics in the tune. You can do this all by ear but it helps to know what the intervals between notes are so you can put a name on what you like to help remember it and duplicate it in another key.


Passar valor de um DialogFragment para o Fragment

Question: Tenho o fragment1 e a DatePickerFragment
No caso DatePickerFragment extends DialogFragment
Gostaria que ao selecionar a data retornasse para o fragment.
Só que está dando isso aqui:
java.lang.NullPointerException: Attempt to invoke virtual method 'void android.app.Fragment.onActivityResult(int, int, android.content.Intent)' on a null object reference

O método de recuperação é esse:
No DatePickerFragment extends DialogFragment
@Override
    public void onDateSet(DatePicker datePicker, int year, int month, int day) {
        //...
        SimpleDateFormat data_br = new SimpleDateFormat( "dd/MM/yyyy" );
        String data = data_br.format( date );
        Intent i = new Intent();
        i.putExtra( "selectedDate",data );
        getTargetFragment().onActivityResult( getTargetRequestCode(), 2, i );

}

E no onActivityResult:
@Override
    public void onActivityResult(int requestCode, int resultCode, Intent data) {
        super.onActivityResult(requestCode, resultCode, data);
        Log.i("RequestCode",""+requestCode);
        if( requestCode == 1 ){

        }

    }

Como faço para retornar o valor selecionado no DialogFragment para o fragmento ?
Answer: Implemente no seu Fragment o DatePickerDialog.OnDateSetListener. Veja um exemplo abaixo:
public class ArticleFragment extends Fragment implements DatePickerDialog.OnDateSetListener {
    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
        Bundle savedInstanceState) {
        return inflater.inflate(R.layout.article_view, container, false);
    }
}

Sendo assim, será necessário incluir o método onDateSet sem precisar usar o onActivityResult. O resultado final seria:
public class ArticleFragment extends Fragment implements DatePickerDialog.OnDateSetListener {
    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
        Bundle savedInstanceState) {
        return inflater.inflate(R.layout.article_view, container, false);
    }

    @Override
    public void onDateSet(DatePickerDialog view, int year, int monthOfYear, int dayOfMonth) {
        // aqui você resgata o ano, mês, dia etc...
    }
}

Comment: Você tem a certeza que isso funciona?
Comment: @ramaral aqui comigo funciona em uma Activity. Não testei com Fragment. Na minha cabeça segue mesma lógica. Vou testar mais tarde e confirmo aqui.
Comment: Perguntei porque não estou a ver como o DatePickerDialog sabe que esse método foi implementado no ArticleFragment.
Comment: Testei e testei e consegui chamando o DatePickerFragment dentro do próprio fragment, ai no onDateSet e apenas setei o valor que eu quis, ai funcionou
Comment: @adventistaam como o Ramaral questionou, eu estava testando aqui em um Fragment. Porque na verdade eu tinha feito apenas em uma Activity. Eu já estava a editar a pergunta
Comment: Beleza então! Estou começando nessa área de comunicação entre fragments
Comment: Como o AP diz que funcionou foi procurar como o DatePickerDialog teria conhecimento do *listener*. Ele é informado no construtor: `DatePickerDialog(Context context, DatePickerDialog.OnDateSetListener listener, int year, int month, int dayOfMonth)`.


How do I calculate the reminder (mod)

Question: I want to calculate the reminder for high power value using casio (fx-991es)
Ex:
$$
87^{17} \ \bmod 77
$$
This method not working because the number is too large
Ex:
$$
9/2=4.5=
4-4.5=0.5=
0.5*2=1
$$
Comment: You can repeatedly square 87 and take Mod 77 after each squaring. After squaring 4 times, you'll have raised 87 to the power $2^4 = 16$, so you then need to multiply the result by 87 to obtain $87^{17}$.
Comment: ... and $87\equiv 10$ to begin with
Comment: Also, what is the significance of that $9/2=\ldots$ line?
Comment: Thank you. But I knew this method, Im seeking for direct method
Comment: @Mahmood The direct method is just taking $87^{17}$. Successive squaring is how you do it when that's not possible
Answer: You could go about like this:
$$
87^{17}\equiv_{77} 10^{17}=10\cdot(10^2)^8=10\cdot100^8\equiv_{77}=10\cdot23^8
$$
then since $23^2=529\equiv_{77}67$ we have
$$
10\cdot(23^2)^4\equiv10\cdot67^4
$$
and you keep on like this always reducing the base by breaking up the exponent.
Hope this helps
Comment: $100\equiv 23\not\equiv 33$
Comment: ohh thats true...thanks (it's early in the morning)
Answer: $87^{17} \equiv 10^{17} \pmod{77}$
$10^2 \equiv 23 \pmod{77}$
$10^4 \equiv (10^2)^2 \equiv 23^2 \equiv 67 \pmod{77}$
$10^8 \equiv 67^2 \equiv 23 \pmod{77}$
$10^{16} \equiv 23^2 \equiv 67 \pmod{77}$
$10^{17} \equiv 10^{16} \cdot 10 \equiv 67 \cdot 10 \equiv 54 \pmod{77}$

How do I calculate the reminder (mod)


Is this cadence an example of the German theory that the mediant chord can have tonic function?

Question: From what I understand of German theory the mediant iii chord can have tonic function as the "tonic counter-relative."
In this Bach chorale...

...the second cadence is on iii. The preceding chord is a leading tone chord so it's kind of a deceptive cadence, but instead of going to vi - which I think German theory would call the "tonic parallel" - it goes to the counter-relative.
Just to complete the idea, if the tonic parallel and tonic counter-relative are substitutes for a normal tonic, then the movement revised as a move to plain tonic would be something like...

Is this a correct application of German theory?
Comment: Interesting. There's a flavour of perfect cadence, a flavour of interrupted cadence. Structurally it feels like a cadence point, harmonically it doesn't fit any of the standard cadence types. I'd be loth to class this as anything other than 'unique'.
Comment: Michael - I had forgotten about this chorale.  A great example of a chordal harmonization of 8-7-6-5 in the melody (which we recently discussed), though the accent is less typical on the 7 and 5.  Bach tends to harmonize it mostly with some variation of the I-V-vi-iii "Pachelbel" sequence in the various versions, though not in the iteration for the second phrase where he invariably does something quite different.
Comment: Fascinating example. Do you know of any analyses of this by others?
Comment: @Richard, no. I stumbled upon while answering this https://music.stackexchange.com/questions/115338, which made this chorale interesting enough already by the plagal cadence.
Comment: Further proof—and maybe I'll annoy some people with this—that "Bach-style four-part writing" means nothing. I'd deduct all kinds of points on this chorale if a student submitted it!
Comment: Academia is a funny place :-)
Comment: Certainly grabs the attention, right! The 371 chorales astonish me all the time. I can hardly play all the way through any of them without grabbing my pencil to make notes of the interesting harmony.
Comment: No 'rule' is broken.  Amazing!
Answer: Well, I suppose this could be characterized as a rare and quite unusual application of the German notion of a Gegenklang.  Though first it should be noted that this comes out of German functional theory of the late 1800s (from Riemann), not something J.S. Bach would have had an inkling of.
Just to be clear about the German theory -- let's consider C major, just to be simple.  As Michael notes in the question, a submediant (vi) chord in Riemannian terms would be a "tonic parallel" chord.  At a deceptive cadence in C major, where we'd see G major to A minor (V-vi), we can therefore think of the A minor chord as having some notion of "tonic" functionality.
The Gegenklang concept, on the other hand, comes from Riemann's notion of Leittonwechselklänge, or "leading tone exchange" chords.  In C major, one "exchanges" the leading tone B in place of C, while retaining the other notes of the C major triad (E and G), thus arriving at E-G-B, or E minor (iii).  Yes, in Riemannian theory, E minor thus can sometimes have "tonic" function, though it's much more frequently labeled as a "dominant parallel" (Dp), that is, in the parallel minor relationship to G major.  (Just as C major and A minor have a "parallel" relationship, so do G major and E minor.)
To be clear, it's pretty rare to label an E minor chord in C major as a Tcp using that "leading tone exchange" idea, rather than the much more common Dp.  (For example, we very frequently see cadences in the 19th century with an apparent "iii6" harmony preceding I at a cadence.  It's not really a "iii" chord in any functional sense there; rather, Anglo-American theory would say that the sixth above the bass is substituting for the fifth of the chord, and it's "really" a dominant or V sort of chord variant.)
Where does the Leittonwechselklänge concept come up?  In minor key deceptive cadences.  Forgive me if those reading already know this, but Riemannian theory does interesting things with the minor key and minor chords.  Riemann was a major proponent of the notion of a dualist concept of harmony, where everything was symmetrical in major and minor.  One formed major chords from the overtone series, and one created minor chords by going downward using a (hypothetical) "undertone" series.
Thus, the "root" of a minor triad is what we'd call the "fifth" in Anglo-American theory.  A G-E♭-C triad thus has root G.  Performing a Leittonwechsel (leading tone exchange) thus moves the root G to its "leading tone" of A♭.  (Note the common use of ♭6-5 melodically in minor as a kind of tendency tone -- this lead Riemann to label this as a "leading tone" in minor.)
Using the leading tone exchange in C minor thus creates a triad with notes A♭-C-E♭.  Hence, to explain the deceptive cadence in minor (V-VI), we make use of the tCp notion, where A♭ major is a kind of tonic substitute.
This is the stereotypical use of the "tonic counter-relative" (or whatever English translation you want to use) from German theory, which is mostly a minor key phenomenon.  Both the "parallel" and "leading tone exchange" notions are used in major and minor keys respectively to describe what in English-language theory would usually be described as "submediant" chords.

Okay, now that we've cleared up what the German theory usually refers to, we can go on to the Bach example.  I suppose an argument could be made to use the Tcp symbol (also a T< in Riemann notation, where the < was often drawn across the letter) to describe this use of a iii chord here.  However, I admit I'm not as fluent with the subtleties of how actual Germans would apply these symbols today to know whether this interpretation would be common.  There are other ways of interpreting the function of this chord, which could also be viewed as having a sort of "dominant" (i.e., Dp) function that "resolves" to the next (I) chord.  (That is, the harmony of the measure would be S-Sp-Dp-T, i.e., A♭-Fmin-Gmin-E♭ following a standard S-D-T functional scheme.  My guess is that's how German theory would probably look at it.)
But I think Michael's idea of interpreting the progression as a sort of leading tone chord resolving to iii is certainly possible.
I don't think Bach would have thought of the cadence in that fashion (if that matters).  Rather, I'm pretty sure he'd see it as a variant of an old-fashioned Phrygian cadence.  In modern theory we typically see Phrygian cadences resolve to a local dominant chord, typically a iv6-V progression.  Here, we have the descending semitone in the bass, and the D in the tenor could be also interpreted as an accented passing tone, thereby instead making the harmony a F minor chord in first inversion resolving to G minor on the next beat.  (Note the anticipation in the alto that also emphasizes the F, and the arrival on an Fmin7 harmony on the off-beat preceding.  The harmony in this bar is gradually morphing from a IV harmony on A♭ to an F minor harmony.)  Without the B♮ it doesn't sound like what we'd think of as a stereotypical Phrygian cadence today, but this sort of motion certainly has its roots in the older Phrygian cadences used in modal music (where chorale tunes sometimes came from).  Note the outward motion of the bass and alto to the octave, also a common hallmark of old Phrygian cadences.
Why did Bach use this rather rare and very old-fashioned cadence type here?  I don't know, but there are two elements I'd point to:

All of Bach's other three settings of this chorale tune arrive on a iii chord here.  The other settings all do a local modulation to iii, however.  (One does a straightforward ii7-V7-i in iii.  Another does a iv-IV6-viio-i in iii.  The last does another variant of a iv-V-i in iii.)  Perhaps for some reason Bach had a iii chord "in his ear" for this moment with the melody, perhaps from previous harmonizations of his own or someone else's.  So rather than doing a weak resolution to I6, he throws in this more rare motion to iii.

It could be simple text painting.  People often analyze Bach's chorales without regard to the text, probably due to what I think was a rather stupid idea of printing them in the Riemenschnieder edition without text -- then again, Riemenschnieder claimed to have made "corrections" based on consultation of original manuscripts, but he also chose to leave in the erroneous "corrections" of parallel fifths introduced by previous editors... but I digress.  Anyhow, the actual text to this setting is not "Herzlich lieb hab ich dich," but rather "Ach Herr, lass dein lieb' Engelein" from the St. John Passion.  In the second iteration of the first phrase particularly we get this cadence at the words "Qual und Pein," both words indicating ideas of pain/torment/agony.  Perhaps a bright and cheery resolution to a simple I6 was contraindicated by Bach with such emotional trappings, and he drew on this more ancient and weirder (more tortured?) cadence.

Anyhow, this is all a very long answer that perhaps veered quite a bit away from the original question, but hopefully it sheds some light on how this particular moment was likely viewed by Bach harmonically and how more modern German theory might interpret it.  Yes, it's possible that one could label this particularly mediant chord with a "tonic" function (and if there was a good case in major key for Leittonwechselklänge, this is one of them), though I'm not certain that'd be the most common interpretation using German theory.
Comment: "There are other ways of interpreting the function of this chord, which could also be viewed as having a sort of "dominant" (i.e., Dp)..." in that case would it be regarded as a half cadence? Or equivalent term from German theory.
Comment: @MichaelCurtis - I'm not certain that's the only or preferred method, but I think if you accept my possible interpretation of Sp-Dp, it would be a variation of what German theory calls a "Phrygischer Halbschluss." That is basically a "Phrygian Half Cadence."  As I noted, this example isn't what we typically call a Phrygian half-cadence in modern terms (in English either) as the destination chord is minor.  I honestly don't know how/whether that would affect the German terminology, but I suspect it would be similar to English -- as in, "this is sort of like X kind of cadence."
Comment: OK, and that makes sense with what you already said about the phrygian cadence. Thanks for the detailed answer, especially the part about harmonic dualism and tonic counter-relative typically in minor. It tricky for me, because I learned U.S. theory, but I was able to follow your explanation.


Mining text with small datasets

Question: I'm interning in an auditing company and have audit reports that require categorising. I have been using simple text mining + classification techniques in R (DocumentTermMatrix in tm package, SVM via e1071 package, etc.) to help.
Unfortunately it is laborious to manually categorise the issues to create the train data, but as of now I have about 50+ samples categorised into about 7 categories. A shockingly small number, I know.
What then is the best way to continue? Is 50 too small a size for training data? I have about 150 reports that requires categorising. The reports are each roughly about 900 - 2000 chars (1-2 pages) long (hence laborious to manually categorize).
Comment: Honestly, I'd stick to categorizing them by-hand, or preparing some heuristic rules to categorize them (e.g. existence of some particular terms in documents etc).
Comment: yes, i've thought about heuristics (e.g. certain terms coming up) but it is hard to come up with rigid rules because everyone writes differently and the presence of a term (e.g. "inaccurate") does not determine the report's category (e.g. "inaccurate report")
Answer: Did you try latent semantic analysis? Maybe you will be able to find meaningful patterns when you visualize the data in 2 or 3 dimensions.
If you have mostly unlabeled data then maybe you can leverage it using semi-supervised approach, for example label propagation.

Mining text with small datasets


Simple animations in Ionic Framework

Question: Can I create simple animations in views of ionic framework. The animations would be very simple ones, like changing of colour, fading, increasing height and width of elements, much like animations jquery does in web pages.
Answer: Yes you can, have a look on this.
A powerful javascript animation engine for web and hybrid mobile apps, inspired by Facebook Pop, built by the Ionic team.
https://github.com/driftyco/collide
<code>var animation = collide.animation({
  // 'linear|ease|ease-in|ease-out|ease-in-out|cubic-bezer(x1,y1,x2,y2)',
  // or function(t, duration),
  // or a dynamics configuration (see below)
  easing: 'ease-in-out',
  duration: 1000,
  percent: 0,
  reverse: false
});

// Actions, all of these return `this` and are chainable
// .on('step' callback is given a 'percent', 0-1, as argument (for springs it could be outside 0-1 range)
// .on('stop' callback is given a boolean, wasCompleted
animation.on(/step|destroy|start|stop|complete/, function() {})
animation.once(...) //same event types
animation.off(...) //works like jquery.off
animation.stop(); //stop/pause at current position
animation.start(shouldSetImmediately); //start from current position
animation.restart();
animation.velocity(n) //starts the animation going at the given velocity ,relative to the distance, decaying
animation.distance(n); //distance for the velocity to be relative to
animation.destroy(); //unbind all events & deallocate

animation.isRunning(); //boolean getter

//These are getters and setters.
//No arguments is a getter, argument is a chainable setter.
animation.percent(newPercent, shouldSetImmediately); //0-1
animation.duration(duration); //milliseconds
animation.reverse(isReverse);

animation.easing(easing); //setter, string|function(t,duration)|dynamicsConfiguration.
// Dynamics configuration looks like this one of these:
// animation.easing({
//   type: 'spring',
//   frequency: 15,
//   friction: 200,
//   initialForce: false
// });
// animation.easing({
//   type: 'gravity',
//   bounce: 40,
//   gravity: 1000,
// });
</code>
Comment: Do you know where I might find an example of this API being used in an Ionic App? Is this code that you would add to a directive?
Comment: An example is given in the repo , see *test.html* : https://github.com/driftyco/collide/blob/master/test.html
Answer: Animations in ionic are just the same as if you were working on a standard html page, the only difference is how to set it up in angular.
It is within the controller that is assigned to a view that you can add your code.
<code>.controller('homeCtrl', ['$scope', function($scope) {
    // Jquery animation
    $(selector).animate({params},speed,callback);

    // Greensock animation
    var photo = document.getElementById("photo"); //or use jQuery's $("#photo")
    TweenLite.to(photo, 1.5, {width:100});

    // or define a function and call it when you want an animation to run
    $scope.animate = function(element) {
        $(element).animate({params},speed,callback);
    }
}])
</code>
Thats the gist of it, you can look into more detailed stuff on YearOfMoo
Comment: So are you providing the recommendation to add Greensock or jQuery along with the Ionic Framework? Is that duplicating any code already embedded in the Ionic Framework designed to handle animations? Also isn't it a better practice to add the animation actions and triggers into a directive and not the controller?
Comment: I'm not recommending anything other than what I did to solve my problem at the time, and it worked brilliantly for me :) Ionic doesn't have the animation capabilities greensock offers but Year of Moo shows how to integrate it into angular so it was a good fit for my project.
Answer: Yes. Nothing in Ionic would prevent this. Did you actually try it? Or are you asking if Ionic ships with anything like that? As far as I know the answer would be no.
Comment: I dont know how would I go on about trying this. Normally, I would do these kinds of stuffs in jquery but where do I write jquery in ionic!

Simple animations in Ionic Framework