SARIMAX Exog doesn't work well at all?

Question: I'm working with demand forecasting and we are using Statsmodels SARIMAX on a regular basis. One thing we have noted is that the model doesn't seem to listen much at all to the EXOG argument to which we pass a seasonal decompose.
I therefore made a test with deterministic synthetic data, where I pass the target variable and a small degree of random noise as exog, and make percent offsets per month every year to see if the model listens to this "seasonality" passed as exog.
My conclusion so far is that when having extremely low noise ("uncertainty" in exog), the model listens, but when adding a reasonable level of noise/uncertainty, the model only very carefully nudges the forecast in the right direction.
Am I misunderstanding how to use Exog, or how the optimization works under the hood?
Do you have any ideas on how to get it working better?
(Our current approach has been to blend it with a long term model that listens better to the seasonality and weight the blending with days ahead in the forecast.)
Python code to reproduce
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
import numpy as np
plt.rcParams.update({'figure.figsize':(20,4), 'figure.dpi':120})

def generate_data(
    weekday_offset=False,
    month_offset=False,
    freq='d',
    start='2017-12-01 00:00',
    stop='2021-01-07 23:00',
    start_value=100,
    label='target'
):

    start = pd.to_datetime(start)
    stop = pd.to_datetime(stop)

    # Create placeholder df
    df = pd.DataFrame(index=[start, stop])
    df = df.asfreq(freq)

    df[label] = start_value

    # Offset day of week
    if weekday_offset:
        weekday_offset_dict = {
            0: 1.2,
            1: 1.1,
            2: 1.05,
            3: 1,
            4: 1,
            5: 0.90,
            6: 0.90
        }
        for weekday in weekday_offset_dict.keys():
            df.loc[df.index.weekday == weekday, label] = (
                    df.loc[df.index.weekday == weekday, label] * weekday_offset_dict[weekday])

    # Offset by month
    if month_offset:
        month_offset_dict = {
            0: 1.15,
            1: 1.2,
            2: 1.15,
            3: 1.1,
            4: 1.05,
            5: 1,
            6: 0.95,
            7: 0.8,
            8: 0.85,
            9: 0.9,
            10: 1,
            11: 1.05,
            12: 1.1
        }
        for month in month_offset_dict.keys():
            df.loc[df.index.month == month, label] = (
                    df.loc[df.index.month == month, label] * month_offset_dict[month])

    return df

# Generate synthetic data
df = generate_data(weekday_offset='True', month_offset=True, freq='d')[['target']]

# Add some noise to target column, this col with noise will be used as exog
df['noise'] = np.random.normal(
    loc=0,
    scale=2, # <-- TRY NOISE LEVELS HERE
    size=len(df)
)
df['target_noise'] = df['target'] + df['noise']
df = df.drop(columns=['noise'])

# Plot noise comparison
df[['target', 'target_noise']].plot(title='Noise to actual comparison')

# Simple train test split
split = pd.to_datetime('2020-06-15')
train = df.loc[df.index<split]
test = df.loc[df.index>=split]

# Define and fit model including exog
params = {'order': (1, 0, 1), 'seasonal_order': (0, 1, 1, 7)}
m = sm.tsa.statespace.SARIMAX(train[['target']], exog=train[['target_noise']], **params)
model = m.fit(disp=0)

# Define and fit model excluding exog
m2 = sm.tsa.statespace.SARIMAX(train[['target']], **params)
model_no_exog = m2.fit(disp=0)

# Define timeframe for forecast
start = split
days_ahead = 100
end = start + pd.Timedelta(days=days_ahead)

# Slice exog for forecast
exog = test.loc[start:end][['target_noise']]

# Make forecasts with and without exog
pred_exog = pd.DataFrame(columns=['exog'], data = model.predict(start, end, dynamic=True, exog=exog))
pred_no_exog = pd.DataFrame(columns=['no_exog'], data = model_no_exog.predict(start, end, dynamic=True))

# Compare them in plot
compare = pd.concat([pred_exog, pred_no_exog, test], axis=1, join='inner')
compare['exog'] = compare['exog'] + 1 # Visibility dummy for plot
compare[['exog', 'no_exog', 'target']].plot(title='Check if listens to exog')
plt.ylim([60, 125])
plt.show()

Answer: The basic issue here here is that your left-hand-side variable, df['target'] is peculiar in a number of ways, and in particular it has no noise. So if you look at the 7-period difference, you get:

With the exception of a relatively few periods, this is very easy to predict (it's equal to zero).
If I add in the the 7-period difference of df['target_noise'], it looks like:

The orange line here is not helpful at predicting most of the periods (which are zeros), although it is somewhat helpful at predicting the periods with jumps.
When you include 'seasonal_order': (0, 1, 1, 7), you are essentially allowing the model to work with the 7-period difference. As a result, it simply is not using df['target_noise'] because it doesn't need to.
If you drop the seasonal order, you'll see that the model uses df['target_noise'] to make its predictions.
Comment: Thanks for taking the time Chad, that clarifies a lot! : - ) I hadn't realised that the differencing is applied to exog as well.
Comment: I can also note that after recalculating our seasonal decompose of the seasonality in the real data, making it multiplicative in absolute numbers and using a trend that we extrapolate into the future as well, then Exog starts making more sense and the model listens a lot to it. So seems the question is solved! :-)
Comment: @ml_enthusiast The differencing is applied to the exogenous regressors as well because `statsmodels`' "SARIMAX" is *not* a SARIMAX model at all, it's a "regression with SARIMA errors". In an actual SARIMAX model, no such transformation would be applied to the exogenous regressors.
Answer: You could find useful the following link "Comparing trends and exogenous variables in SARIMAX, ARIMA and AutoReg"
The article addresses details on the statsmodels implementation of SARIMAX and ARIMA models, specifically when using exog variables, there are code snippets and interesting sections as how "Reconstructing residuals, fitted values and forecasts in SARIMAX and ARIMA"
https://www.statsmodels.org/dev/examples/notebooks/generated/statespace_sarimax_faq.html

SARIMAX Exog doesn't work well at all?


Hover somente em algumas colunas

Question: Tenho uma tabela com 6 colunas, mas só quero dar hover nas 4 primeiras colunas. É possível fazer isto?
Estou usando MVC 4 e bootstrap
Já estou usando um método em Jquery:
$(".table tbody tr").hover(function () {
        $("table tbody tr .cell-padrao").each(function (index) {
            $(this).css("background-color", "#FFF");
        });

        $(this).each(function (index) {
            $(this).find(".cell-padrao").css("background-color", "#eee");
            $(this).find(".cell-padrao").css("cursor", "pointer");
        });
    });

Mas toda vez que eu tiro o mouse da Linha o hover continua lá. Preciso que ele saia. Dá pra melhorar este método ou tem algo em CSS que possa ser feito?
EDIT:
Preciso que ele dê hover nas 4 primeiras(juntas) colunas e as outras 2 colunas fiquem como estão, pois possuem hovers proprios
Exatamente assim:
http://uploaddeimagens.com.br/imagens/grid-png--3, mas quando sair da tr o hover suma também
Answer: isso pode ser feito diretamente no CSS
HTML:

        Coluna 1
        Coluna 2
        Coluna 3
        Coluna 4
        Coluna 5

CSS:
.colorido:hover{color: red;}

DEMO

Segunda opção com todos os valores selecionados:
HTML

        Coluna 1
        Coluna 2
        Coluna 3
        Coluna 4
        Coluna 5

Javascript/jQuery
$("table").mouseenter(function(){
    $(".colorido").css('color','red');
}).mouseleave(function() {
    $(".colorido").css('color','black');
});

DEMO

Terceira e melhor opção com adição e remoção de classes:
HTML

        Coluna 1
        Coluna 2
        Coluna 3
        Coluna 4
        Coluna 5

Javascript/jQuery
$("table").mouseenter(function(){
    $(".colorido").addClass('novaClasse');
}).mouseleave(function() {
    $(".colorido").removeClass('novaClasse');
});

CSS
.novaClasse{color: red; cursor: pointer;}

você pode ainda dar um hover nessa nova classe alterando o CSS para
.novaClasse{color: red; cursor: pointer;}
.novaClasse:hover{color: green;}

DEMO
Comment: Dessa maneira ele dá hover e coluna por coluna, eu preciso que quando ele passar o mouse em cima da primeira até a quarta coluna ele dê hover nas 4 juntas.
Comment: atualizei a resposta de acordo com sua duvida
Comment: @ThiagoAlex Por favor edite a pergunta para deixar isso claro, pois não dá para saber que essa é a sua intenção sem ler seu comentário acima. Veja que a maioria das respostas que você recebeu não faz o que você quer.
Comment: de fato não deu pra entender a pergunta. Acredito que as opções que deixei acima vão ajudar a resolver seu problema Alex, caso for util, não esqueça de positivar a resposta.
Comment: @ThiagoAlex isso resolveu o seu problema ? não esqueça de finalizar o topico.
Answer: Você pode usar somente css nth-last-of-type(-n+1)
Ficaria:
table tr td:not(:nth-last-of-type(-n+2)):hover{
    color:red;
}

JS Fiddle
Answer: Para a melhor compatibilidade com os browseres, recomendo que faça isso usando o seguinte CSS:
table tr td:first-child:hover,
table tr td:first-child + td:hover,
table tr td:first-child + td + td:hover,
table tr td:first-child + td + td + td:hover
{
    color: red;
    cursor: pointer;
}

Dessa forma vai funcionar até mesmo no IE8.
Exemplo no jsfiddle
jsfiddle embedded: permite visualizar no IE8
Assim, você evita usar javascript para fazer esse efeito, e dá um bom suporte aos browsers.
Answer: Imagi no que voce nao queira apenas as 4 primeiras, mas na realidade vincular X linhas a um determnnado contexto, podendo ser 1, 2 ou N linhas. Desta forma considero que utilize um identificador de contexto ex: data-codigo em cada linha e na função de hover vc identifica as linhas que tem o mesmo contexto e destaca todas de uma vez:
Solução sugerida: JS FIDDLE

Contexto 1

Contexto 1

Contexto 1

Contexto 1

Contexto 2

Contexto 2

Contexto 2

Contexto 2

Ajustei seu JS para tratar esses casos:
$(".table tbody tr").hover(
        function () {
            var jqTab = $(this).closest('table');
            jqTab.find("tr[data-codigo=\"" + $(this).data('codigo') + "\"]").css("background-color", "#FF0");
        },
        function() {
            var jqTab = $(this).closest('table');
            jqTab.find("tr[data-codigo=\"" + $(this).data('codigo') + "\"]").css({
                 "backgroundColor": "#CCC",
                 "cursor": "pointer"
            });
        }
 );

Answer: Faça mais um evento baseado no .mouseleave():

http://api.jquery.com/mouseleave/

Hover somente em algumas colunas


Regras de negócio sempre são relacionadas à validação?

Question: Desde que comecei a estudar orientação a objetos ouço muito sobre as regras de negócio. Basicamente, pelo que entendi até hoje, um objeto deve ter métodos encapsulado as regras de negócio e a modificação do seu estado deve ser feita através desses métodos e não através de acesso direto à seus atributos. Pelo que eu percebi existe tanta ênfase nisso que um objeto sem métodos é até chamado de anêmico.
O problema maior é que eu até hoje não entendi muito bem o que são realmente essas regras de negócio. Todos os exemplos que eu vi, as regras de negócio tem a ver com validação. Um desses exemplos é uma classe Usuario representando um usuário de um sistema, e as regras de negócio impostas no exemplo tem a ver com validações dos dados (garantir que sejam informados dados requeridos, garantir que o e-mail seja um e-mail válido, etc).
Outro exemplo muito usado é de sistema bancário e aí as regras de negócio são outra vez validações: por exemplo verificar se o saldo da conta é suficiente pra fazer um saque e etc. Dessa forma parece que as regras de negócio sempre são validações pra verificar se certa operação pode ou não ser realizada ou verificar se os dados informados são realmente consistentes.
Essa impressão está correta? As tais regras de negócio sempre tem a ver com validações? Se não, o que realmente são as regras de negócio?
Answer: De fato, é muito comum que regras de negócio sejam validações, mas não necessariamente.
A interpretação do que é regra de negócio e o que é mecanismo nem sempre é muito clara (fiz uma pergunta e não obtive respostas satisfatórias), e dependendo da metodologia utilizada pode haver a preferência por uma coisa ou outra.
Na verdade, se pensarmos nos tipos de objetos também há divergências sobre o que deve fazer parte do tipo como regra de negócio e o que deve ser algo externo ao tipo ainda sendo uma regra de negócio. Muitas vezes isto é implementado através de regras (rules), restrições (constraints), políticas (policies) ou tratamentos (traits). Que nada mais são que outros tipos auxiliares ao tipo principal. Algumas vezes um mecanismo de regras de negócio é utilizado para isto.
Pegando o exemplo da conta bancária, você já sabe as várias validações que existem. Mas o objeto só valida dados? Ele não pode executar nada?
Vamos dizer que você tenha um método chamado ObtemContasInativas(). Isto não seria uma regra de negócio? Não é validação. Nela consta a forma como uma conta é considerada como inativa e como fornecer dados delas. Note que quais informações e em que ordem também são regras de negócio incrustadas dentro deste método hipotético. Na verdade, é possível que este método faça uso de outro para saber se uma conta individual está inativa ou não, provavelmente um EhInativa().
Muito provavelmente o primeiro método não está na classe da conta bancária em si, não faria sentido estar. Mas ainda é uma regra de negócio. Já o segundo método já faz mais sentido estar no tipo da própria conta bancária. Mas não necessariamente. Existem algumas razões para se optar por não estar. Mas isto não vem ao caso aqui.
Outro método poderia ser EncerrarConta() que poderia ser chamado automaticamente por algum evento, talvez verificado por um método InativoMais180Dias() ou manualmente por alguma função do sistema. E obviamente ela faria uma série de procedimentos para garantir que a conta se encerre. Tudo isto são regras de negócio.
Exatamente como vai acessar o banco de dados para pegar estas informações, para gravar o que for necessário para o encerramento provavelmente faz parte do mecanismo e não entra nestes métodos. Eles devem delegar estas especificidades para outros métodos de mecanismos, muito provavelmente em outras classes.
Então percebe-se que regras de negócio podem sim determinar como informações são obtidas, como executar determinados algoritmos, etc. Não são apenas verificações se o dado que está tentando entrar está certo ou errado.
Até mesmo o que fazer no caso de uma tentativa de entrada de dado inválido pode ser uma regra de negócio por si só. Deve logar? Deve chamar algum processamento auxiliar? Deve mudar algum estado de algum objeto?
Regra de negócio é qualquer coisa que determine o que você deve fazer em qualquer situação manipulando (lendo ou escrevendo, total ou parcial) aquela informação em uma forma abstrata. Em uma forma que faça sentido para o usuário.
Quando a manipulação fica mais concreta, como WriteOnDataBase() ou DisplayOnScreen() provavelmente já fazem parte do mecanismo. Normalmente são coisas que fazem sentido para o desenvolvedor apenas, mas são fundamentais para o sistema funcionar.
Então você pode validar, garantir (ligeira diferença semântica), registrar, consultar, classificar, condicionar, relacionar, executar, verificar, permitir/proibir, calcular, delegar, derivar, sinalizar, observar, etc. A quantidade de ações possíveis, mesmo que de forma genérica, como listada, pode continuar.
Coloquei no GitHub para referência futura.
Comment: Então métodos como, `obterClientesInativos()`  ou `listarContaSalario()`  que obtêm informações especificas de algum objeto pode ser considerados regras de negocio, certo?
Comment: Genericamente falando são sim.
Answer: Validações são, sim, regras de negócio. Mas existem outras.
Considere, por exemplo, outras regras de negócio do banco que vão além da verificação de saldo na conta antes de permitir um saque:
Um banco fornece empréstimos. Você vai ao banco e pede 100 mil reais para pagar em 10 vezes. O sistema vai automaticamente avaliar se você pode tomar este empréstimo e também a taxa de juros. Para tanto, as regras de negócio de liberação de crédito são acionadas. Elas incluem:

Uma parcela mensal de 10 mil reais é maior que 30% da sua renda?
Você tem sido um bom pagador em empréstimos anteriores?
O montante geral de clientes com perfil parecido com o seu tem quitado suas dívidas?
Qual é a taxa de juros básica em vigor?
...

Um cálculo complexo é feito considerando estas e outras informações e o empréstimo é liberado desde que aprovado por alguém com alçada superior ao gerente da sua conta. Então este gerente encaminha o empréstimo para aprovação, que entra em uma fila e vai ser avaliado por outro gerente, o que também é uma regra de negócio.


Recommendations for textbooks covering current data mining/machine learning techniques for fraud detection?

Question: I work in the health insurance field, but a general treatment of fraud detection methodologies would still be helpful.
So far I've discovered brief articles outlining particular techniques used in fraud detection, but not a comprehensive overview.
Answer: [Note: I wanted to simply "add a comment" here but the system is not allowing me]. I do not have particular experience with fraud detection specifically but my understanding is that it is closely related to anomaly and outlier detection and my response here reflects that understanding. Anyway, this paper provides a pretty comprehensive overview of various anomaly detection techniques, especially those prevalent in the machine-learning/pattern-recognition literature. Note, however, the paper is a little dated (from 2007) so you might want to find an additional reference for newer methods. Another potential reference might be this book.
Comment: +1 Thank you, this is helpful. I'm guessing that part of the reason there are no textbooks specifically describing fraud detection is that analysts are simply using existing statistical/machine learning/data mining anomaly detection techniques off the shelf. Another reason - homebrewed methodologies, plus information about which features analysts typically model for fraud detection, may be proprietary knowledge. Companies may not want to tip off potential fraudsters by revealing their "secret sauce" for fraud detection.


Uncertainty in spatio-temporal forecasting

Question: We have a time series gridded panel dataset (spatio-temporal). The dataset is in 3D, where each (x,y,t) coordinate has a numeric value (such as the sea temperature at that location and at that specific point in time). So we can think of it as a matrix with a temporal component. The dataset is similar to this but with just one channel:

I am trying to predict/forecast the nth time step values for the whole region (i.e., all x,y coordinates in the dataset) given the values for the n-1 time steps and the uncertainty.
Can you all suggest any model/architecture/approach for the same? I was initially thinking of the Log-Gaussian Cox Process but it seems it's mostly applicable to point processes, while my dataset is gridded with each grid having a numeric value.
Answer: Markov random fields address "neighborhoods", in your case neighbors in time and space. In computer vision, four-neighbor (N-E-S-W) and eight-neighbor models are prevalent (N-NE-E-SE-S-SW-W-NW).
If necessary (start first with a simple model), you could model larger neighborhoods with "filters", using techniques similar to Zhu, Wu, and Mumford. "Filters, random fields and maximum entropy (FRAME): Towards a unified theory for texture modeling." International Journal of Computer Vision 27 (1998): 107-126. I define several hundred element "neighborhoods" with exchange traded funds (ETFs) in my paper Keane "Portfolio Variance Constraints." ICPRAM. 2019.
A nice framework for n-th time step values is West and Harrison. Bayesian forecasting and dynamic models. Springer Science & Business Media, 2006. The framework uses discrete timesteps as you do, time-of-day or time-of-year  cyclical components are addressed, as are matrix variate models.


How to store some info while user is submitting rating with Fivestar module?

Question: I want to store this rating on different site and I am doing this with the help of RESTful api. I have altered "fivestar_custom_widget" form with hook_form_alter. My call is not working beacuse it is not removing div which has 'field-name-field-fivestar-rating' class and not adding div which has 'rating-wrapper' id.
function recipe_form_alter($form, $form_state, $form_id) {
  if ($form_id == 'fivestar_custom_widget') {
    $form['#submit'][] = 'fivestar_rest_api';
  }
}

function fivestar_rest_api($form, $form_state) {
   $ch = curl_init("http://www.example.com/");
   curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
   curl_setopt($ch, CURLOPT_HEADER, 0);
   $data = curl_exec($ch);
   curl_close($ch);
   $json = json_decode($data,true);
   $errorMessage = $json['errorMessage'];
   $errorCode = $json['errorCode'];
   $commands[] = ajax_command_remove('.field-name-field-fivestar-rating');
   $commands[] = ajax_command_append('#rating-wrapper', 'Thankyou for your rating.');
   return array('#type' => 'ajax', '#commands' => $commands);
 }

Answer: Submit handler was not working so I have decide to do it with validate handler like this

function recipe_form_alter($form, $form_state, $form_id) {
  if ($form_id == 'fivestar_custom_widget') {
    $form['#validate'][] = 'fivestar_rest_api';
  }
}

function fivestar_rest_api($form, $form_state) {
   $ch = curl_init("http://www.example.com/xxx");
   curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
   curl_setopt($ch, CURLOPT_HEADER, 0);
   $data = curl_exec($ch);
   curl_close($ch);
   $json = json_decode($data,true);
   $errorMessage = $json['errorMessage'];
   $errorCode = $json['errorCode'];
   $commands[] = ajax_command_remove('.field-name-field-fivestar-rating');
   $commands[] = ajax_command_append('#rating-wrapper', 'Thankyou for your rating.');
   return array('#type' => 'ajax', '#commands' => $commands);
 }

Now it is working fine form me.


CSS aggregration breaks site - how to troubleshoot?

Question: I have a number of D7 sites that "broke" recently when I applied 7.39 and a few other updates. By "broke" I mean the site's CSS styling completely disappeared. I can easily fix by disabling CSS aggregation, so I know that's the cause.
At first I thought it might be a Omega 3.x problem since all sites initially affected, but then a site based on Bootstrap had the same problem. (The CSS doesn't always break right away).
What steps can I take to troubleshoot to determine the root cause?
Edit - to add more detail based on Clive's questions below:

Page source does contain <link>'s to aggregated CSS files (5 total)
Aggregated CSS files do exist on server
Status codes are 304 (4) and 200 (1)
Folder permissions seem OK
Comment: maybe there is several reason for this problem but I think the aggregation cant create fie in public directory or you have a syntax mistake in one of your CSS  files
Comment: What were your "few other updates"?
Comment: I had 6 sites total break, and I believe the only updates in common with all were  Drupal 7.39 and ctools 1.9
Comment: Is anything showing up as red or yellow on the Status Report?
Comment: Status report looks good.
Answer: Some initial things to check:

Clear the cache (if you're running advagg, clear its cache). You can end up with stale file references sometimes.
Find out what's happening to the missing CSS:

When the styles go missing, does the page source contain a <link> to an aggregated CSS file? If not, check for modules/themes implementing hook_html_head_alter(), hook_css_alter(), or hook_preprocess_html(), and see if they're doing anything strange.
If the source does contain the <link>, what status code is the browser returning for the request? 404? You might have a stale cache issue. Clear all caches including those on any reverse proxy you're running. It might easily be something else, would require more intimate debugging. 500? Check the server logs, that should give you clues as to where things are failing.
If it's 200, does the requested file actually contain the styles it's supposed to? If not, you might be looking at a cache issue again, or potentially an issue with build tools (Compass/Gulp/Grunt/etc).

Switch the theme to a core one. Does it have the same problem? If so, it's probably a module or server setting at fault. If not, it's likely to be a problem with the theme.
Make sure permissions for your files folder are at the right level - the web server must be able to write to that folder to store the aggregated files. This is less likely to be an issue if the files are being created successfully, but it's worth double-checking you don't have an intermittent file system issue (if it's over NFS for example)

Getting the answers to those questions will help to narrow down the source of the problem, but unfortunately it's difficult to give specific advice as to where you should look in the code, as there are a lot of points problems could occur, and a lot of different modules you could have installed, at varying versions.
Comment: Great list, Clive, thanks. I've got a new problem now in that I can't reproduce (so far). What's odd is that it was reproducible a few days ago when I first discovered.
Comment: @DaveBruns - Most likely a cache issue. To avoid missing css in the future the [advagg module](https://www.drupal.org/project/advagg/) should be used.
Comment: I was able to reproduce on one site and added some more info above. @mikeytown2 - I've had advagg recommended before and will take a look - thanks.
Answer: I had this happen recently on a Backdrop site and had to really dig around to find the answer. Adding it here in case anyone else runs into this and none of the other answers here solve it.
As @Doomd writes here, using @import with Google fonts can cause the aggregation to break, apparently due to semicolons in the URL.

One thing that broke the css was a google fonts import. It seems the semicolon that separates the different font weights in the google fonts url confuses the aggregator:

@import url("https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;700&display=swap");

So I changed it to this:

@import url("https://fonts.googleapis.com/css2?family=Open+Sans:wght@400&display=swap");
@import url("https://fonts.googleapis.com/css2?family=Open+Sans:wght@700&display=swap");
Answer: I am experiencing this problem as well. I'm adding a new answer since I don't have enough reputation to comment on Clive's response.
I don't have a final fix for this, but I have found the following:

The CSS from my theme (plus the base theme and module CSS) is being aggregated and loaded into the page as I would expect. I can inspect it in Chrome's development tools and confirm that it's there.
However, only the CSS from the first CSS file in my theme is actually being applied. I'm using Adaptive as a base theme, so this means global.base.css.
If I copy my CSS overrides into this file, they are applied to the site. This is a fix of sorts, since it now allows me to aggregate the CSS without any problems. It's an imperfect fix since the issue/bug remains.
The issue relates to core CSS aggregation. Installing Advagg neither helps nor hinders.
I ran CSS validation on my CSS overrides. I didn't see any showstoppers. However, the fact that the code works fine in the first CSS file listed in my theme's info file but not in the fifth file when aggregation is enabled, suggests it probably isn't a validation issue.

One possibility is that there is a bug in AdaptiveTheme that breaks when D7.39 and/or CTools 1.9 are installed. I haven't ruled that out yet. But I have other sites with this combination that are not affected so I'm disinclined to point the finger at AdaptiveTheme.
I'm now working through the modules installed on this site and a similar site that isn't affected to try and find a module that might be causing the problem.
Comment: After further investigation, it appears that there is one CSS file in my theme that is causing the problem. This file is the second file in the theme. I tried moving my other CSS files above this file, and it seems to have resolved the issue.
Comment: The file causing the issue is global.style.css, which is part of at_subtheme. I replaced it with the original file from at_subtheme, which also resolved the issue.

So at this point, I'd say it's some faulty CSS in my copy of that file, which I now need to track down and remove.
Comment: That sounds like a bad bracket somewhere.
Comment: Thanks ... yes, it turned out I'd transposed a ; and ) in a single line of code.
Comment: Similarly, I just experienced this due to an unclosed comment at the beginning of a file (which itself only contained a few styles that were overridden elsewhere, so the fact that those styles weren't being applied wasn't apparent.) So checking for invalid code is definitely worth doing.
Answer: Problems with aggregated CSS files are very often because of unclosed or mismatched curly brackets in one of the individual files.
When there is an unclosed curly bracket in 1 out of 15 CSS files, the error is limited to that file, and some browsers will assume the closing bracket at the end of the file. The other 14 correct CSS files will apply normally.
When those 15 CSS files are concatenated together through aggregation, that unclosed bracket now contains all of the style rules that appear after it. They will not be evaluated.
You may want to check each individual CSS file. A linter or a good IDE with error highlighting can work wonders.
Answer: After inspecting the .htaccess file, I got it working by removing this:
<IfModule mod_headers.c>
    # Serve gzip compressed CSS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.css $1\.css\.gz [QSA]

    # Serve gzip compressed JS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.js $1\.js\.gz [QSA]

    # Serve correct content types, and prevent mod_deflate double gzip.
    RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1]
    RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1]

    <FilesMatch "(\.js\.gz|\.css\.gz)$">
      # Serve correct encoding type.
      Header set Content-Encoding gzip
      # Force proxies to cache gzipped & non-gzipped css/js files separately.
      Header append Vary Accept-Encoding
    </FilesMatch>
  </IfModule>

I am still going to dig a bit deeper to find out why, but for the time being, this solves my problem.
I think that the aggregation module also tries to compress files. So by having this in the htaccess file, you are gzipping twice. I THINK. I could be wrong. Either way, this solves it.
Answer: I stumbled upon this with D9, trying to add a css resource to a library.
I eventually solved it adding a couple of properties to that css :
global-styling:
  css:
    theme:
      css/booswatch_flatly.min.css: { minified: true, preprocess: false}
      ...


Probability of sum of independent normally distributed variables

Question: The weight of an adult swan is normally distributed with a mean of 30 pounds and a standard deviation of 9.8 pounds. A farmer randomly selected 36 swans and loaded them into his truck. What is the probability that this flock of swans weighs > 1010 pounds?
How can I solve this question? I can do it for one swan but how can I do it for a flock?
Comment: Because this is a homework/self-study question, you should add the `self-study` tag. See http://stats.stackexchange.com/tags/self-study/info
Comment: Not only should you see the `self-study` tag as Patrick suggests, you should add it to your tags, read its [tag-wiki](http://stats.stackexchange.com/tags/self-study/info) and modify your question to follow the guidelines on asking such questions. In particular, you'll need to clearly identify what you've done to solve the problem yourself, and indicate the specific help you need at the point you struck difficulty.
Answer: This doesn't require any appeal to the central limit theorem, you only need to use the fact that a sum of independent normal random variables is itself normal with mean and variance equal to the sum of the means and variances of the summands.  So if $X_i$ is the weight of the $i^\text{th}$ swan we have $S \equiv\sum_{i=1}^{36} X_i \sim$ normal$(\mu = 1080, \sigma^2 = 3457.44)$.  Therefore
\begin{align}
P(S > 1010) &= 1 - P(S \leq 1010) \\
&= 1 - P \left ( \frac{S - 1080}{\sqrt{3457.44}} \leq \frac{1010 - 1080}{\sqrt{3457.44}} \right ) \\
&= 1 - P(Z \leq -1.19)
\end{align}
where $Z \sim$ normal$(0, 1)$.  The rest can be done by either using software or consulting a standard normal table.


Is there a function to transform yyyyddd to yyyy-mm-dd from a csv file?

Question: I'm trying to convert dates in a csv file from yyyyddd (where day is day/365 in the year) into yyyy-mm-dd. I've been trying to experiment with datetime, but it seems this functions works with data created in python, not in a csv that was imported in.
Comment: What are some of the things you have tried?
Comment: Welcome to StackOverflow. To help you better we need some example data. 5 rows is enough to help you. See more information here about making a good `pandas` question: https://stackoverflow.com/questions/20109391/how-to-make-good-reproducible-pandas-examples
Comment: The format specifier for `ddd` is `%j` When reading the date in `YYYYddd` format, you would set your parser to `%Y%j`
Answer: The datetime library can be used to parse the YYYYddd format into YYYY-mm-dd.
<code>>>> from datetime import datetime
>>> date = '2019001'
>>> parsed = datetime.strptime(date, '%Y%j')
>>> print(datetime.strftime(parsed, '%Y-%m-%d'))
2019-01-01
</code>


How to work out this probability (100 people, 2/3 chance of being correct, 55-45 split)

Question: I'm trying to work out the answer to this Q, and i'm stuck.
Assume that we have 100 voters and each voter has a two-thirds chance choosing correctly between binary options. We are conducting a binary vote between A and B. Further suppose that the difference in votes between the two positions is a 5% split: 55 voters choose A and 45 choose B. The voters are both independent and sincere.
How confident should we be, based on this distribution, that option A is correct?
Comment: My Bayes-sense is tingling! Start by thinking: Assuming that option A is correct, what is the probability that 55 voters choose A and 45 voters choose B?
Comment: @MattiP. That's relatively low, just because there are so many possible outcomes. One usually says "What is the probability that _at least_ 55 voters choose A" (or possibly "at most", depending on what you're after) instead.
Comment: Are the voters making independent choices?
Comment: Yes! sorry, I should have included that
Comment: We don't have enough information. Even if voters are independent, this doesn't imply that given A is the right choice, the probabilities of their answers are not independent.
Answer: If the voters are independent, given that $A$ is correct the chance of getting this vote is ${100 \choose 55}\frac {2^{55}}{3^{100}}$.  Given that $B$ is correct the chance of getting this vote is ${100 \choose 45}\frac {2^{45}}{3^{100}}$.  This gives the chance $A$ is correct as $\frac {1024}{1025}$ as the binomials are equal.  If the voters are not independent we need to know about the correlations to be able to say anything.


Fingering for arpeggio in Scriabin's piano sonata no. 1

Question: What fingering is recommended for the righthand section here? I'd originally been using 1-2-3-1-2-3-1-2-3-4-135, but at around dotted quarter = 80-85 it's started to feel quite difficult and I've hit a bit of a wall on getting it faster, and I've been playing with using 1-2-4-1-2-4-1-2-4-5-134. The arpeggios themselves seem a bit more natural (which makes sense, I realize 1-2-3 was not in line with standard arpeggio fingerings -- I've been away from the piano for many years and wasn't really thinking) but the 4-5-134 bit is proving difficult to get clean, although I've only been practicing this fingering for a day or two.
I just want to get others' thoughts on the fingerings before I spend too much time trying to refine the 1-2-4 one only to hit a wall like I did with 1-2-3 and realize fingering wasn't my problem.

I have medium-sized hands, can reach a comfortable 9th and an uncomfortable white-key 10th.
Answer: I think you're on the right track.
For the arpeggio, 1-2-4 makes the most sense. It takes advantage of the space between fingers 1 and 2 and the greater space between 2-4 than 2-3.
For the final repetition within the arpeggio, however, I'd switch to 1-2-3, which is to say 1-2-3-4-135 for the end of the passage. If necessary, some very light pedal could be used to ease the "stretch" from 2-3 if a break in the sound can't be avoided fingers-only.
Comment: @voskresenie Glad it's helpful. The key to remembering the switch from 1-2-4 to 1-2-3, in my experience, is to be aware of where you're headed. When conscious of the chord coming up at the end of the run, I find it easier to make the fingering adjustment. In fact, as a practice technique, try placing the chord early: in other words, practice 1-2-4, 1-2-4, 1-2-135. That will help ensure your entire hand is in the right place at the right time.
Comment: @voskresenie Perfect. You understand both the technique and the purpose. It's a good tool in general. I frequently will play (parts of) a passage as a block chord in order to determine the best hand position.
Comment: Great, that was the other thing I'd played with a tiny bit but I was having more trouble switching from 1-2-4 to 1-2-3 at the end than either fingering individually -- but again, I haven't spent a ton of time on these new fingerings yet, so with practice will probably be fine. My bigger problem with 1-2-3 wasn't the 2-3 stretch, it was passing the 1 under after -- to make the stretch more natural, my palm was angled inward slightly, then I had to straighten it to get the 1 there cleanly. But the last instance has no thumb pass so that shouldn't be an issue.
Comment: Ahh, that's an interesting idea -- you're saying, practice skipping the last F-G and go straight to the chord? I hadn't thought of that but I think that's a good idea. The problems I've had with hitting the chord cleanly at faster tempos have mostly been from focusing on the two notes leading up to it and then my thumb isn't over the Ab (hovering too close to edge of keyboard) when it needs to be and I kinda frantically smash it over. That exercise will I think be helpful in addressing that (as well as the 1-2-4 to 1-2-3 switch).
Comment: Superb recommendations, thank you. Will practice this way for a few days and reply again with how it goes!
Comment: @voskresenie Please do post a follow-up. I'd love to hear how it works (or doesn't) for you.


How to use props and watch in vue?

Question: I am a beginner with VueJS and I use vue-chessboard library for my project. But I am so confuse about the way to use of the library, I see that when we use showThreats, then we can use ":showThreats", but we do not use ":" for "orientation". There are code of vue-chessboard:
vue-chessboard
export default {
  name: 'chessboard',
  props: {
    ...,
    showThreats: {
      type: Boolean,
      default: false,
    },
    onPromotion: {
      type: Function,
      default: () => 'q',
    },
    orientation: {
      type: String,
      default: 'white',
    },
  },
  watch: {
    fen: function (newFen) {
      this.fen = newFen
      this.loadPosition()
    },
    orientation: function (orientation) {
      console.log('watch orientation________', orientation)
      this.orientation = orientation
      this.loadPosition()
    },
    showThreats: function (st) {
      this.showThreats = st
      if (this.showThreats) {
        this.paintThreats()
      }
    },
  },
  methods: {...
  }
}

when I use this library
<chessboard :orientation="black"/>

Then the browser will notice:
vue.runtime.esm.js?2b0e:619 [Vue warn]: Property or method "black" is not defined on the instance but referenced during render. Make sure that this property is reactive, either in the data option, or for class-based components, by initializing the property. See: https://v2.vuejs.org/v2/guide/reactivity.html#Declaring-Reactive-Properties.
And there is no happen if I use:
<chessboard orientation="black"/>

But with the showThreats, the document give that:
<chessboard :showThreats="true"/>

I can not understand why? Thank you for helping me ^^
Comment: The difference with ```:``` is that one binds to a value and the other doesn't. in ```orientation="black"``` you don't need ```:``` because all you do is pass a **string**. In your second scenario ```:showThreats="true"``` you need ```:``` because you are passing a **javascript boolean**
Comment: aha, I see the answer, https://stackoverflow.com/questions/45175527/vuejs-component-props-as-string?rq=
Comment: Using a colon in front of an attribute is equivalent to `v-bind`. That means that an attribute, like `:orientation="black"` is binding the orientation attribute to the variable `black`, but if you use `orientation="black"`, then you are just binding orientation to the string `black`
Answer: When you use :some-property, that is a shorthand for v-bind:some-property which means it will bind a variable to that component from parent (two way data binding which is explained in documentation). When you use it without ":" it will only pass a string to that component. So when you pass :orientation="black" it expects that you have variable named black inside parent component. If You write :orientation="'black'" in that case it would only pass a string, but it is easier to write just orientation="black".

How to use props and watch in vue?


Let $\mathcal{T}_1$ and $\mathcal{T}_2$ be topologies of a space $X$. Are the following claims true?

Question:
Let $\mathcal{T}_1$ and $\mathcal{T}_2$ be topologies of a space $X$. Are the following claims true?

If $(X, \mathcal{T}_1)$ is Hausdorff and $\mathcal{T}_1 \subset \mathcal{T}_2$, then $(X, \mathcal{T}_2)$ is Hausdorff.
If $(X, \mathcal{T}_1)$ is Hausdorff and $\mathcal{T}_2 \subset \mathcal{T}_1$, then $(X, \mathcal{T}_2)$ is Hausdorff.
If $(X, \mathcal{T}_1)$ is compact and $\mathcal{T}_1 \subset \mathcal{T}_2$, then $(X, \mathcal{T}_2)$ is compact.
If $(X, \mathcal{T}_1)$ is compact and $\mathcal{T}_2 \subset \mathcal{T}_1$, then $(X, \mathcal{T}_2)$ is compact.

The first two don't seem to be true. If $(X, \mathcal{T}_1)$ is Hausdorff, then for any $x, y \in (X, \mathcal{T}_1)$ we can find respective neighborhoods $U$ and $V$ that are disjoint, but it could be that $\mathcal{T}_2$ contains open sets for disjoint points that cannot be separated.
The second one I think a counterexample would just be any $\mathcal{T}_1$ Hausdorff and $\mathcal{T}_2 = \{X, \emptyset\}$. Then $(X, \mathcal{T}_2)$ is certainly not Hausdorff.
For $3$ if $\mathcal{C}$ is a $\mathcal{T}_1$-cover for $X$, then there exists a finite $\mathcal{T}_1$-subcover $\mathcal{C}'$ and each element in $\mathcal{C}'$ is also in $\mathcal{T}_2$ so this is likely a true result.
For $4$ if we take $X = [0,1]$ with $\mathcal{T}_1$ the standard topology and $\mathcal{T}_2$ the subspace topology on $\mathbb{Q}$, then $\mathcal{T}_2 \subset \mathcal{T}_1$, but $(X, \mathcal{T}_2)$ is not compact since a rational sequence in $X$ that converges to some irrational point doesn't have a convergent subsequence right?
Answer: $\newcommand{\T}{\mathcal{T}}$I'm not sure why but you seem to have these back to front. Some hints and commentary are given below.
For $(1)$, if $(X;\T_1)$ is Hausdorff and $x,y\in X$ are two distinct points, then we know there are $U,V\in\T_1$ with $x\in U,\,y\in V$ and $U\cap V=\emptyset$. Can we use the same $U$ and $V$ in $(X;\T_2)$? Notice that $\T_2$ is not introducing any new points, only new neighbourhoods.
For $(2)$, your counterexample is correct... if $X$ has cardinality greater than $1$. A small detail but one we should still observe.
For $(3)$, you've observed that if $C$ is a $\T_1$-cover of $X$ then $C$ has admits a finite subcover. Sure. But what if $C$ is a $\T_2$-cover of $X$?
For $(4)$, your counterexample is wrong since you haven't changed the topology on $X$ but rather you've changed $X$ itself (replaced $[0,1]$ with $[0,1]\cap\Bbb Q$). Changing only the topology, I ask you: if $C$ is a $\T_2$-cover of $X$, is it a $\T_1$-cover of $X$ too? Can we use that?
Follow-up challenge:

Show that, if $X$ is any set and $\T$ any topology on $X$ such that $(X;\T)$ is a compact Hausdorff space, then the only comparable topology on $X$, with respect to $\T$, that makes $X$ a compact Hausdorff space is $\T$ itself.
Differently put, show that if you make $\T$ any finer or coarser then $(X;\T)$ is no longer compact Hausdorff.
Comment: +1. To the proposer: It is useful to examine some of the simplest cases to see whether a general  statement is true.  If (3) were true then every topology $\mathcal T_2$ is compact (which of course is false) because if $\mathcal T_1=\{\emptyset, X\}$ then $\mathcal T_1$ is compact and $\mathcal T_1\subseteq\mathcal T_2$
Comment: In $(3)$ I think one could also take $\mathcal{T}_1$ to be the indiscrete topology to obtain a counterexample? @FShrike
Comment: @SleepWalker Yes, as Daniel points out. Take any noncompact topology on $X$ and call it $\mathcal{T}_2$, a refinement of the compact, indiscrete $\mathcal{T}_1$. That $\mathcal{T}_1$ is compact does not imply $\mathcal{T}_2$ is.


Using Codeigniter Routing and .htaccess URL rewriting

Question: I wanted to understand Codeigniter URL Routing and .htaccess URL Rewriting.
Are both same?
Basically, I want to hide my ID which is passed to the controller. Do I want to understand how can I use any of them to do so?
Also is it possible that I can just show my domain and the folder name and hide another part of URL
Eg.
www.example.com/distributor/tenant/info/1
www.example.com/distributor
So can I hide my controller and id using .htaccess
My Current .htaccess looks like this
<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteBase /bizcloudds/
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule ^(.*)$ index.php?/$1 [L]

</IfModule>

Can this same thing be done with Codeigniter Routes?
I saw something like this on codeigniter's documentation on Routes :
$route['blog/joe'] = 'blogs/users/34';

But I don't understand how will I get 34 i.e ID of the record in my routes files?
I have very little knowledge of .htaccess and Codeigniter Routes, So please explain.
Comment: _"Basically, I want to hide my ID which is passed to the controller"_ – that is of course impossible. If the client requests just `/distributor`, then there is _no way_ for the server to relate that to any specific id. You could use POST for all your requests, so that the user would not directly _see_ the ID value you are passing - but if you think that would make your application a single bit "safer", then you would wrong as could be already.
Comment: _"Also is it possible that I can just show my domain and the folder name and hide another part of URL"_ – why would you even want to do that? It will keep people from bookmarking specific pieces of content, it will hinder search engines from indexing individual pieces of content ... This is the type of question that usually gets asked by people who, to put it bluntly, have little to no idea how the internet works ...
Comment: Yes, many sites like f.e. URL shorteners use a system of "cryptic" IDs, mainly so that people won't be able to guess the ID of other items by simply in- or decrementing parts of the ID. This usually means some sort of hashing algorithm gets used to translate normal, numeric database IDs into such outside identifiers.
Comment: They are not the same. Routes are used to change CI's standard segment-based approach to URLs - [Documentation Here.](https://www.codeigniter.com/user_guide/general/urls.html)  Consider using session data to pass the ID.
Comment: Use post method. so it will be possible to solve your probelm
Comment: @FairyDancer post method is an option but i dont want to go with that because i dont want multiple forms in a single page.
Comment: @CBroe Can i do something like encrypted ID ? just wondering usually i see on websites at back they have some random string basically not human readable, so is there any way i can do that ? Or it is totally something different than this
Comment: Okay thanks i can try that for my case, but i don't want to use any third party apps for that
Answer: By using post method you can find solution for your question. For example
<form action="page_url" method="POST">
<input name="id" value="16" type="hidden">
<input type="submit" value="Show Details">
</form>

Hope you understand. If you need more guidance on this, I'm happy to help you out.
Comment: yes we can make this form in common to all pages, and specific class to the link and use javascript to submit the form on click on the link which have the class. So it will pass value as post. It works for me. I used this method for one of my project. You have to make little bit tricky on it. It will work dynamic
Comment: Thanks i have this in option, but i have many pages where i have lists, clicking on them sends id to controller, so for all thoses list pages i will have forms which i dont want! Can there be anything more dynamic  ?
Comment: Okay let me check i will get back to you if I need any help :)


Cache sometimes is not being updated on saving operations

Question: I'm maintaining a pretty big news site, with many nodes and with many users who can create and edit nodes. The site is using the Paragraphs and the Memcache modules.
Editors can rearrange the blocks on the front page, add and edit articles. The problem is that occasionally (and sometimes pretty often) when editors do something on the site, the changes won't be immediately visible; they won't be even visible in the edit form when editors try to edit the node again. It will appear after few edits.
I know the change is effective since the first edit, as a call to dmemcache_object()->flush() makes the changes visible. This makes me think it's a problem with Memcache. However, replacing it with Redis doesn't fix the issue.
Core cache is basically disabled as we are caching everything in the CDN, but editors have direct access to the site, without CDN, and experience this problem there, so it's not CDN related. The following  is the cache configuration.

I cannot reproduce the problem locally, even if I'm also using Memcache/Redis.
Sometime, when editors create a new node with paragraphs, the node appears "empty." During the edit, there are also no paragraphs. However dmemcache_object()->flush() seems to fix this problem. (I tried it few times.) Adding a call to dmemcache_object()->flush() in a save/update hook implementation doesn't help, though.
What can I do to diagnose and fix this problem? It is particularly hard to approach because it happens only on the production server.
When I use the Redis module and add the following lines to the  settings.php file, I get a problem with the path aliases, which aren't shown anymore.
$conf['path_inc'] = 'sites/all/modules/contrib/redis/redis.path.inc';

When that line is removed and the cache is cleared, the path aliases are correctly shown.
Also, the database is located on a different server.
Answer: Try Expire module  instead you try to clear cache manually.

This module provides configurable actions upon events that will expire
  URLs from caches like reverse proxy caches, internal page caches,
  etc.This module make more sense when Minimum Cache Lifetime setting is
  set to a value other than none.
There is now integration with the following modules:
Boost - deletes expired pages (files) from the file system.
Varnish - integrates over an administrative socket.
Fastly - Varnish/CDN caching
Purge - expires urls by sending purge requests to Varnish, Squid and Nginx over HTTP.
Acquia Purge - allows Drupal sites hosted on Acquia Cloud to automatically purge pages from their Varnish powered load balancers..
Memcache Storage - allows to delete pages that were cached in a memcached instances.
Akamai - provides an integration with the Akamai Cache Control Utility (CCU) Web Service
Recacher - Page cache warmer.

and Memchache Storage mdoule
Features
This module is an alternative to Memcache API and Integration module focused on different set of features. Available features (D7):

    You can move all cache to the memory.
    You can move users' sessions to the memory.
    You can move lock system to the memory.
    Module can store cached pages for anonymous users as a plain text, so proxy servers (like Nginx with enabled ngx_http_memcached) can get such pages directly from memcached pool and avoid heavy requests to the backend.
    You can clear data from single cache bin, cache cluster or flush all cached data from Drupal UI (screenshot).
    You can find helpful nice debug output (screenshot).
    Integration with Drush
    Integration with Administration menu. Module provides link for flushing memcached data from navigation toolbar.
    Active development and support.

Comment: I'm using Expire module already, sorry if I didn't make this clear. However it doesn't help. I think adding `dmemcache_object()->flush();` in multiple places (submit handlers etc) helped things a little bit, but not fixed it. Also I got a feedback that during `redis` time things got much worse. Redis was using a bigger pool of RAM, so maybe this is related. Hmm. If problem was related to `memcache` AND `redis` modules, I doubt that changing a backend once more will help. Could it be broken RAM or something?
Comment: @ŁukaszZaroda you're welcome,maybe ,this is abroad  and maybe there are many reason than cause it. is there any error in log? did you try disable redis temporary?
Comment: Do you know any possible cause of this behavior, that I could check/consider? I couldn't find anything in logs. The problem is that I cannot really disable memcache/redis on production because it would put too much stress on the server, but in desperation I could try this to confirm if would stop happening.
Comment: Do you got the solution ? I am also facing the same issue
Comment: @RifasAli who? me or Lukasz? :). So please mention him. he didn't notified
Comment: @ŁukaszZaroda Do you got the solution ? I am also facing the same issue
Comment: @RifasAli Unfortunately not, I just added a button running `dmemcache_object()->flush();` if it happened as a workaround, and I didn't get any reports of the issue since. I'm running CentOS 6.10 and you?


Data has two trends; how to extract independent trendlines?

Question: I have a set of data that is not ordered in any particular way but when plotted clearly has two distinct trends. A simple linear regression would not really be adequate here because of the clear distinction between the two series. Is there a simple way to get the two independent linear trendlines?
For the record I'm using Python and I am reasonably comfortable with programming and data analysis , including machine learning but am willing to jump over to R if absolutely necessary.
Comment: Best answer I have so far is to print this on graph paper and use a pencil and ruler and calculator ...
Comment: Just to make sure: your client doesn't have any additional data that would indicate which measurements come from which population? This is 100% of the data that you or your client have or can find. Also, 2012 looks like either your data collection fell apart or one or both of your systems fell through the floor. Makes me wonder if trend lines up to that point matter much.
Comment: I don't have any personal experience with it, but I think [statsmodels](http://statsmodels.sourceforge.net/stable/index.html) would be worth checking out.  Statistically, a linear regression with an interaction for group would be adequate (unless you're saying you have ungrouped data, in which case that's a bit hairier...)
Comment: Unfortunately this isn't effect data but usage data, and clearly usage from two separate systems mixed up into the same data set. I want to be able to describe the two usage patterns, but I can't go back and recollect data as this represents about 6 years worth of information collected by a client.
Comment: Maybe you can compute pair-wise slopes and group them to two "slope-clusters". However this will fail if you have two parallel trends.
Comment: Could you crudely filter the data to isolate the two trendlines, then use those trendlines on the whole dataset to identify points on the trend less crudely, and separate the dataset on that basis?
Comment: No the client can't seperate the records with the available information. The 2012 data will improve as they clean up their recent data, we are getting the next iteration of this data in the coming days with the 2012 data fixed and brought up to date. While I can't disclose what this data is of, I can say it is definitely what the client wants, to find out how consumption will move in the coming years.
Comment: Is your solution for price data and technical analysis?  If not, what is the problem domain?
Answer: To solve your problem, a good approach is to define a probabilistic model that matches the assumptions about your dataset. In your case, you probably want a mixture of linear regression models. You can create a "mixture of regressors" model similar to a gaussian mixture model by associating different data points with different mixture components.
I have included some code to get you started. The code implements an EM algorithm for a mixture of two regressors (it should be relatively easy to extend to larger mixtures). The code seems to be fairly robust for random datasets. However, unlike linear regression, mixture models have non-convex objectives, so for a real dataset, you may need to run a few trials with different random starting points.
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg as lin

#generate some random data
N=100
x=np.random.rand(N,2)
x[:,1]=1

w=np.random.rand(2,2)
y=np.zeros(N)

n=int(np.random.rand()*N)
y[:n]=np.dot(x[:n,:],w[0,:])+np.random.normal(size=n)*.01
y[n:]=np.dot(x[n:,:],w[1,:])+np.random.normal(size=N-n)*.01

rx=np.ones( (100,2) )
r=np.arange(0,1,.01)
rx[:,0]=r

#plot the random dataset
plt.plot(x[:,0],y,'.b')
plt.plot(r,np.dot(rx,w[0,:]),':k',linewidth=2)
plt.plot(r,np.dot(rx,w[1,:]),':k',linewidth=2)

# regularization parameter for the regression weights
lam=.01

def em():
    # mixture weights
    rpi=np.zeros( (2) )+.5

    # expected mixture weights for each data point
    pi=np.zeros( (len(x),2) )+.5

    #the regression weights
    w1=np.random.rand(2)
    w2=np.random.rand(2)

    #precision term for the probability of the data under the regression function
    eta=100

    for _ in xrange(100):
        if 0:
            plt.plot(r,np.dot(rx,w1),'-r',alpha=.5)
            plt.plot(r,np.dot(rx,w2),'-g',alpha=.5)

        #compute lhood for each data point
        err1=y-np.dot(x,w1)
        err2=y-np.dot(x,w2)
        prbs=np.zeros( (len(y),2) )
        prbs[:,0]=-.5*eta*err1**2
        prbs[:,1]=-.5*eta*err2**2

        #compute expected mixture weights
        pi=np.tile(rpi,(len(x),1))*np.exp(prbs)
        pi/=np.tile(np.sum(pi,1),(2,1)).T

        #max with respect to the mixture probabilities
        rpi=np.sum(pi,0)
        rpi/=np.sum(rpi)

        #max with respect to the regression weights
        pi1x=np.tile(pi[:,0],(2,1)).T*x
        xp1=np.dot(pi1x.T,x)+np.eye(2)*lam/eta
        yp1=np.dot(pi1x.T,y)
        w1=lin.solve(xp1,yp1)

        pi2x=np.tile(pi[:,1],(2,1)).T*x
        xp2=np.dot(pi2x.T,x)+np.eye(2)*lam/eta
        yp2=np.dot(pi[:,1]*y,x)
        w2=lin.solve(xp2,yp2)

        #max wrt the precision term
        eta=np.sum(pi)/np.sum(-prbs/eta*pi)

        #objective function - unstable as the pi's become concentrated on a single component
        obj=np.sum(prbs*pi)-np.sum(pi[pi>1e-50]*np.log(pi[pi>1e-50]))+np.sum(pi*np.log(np.tile(rpi,(len(x),1))))+np.log(eta)*np.sum(pi)
        print obj,eta,rpi,w1,w2

        try:
            if np.isnan(obj): break
            if np.abs(obj-oldobj)<1e-2: break
        except:
            pass

        oldobj=obj

    return w1,w2

#run the em algorithm and plot the solution
rw1,rw2=em()
plt.plot(r,np.dot(rx,rw1),'-r')
plt.plot(r,np.dot(rx,rw2),'-g')

plt.show()

Answer: Elsewhere in this thread, user1149913 provides great advice (define a probabilistic model) and code for a powerful approach (EM estimation).  Two issues remain to be addressed:

How to cope with departures from the probabilistic model (which are very evident in the 2011-2012 data and somewhat evident in the undulations of the less-sloped points).
How to identify good starting values for the EM algorithm (or any other algorithm).

To address #2, consider using a Hough transform. This is a feature-detection algorithm which, for finding linear stretches of features, can efficiently be computed as a Radon transform.
Conceptually, the Hough transform depicts sets of lines.  A line in the plane can be parameterized by its slope, $x$, and its distance, $y$, from a fixed origin.  A point in this $x,y$ coordinate system thereby designates a single line.  Each point in the original plot determines a pencil of lines passing through that point: this pencil appears as a curve in the Hough transform.  When features in the original plot fall along a common line, or near enough to one, then the collections of curves they produce in the Hough transform tend to have a common intersection corresponding to that common line.  By finding these points of greatest intensity in the Hough transform, we can read off good solutions to the original problem.
To get started with these data, I first cropped out the auxiliary stuff (axes, tick marks, and labels) and for good measure cropped out the obviously outlying points at the bottom right and sprinkled along the bottom axis.  (When that stuff is not cropped out, the procedure still works well, but it also detects the axes, the frames, the linear sequences of ticks, the linear sequences of labels, and even the points lying sporadically on the bottom axis!)
img = Import["https://i.stack.imgur.com/SkEm3.png"]
i = ColorNegate[Binarize[img]]
crop2 = ImageCrop[ImageCrop[i, {694, 531}, {Left, Bottom}], {565, 467}, {Right, Top}]

(This and the rest of the code are in Mathematica.)

To each dot in this image corresponds a narrow range of curves in the Hough transform, visible here.  They are sine waves:
hough2 = Radon[crop2, Method -> "Hough"]  // ImageAdjust

This makes visually manifest the sense in which the question is a line clustering problem: the Hough transform reduces it to a point clustering problem, to which we can apply any clustering method we like.
In this case, the clustering is so clear that simple post-processing of the Hough transform sufficed.  To identify locations of greatest intensity in the transform, I increased the contrast and blurred the transform over a radius of about 1%: that's comparable to the diameters of the plot points in the original image.
blur = ImageAdjust[Blur[ImageAdjust[hough2, {1, 0}], 8]]

Thresholding the result narrowed it to two tiny blobs whose centroids reasonably identify the points of greatest intensity: these estimate the fitted lines.
comp = MorphologicalComponents[blur, 0.777]) // Colorize

(The threshold of $0.777$ was found empirically: it produces only two regions and the smaller of the two is almost as small as possible.)

The left side of the image corresponds to a direction of 0 degrees (horizontal) and, as we look from left to right, that angle increases linearly to 180 degrees.  Interpolating, I compute that the two blobs are centered at 19 and 57.1 degrees, respectively.  We can also read off the intercepts from the vertical positions of the blobs.  This information yields the initial fits:
width = ImageDimensions[blur][[1]];
slopes =  Module[{x, y, z}, ComponentMeasurements[comp, "Centroid"] /.
          Rule[x_, {y_, z_}] :>  Round[((y - 1/2)/(width - 1))  180., 0.1]
  ]

{19., 57.1}

In a similar fashion one can compute the intercepts corresponding to these slopes, giving these fits:

(The red line corresponds to the tiny pink dot in the previous picture and the blue line corresponds to the larger aqua blob.)
To a great extent, this approach has automatically dealt with the first issue: deviations from linearity smear out the points of greatest intensity, but typically do not shift them much.  Frankly outlying points will contribute low-level noise throughout the Hough transform, which will disappear during the post-processing procedures.
At this point one can provide these estimates as starting values for the EM algorithm or for a likelihood minimizer (which, given good estimates, will converge quickly).  Better, though, would be to use a robust regression estimator such as iteratively reweighted least squares.  It is able to provide a regression weight to every point.  Low weights indicate a point does not "belong" to a line.  Exploit these weights, if desired, to assign each point to its proper line.  Then, having classified the points, you can use ordinary least squares (or any other regression procedure) separately on the two groups of points.
Comment: Hough transform is widely used in the Computer Vision field to identify straight lines in an image. Why shouldn't it be used in statistics also? ;)
Comment: Pictures tell a thousand words and you have 5. This is incredible work from a quick graph i made just for the purpose of this question! Kudos!
Comment: That's an interesting question, @Lucas. In many statistical applications, though, there is an asymmetry between the $x$ and $y$ variables: one is viewed as accurately measured and the other is viewed as subject to random variation. For such applications, the image-processing methods will not give quite the right results (and in many applications, they will fail to give useful results at all). But from time to time there may be some fruitful overlap between the two fields: which I believe is the point you intended to make.
Comment: Yes. Imagine, for example, the amount of outliers involved in comparing two images to detect if they are from the same subject. And, most of all, imagine having to do it in real time. "Speed" is a very important factor in Computer Vision, and not so important in Statistics.
Comment: @RoyalTS Thank you for pointing out the need for a fix to one of the code snippets. By the time I found your suggested change it had been rejected (correctly, because it wasn't quite right, but never mind that: I am grateful you noticed there was an error).  I fixed it by removing the reference to `rotation`, which was set to zero originally and therefore made no difference.
Comment: Sorry about the incorrect fix and thanks for replacing it with a correct one!
Answer: I found this question linked on another question. I actually did academic research on this kind of problem. Please check my answer "Least square root" fitting? A fitting method with multiple minima for more details.
whuber's Hough transform based approach is a very good solution for simple scenarios as the one you gave. I worked on scenarios with more complex data, such as this:

My co-authors and I denoted this a "data association" problem. When you try to solve it, the main problem is typically combinatorial due to the exponential amount of possible data combinations.
We have a publication "Overlapping Mixtures of Gaussian Processes for the data association problem" where we approached the general problem of N curves with an iterative technique, giving very good results. You can find Matlab code linked in the paper.
[Update] A Python implementation of the OMGP technique can be found in the GPClust library.
I have another paper where we relaxed the problem to obtain a convex optimization problem, but it has not been accepted for publication yet. It is specific for 2 curves, so it would work perfectly on your data. Let me know if you are interested.
Comment: I am sad to see that over two years nobody else has upvoted this original and valuable reply. In the meantime, has the last paper you mentioned been accepted?
Comment: The paper has indeed been accepted, just a few months ago. You can download it here https://gtas.unican.es/pub/378. This is actually a quite rare problem (which may explain its lack of popularity), but we still managed to find some interesting applications. Have a look at the experiments at the end of the paper if you like.
Answer: user1149913 has an excellent answer (+1), but it looks to me that your data collection fell apart in late 2011, so you'd have to cut that part of your data off, and then still run things a few times with different random starting coefficients to see what you get.
One straightforward way to do things would be to separate your data into two sets by eye, then use whatever linear model technique you're used to. In R, it would be the lm function.
Or fit two lines by eye. In R you would use abline to do this.
The data's jumbled, has outliers, and falls apart at the end, yet by-eye has two fairly obvious lines, so I'm not sure a fancy method is worth it.


How to adjust the length of [node:summary] for metatags?

Question: How can I tweak the amount of characters for [node:summary], so that the description metatag never shows more than 150 characters?
On admin/config/search/metatags/config/node I can only see for description token [node:summary], which does not allow me to control the amount of characters that get displayed on the source code. I would need to trim it to max 150 characters, but I have no idea how/where this can be done.
Answer: You can do a few different things.
Assuming you are using the metatag module.
1) Prevent metatags from saving with more than 150 chars using a presave hook:
function hook_metatag_presave(&$metatags, $entity_type, $entity_id, $revision_id, $langcode) {
  dsm($metatags);//debug what the array is. You can trim the values for the metatags here.
}

2) Alter the metatag before bing cached:
/**
 * Alter metatags before being cached.
 *
 * This hook is invoked prior to the meta tags for a given page are cached.
 *
 * @param array $output
 *   All of the meta tags to be output for this page in their raw format. This
 *   is a heavily nested array.
 * @param string $instance
 *   An identifier for the current page's page type, typically a combination
 *   of the entity name and bundle name, e.g. "node:story".
 * @param array $options
 *   All of the options used to generate the meta tags.
 */
function hook_metatag_metatags_view_alter(&$output, $instance, $options) {
  if (isset($output['description']['#attached']['drupal_add_html_head'][0][0]['#value'])) {
    $output['description']['#attached']['drupal_add_html_head'][0][0]['#value'] = truncate_utf8($output['description']['#attached']['drupal_add_html_head'][0][0]['#value'],150);
  }
}


Get generic type class during runtime

Question: I've been trying to figure out how to determinate the classname of a generic type at runtime, but maybe I need to rephrase my thinking regarding this.
For example I have:
Map<String,Integer> map = new PersistentHashMap<String,Integer>();

Will I be able to retrieve the classnames java.lang.String and java.lang.Integer from the PersistentHashMap for example?
Comment: Why would you want to know the names of the classes?
Comment: Need that since I'm doing a persistent Map, it needs to get and store the class name
Comment: Again, why would you need it?
Comment: Huh? I need to store the class name to a persistent storage
Comment: usually the workaround is to have the constructor for your class recieve and store two Class instances explicitely
Comment: @radai Can you site some example?
Comment: I don't understand how will you use the class types. Knowing this, the problem might be of your design and not of code. By the way, at runtime every `Map` will be turned into a plain `Map`.
Comment: @LuiggiMendoza I understand your part, however, I can actually store objects without dealing with the K,V generic type, however I just want to know how to do this and perhaps use it in my PersistentHashMap
Comment: @LuiggiMendoza For example use it to further improve query, etc.
Comment: I can't see any query improvisation here (at least that you provide a **real world** example). As stated in SLaks' answer, you can't. As stated by @radai, the only way you can *know* the types is passing them in your class constructor or somewhere else. Also, I can't find `PersistentHashMap` as a Java standard type.
Comment: @LuiggiMendoza PeristentHashMap is just an example, something similar to the one I have in GitHub. Yah, type erasures, and Generic types are just for compile time correctness...
Comment: @xybrek - guice has type literals for these things - https://google-guice.googlecode.com/git/javadoc/com/google/inject/TypeLiteral.html - but its probbaly overkill for you. just change your constructor to be PersistentHashMap(String.class, Integer.class) for example, and then you have 2 Class instances that you can store and use with reflection (better store just the class names as Strings)
Comment: The doubt still remains: what is the real purpose for this? Just plain *curiosity*?
Comment: @radai I guess this is the only way, but as much as I want to make my "API" simple, I will just rely on the getting of the classname during the Map put/get methods... thanks anyway.
Comment: @LuiggiMendoza hmmm... let see, well, when I do Map, I want to effectively put the class name as property of the Entity (see GAE Entity) like e.put("map.key", "thekey"); e.put("map.key.type", "java.lang.String"); e.put("map.value", serialize(someobject));  e.put("map.value.type", "com.mycompany.SomeClass"); // Something like this...
Comment: And the **real world** usage for this is...
Comment: @LuiggiMendoza I mean I'm storing Map entries as GAE Datastore entities :-)
Comment: @LuiggiMendoza What real word use are you trying to ask? I'm using my library for my web project
Comment: Looks more like a design problem then. What is your real problem (since this is the solution you tried to use but didn't work)?
Comment: Also, by **real world** I mean that the code is in a real working application and other people could read the question and use it in its own application (for example, me).
Comment: @LuiggiMendoza hmmm, my mapping library works pretty well, and I will be able to do java.util.Map query, i.e Iterator for billions of records (without performance and memory) issue because I just mapped the interface to the existing query interface provided by the GAE Datastore...
Comment: Sorry, but I don't understand your problem.The only thing I can say is that you can't do what you're trying. Also, I can't provide any further help.
Comment: @LuiggiMendoza and that I don't have to deal with GAE Datastore Entities and stuff like that, simple POJO's would do for us to store to the GAE Datastore, hence: Map m = Datastorecast.getMap("messages"); m.put("luiggi", "hello world!"); // And from what I see, Project Voldemort works just like a big map or something
Comment: @LuiggiMendoza No problem. I understand that generic types are compile time only. The best answer would be from radai I guess
Answer: Due to type erasure, this is not possible.

Get generic type class during runtime


How to prove this collection is a sigma algebra

Question: This is one of the previous comp question. I would appreciate if somebody can give me a proof.
Let $\mathbb A= \{E \subset X: E $ is a countable or $E^c$ is countable $\}$. To prove this collection is a sigma algebra.
What I think is I need to prove there is a empty set in the collection which is vacuously true because empty set has finite number of elements, namely $0$ elements. To prove complement is there I think I need to show for the two cases. That's where I confused. Would somebody explain that to me? I also want to see how the infinite union of the elements of the collection belong to it?
Comment: A countable union of countable sets is countable. And the countable union of sets whose complement is countable should make you reach for de Morgan's laws and think for a bit.
Comment: For countable union, suppose $E = \bigcup_n E_n$. If all $E_n$ are countable, then it's obvious that $E$ is countable. Otherwise, one of $E_n$ must have a countable complement, let's say it's $E_1$. So $E_1^c$ is countable. We see that $E^c = \bigcap_n E_n^c = E_1^c \cap \bigcap_{n>1} E_n^c \subset E_1^c$, so $E^c$ must also be countable.
Comment: The complement should be immediate. $E \in \mathbb A$ means $E$ is countable or $E^c$ is countable. If $E \in X$ is countable, then $(E^c)^c = E$ is countable. If $E^c$ is countable, then $E^c$ is countable. (I'm repeating myself here.) In both cases, we get either $(E^c)$ is countable or $(E^c)^c$ is countable, meaning $E^c \in \mathbb A$.
Comment: @Tunococ, how about the infinite collection?
Comment: Consider $A \in \mathbb{A}$. By definition $A$ is countable or $A^c$ is countable.

If $A$ is countable, then $(A^c)^c = A$ is countable and hence $A^c \in \mathbb{A}$.

If $A^c$ is countable, then since $A^c$ is countable, $A^c \in \mathbb{A}$.
Answer: Take $E\in \mathbb A $. Then $E$ ou $E^c$ is countable. So $E^c\in\mathbb A$. Now take a infinite enumerable sets $E_n$ of $\mathbb A$. If all $E_n$ is enumerable than $\cup E_n$ is also enumerable. Suppose at least one of $E_n$ is not countable. Then $(\cup E_n)^c$ is countable because  $(\cup E_n)^c=\cap E_n^c$ and at least one of $E_n^c$ is enumerable
Answer:
The empty set belongs to $A$ because it is countable.
$X\in A$ because $X^c$ is the empty set, therefore, countable.
if $B\in A$ is countable then $B=(B^c)^c$ is countable. Theferefore, $B^c\in A$
if $A_n$ is countable for all n, then $\cup{A_n}$ is countable. If $A_n^c$ is countable for some m, then, since $A_m\subset \cup A_n$, we have $(\cup A_n)^c\subset A_m^c$,which is countable. Then $(\cup A_n)^c$ is countable and, by the third item, $\cup A_n$ is countable.


Connecting Quickbooks OAuth through Google Apps Script

Question: I'm trying to connect Quickbooks API through OAuth. I've added the OAuth 2 library to the Apps Script and deployed the Web App. The Web App URL is what I'm using as an URI in the OAuthService call and I've also set it up in the Quickbooks URI parameters. I get the Quickbooks site to open to authorize, but then I keep getting errors from Google or Intuit.
Code looks like this:
function getAuthorizationUrl() {
  console.log('getAuthorizationUrl called');
  var service = getOAuthService();
  if (!service.hasAccess()) {
    var authorizationUrl = service.getAuthorizationUrl();
    return authorizationUrl;
  }
}

function getOAuthService() {
  var REDIRECT_URI = 'https://script.google.com/macros/s/[SCRIPT_ID]/usercallback/';
  return OAuth2.createService('QuickBooks')
      .setAuthorizationBaseUrl('https://appcenter.intuit.com/connect/oauth2')
      .setTokenUrl('https://oauth.platform.intuit.com/oauth2/v1/tokens/bearer')
      .setClientId(CLIENT_ID)
      .setClientSecret(CLIENT_SECRET)
      .setRedirectUri(REDIRECT_URI)
      .setCallbackFunction('authCallback')
      .setPropertyStore(PropertiesService.getUserProperties())
      .setScope('com.intuit.quickbooks.accounting')
      .setTokenHeaders({
        'Content-Type': 'application/x-www-form-urlencoded'
      });
}

I'm currently getting this error even though the URIs are the same in both ends.
Error message

The redirect_uri query parameter value is invalid. Make sure it is
listed in the Redirect URIs section on your app's keys tab and matches
it exactly. Click here to learn more about the redirect_uri query
parameter.

Any suggestion? I've tried all possible configurations for the URI.
EDIT:
Solution: Use OAuth2.getRedirectUri()
Comment: The message is pretty clear, so you may want to double-check your redirect URI. The OAuth2 library doesn't require setting the redirect URI as it's provided automatically, so you may want to remove that set call and use the value provided by `service.getRedirectUri()`.
Comment: @Diego Believe me I had tripled checked and the URIs where the same. What worked now is using `OAuth2.getRedirectUri()`.
Answer: Solution: Use OAuth2.getRedirectUri()
Comment: Please explain how and why for future readers to understand


How to send email with commas in the name?

Question: I am trying to send email of the format: 'Tom Smith, Jr <email@example.com>'; but it fails due to the comma in the name.
This closed issue: https://www.drupal.org/project/drupal/issues/2936032 dealt with handling commas in the site name for handling the From address. It basically added a helper function to wrap the name part in double quotes. But this doesn't work for the To address.
Any suggestions other than simply removing the comma?
Answer: Tom Smith, Jr <firstname.lastname@example.com> and 'Tom Smith, Jr <firstname.lastname@example.com>' are not valid email addresses, but "Tom Smith, Jr" <firstname.lastname@example.com> is. Drupal doesn't "fix" addresses to make them RFC 2047 compliant, you need to do that yourself or use a helper module (such as Mime Mail) which will take care of a lot of these little details for you. This is especially important if you want to use non-ASCII characters in your email addresses.
Drupal 8 core provides a \Drupal\Component\Utility\Mail::formatDisplayName() method you can use, but it is well-known to do the wrong thing in many cases and has been removed in Drupal 9.2. This is part of the reason the above-mentioned helper modules exist.
Comment: Thanks but adding double quotes as I mentioned, doesn't work.
Comment: Is it possible the failure comes from the mail server? I am using Papercut SMTP as a test mail server. There isn't much for error reporting; any issue is just returned as "can't send". It is hard to imagine that Drupal mail can't handle "last, first" as an email name.


Concatenate the result of two concatenations in mySQL

Question: I have this mySQL query:
SELECT
CONCAT_WS('=>',column_1,column_2,column_3)
AS column_union
FROM table

In which the result is the combination of these 3 columns with => as delimiter.

Is it possible to concatenate in the same query the result of the first concatenation with any other columns?
For example:
SELECT CONCAT_WS('#**#',column_4,column_5,column_union)
AS another_column_union
FROM table

In which the final result another_column_union should look like this way:
value_column_4#**#value_column_5#**#v1=>va=>v0

Comment: Wrap your first query up in a derived table, then continue.
Answer: You need to use a view, a subquery, or repeat the expression.  It can also be simplified to:
SELECT concat_ws('=>', column_1, column_2, column_3) as column_union,
       concat_ws('#**#', column_4, column_5,
                 concat_ws('=>', column_1, column_2, column_3)
                ) as another_column_union
FROM table

Comment: Hi @Gordon, thank you for your reply. I have forgot to specify some important things in the the topic question. Take a look to the delimiter and the final result that I expect in the final concatenation query.


Permissions by taxonomy for taxonomy and nodes

Question: I have following setup:

Taxonomy Departments (hierarchical in at least 3 levels)

Location A

Team A.1

Subteam A.1.I
Subteam A.1.II
Subteam A.1.III

Team A.2

Location B

Team B.1
Team B.2

Taxonomy "Worker" (taxo ref to "Department")
Content Type "performance review" (taxo ref to "Worker" and to "Department"), one user may have multiple of reviews (they can be done from time to time, like every 3 months)
Users with a specific role ("Manager")

What I need is:

Managers should only see specific departments and all workers in that department
If the manager can see Team A.1 (or Location A) he can see all (Teams and) Subteams also (and all workers belonging to those)
Managers should only see the performance reviews of workers he has access to

The question is:

How do I best go forward to get there?

Problems I found with existing modules and more problems:

You can give node acess based on taxonomy, but those modules don't give access to taxonomy based on taxonomy
To me it would be logic to give the user accounts a new field with reference to the "Department" and build the permissions from there. "Acces by Reference" for example would need me to give the "Department" a reference to users (which you could argue for, but in this project I would prefer the other way (or a synced both-way-solution).
Just in case you wonder about this: The review references the department (instead of always getting it from the referenced Worker taxo), because when workers switch to another department, the review should stay in the department in which it was created

If the only way forward is programming my own module:

Is this a good starting point: https://api.drupal.org/api/drupal/modules%21node%21node.api.php/function/hook_node_access_records/7.x ?
Answer: I Think You need to take a look at Permissions By Term and you may need to get an extra step to implement some code to provide the required term and assign it for the required permission according to your required hierarchy.
Comment: Thanks. PBT for D7 did not work at all for me.
For now I have a workaround with TAClite - it is a lot more hustle, because I have to give each manager the department permissions and access to every worker in that department also has to be set each time.

That is far from ideal - I just do it that way for now to meet customer demands. So I am still open for more ideas and roads to success.


Bootstrapping power estimates for a bootstrap test

Question: Assume I want to use a (nonparametric) bootstrap test for a hypothesis with a sample size of $n_1$ and I already have $n_0$ actual samples on which to base my power estimates. Usually, we would also have $n_1 > n_0$.
Is it a valid procedure to estimate the power of my test by using nested bootstrapping? Basically, I would repeatedly sample $n_1$ samples with replacement from my $n_0$ samples and apply my non-parametric bootstrap test to each of these samples. Finally, I would look at the percentage of the bootstrap tests which was significant at my $\alpha$ level. Are there pitfalls I have to look out for?
I already did some googling without much success, perhaps because I do not know the proper search terms. Therefore it would be also nice to know how the procedure (if it exists) is called, so I can find references.
Comment: Also:  http://cdn.memegenerator.net/instances/400x/37929308.jpg
Comment: The technique you describe seems to be a fusion of the double bootstrap and the m-out-of-n bootstrap. [This](http://www.sciencedirect.com/science/article/pii/S0167947306000879) seems to be slightly related but doesn't cover the m-out-of-n aspect and I have only read its abstract.
Comment: Also your application of the m-out-of-n bootstrap is non-standard (usually it is used to restore consistency not to simulate a larger sample size) and I am unsure if it will work as you envisage.
Comment: So if I've got you correctly, you've got some sample from a larger population.  You're thinking of using that small sample to make lots of different samples (using some resampling procedure), and then applying your test (which happens to involve bootstrapping) to see how often your test is successful, getting power.  Is that right?  If so, it would seem to me that the validity would depend quite a lot on how well the small sample approximates the big-sample-to-be-collected.
Answer: I don't think use bootstrap to artificially increase your sample size would be a good idea. Any violation of the assumption of the independence of the observations dramatically increase the odds od a spurious result (which would be the case when n1 is significantly greater than n0).
I would estimate the confidence interval of the effect size (the strength of the difference / relationship you are trying to use) and assume the lower bound as the true effect. Then it would be easy to estimate the power.
[note: I assume you already have a significan result with n0 observations. Otherwise your data is compatible* with the null hypothesis and there is no way to have a conservative estimate of the powere - unless you are using the wrong test. Power analysis assume the knowledge of the "real" effect size so there is no way to use theme to "bypass" inferential statistics]
*likely to be observed if the null hypotesis is true
Answer: As you bootstrap you assume that the new bootstrapped distribution is equivalent to the original distribution. If $n_1>n_0$ then you are forced to draw repeated values of $n_0$, which leads to several problems. First of all, the bootstrapped distribution will only have de facto $n_0$ values, the rest being copies, and these copies will only lead to a stronger weighting of these -- randomly drawn -- value when doing e.g. a correlation analysis. Then there is the issue on how your tests might require non-correlated data, as referred by @nic.
If you want to really attain $n_1$ real independent values, I think you have to analyse the $n_0$ values, understand which distribution they are likely to follow and then draw $n_1$ values from this assumed distribution. It is certainly not as generic, but, IMO, is more transparent and you will not suffer from the issues mentioned above.


What can I do to avoid such behavior?

Question: Ok so from this morning, I've rejected 10+ suggested edits from this user: https://magento.stackexchange.com/users/38476/nits
What he was doing was adding the magento-1.9 tag to every Magento 1 questions. Most of the time there was no clue the OP was running 1.9 so I rejected almost every edit he made saying it could cause harm.
However, it seems like most of the edit I rejected still got approved -_-:

https://magento.stackexchange.com/review/suggested-edits/62077
https://magento.stackexchange.com/review/suggested-edits/62130
https://magento.stackexchange.com/review/suggested-edits/62133
https://magento.stackexchange.com/review/suggested-edits/62144

I don't feel like going through every single post he edited and remove the tag, what can I do ?
Comment: Someone call the romanian army!
Comment: There are other moderators you know.
Comment: Calling @Nits can you please refrain from adding 1.9 tags to questions which aren't obvious they're running 1.9?
Comment: all I can see 2 against 1 in voting..
Comment: @Marius yeah I know sorry if you felt spammed by me, didn't want to be annoying I just felt overwhelmed by all those edits that I coudln't handle
Answer: I have also seen these kinds of edits and rejected them checking the "causes harm" option and adding a comment like "The OP doesn't state anywhere that this question is about magento-1.9"
I do agree with Fabian that you can do a "reject and edit", but sometimes the question text doesn't even clearly indicate that it is about Magento 1 only, so changing the magento-1.9 tag into magento-1 doesn't seems right to me.
I agree with Qaisar that we should make a clear difference between Magento 1 and 2, but only when this is applicable. If the OP doesn't mention it, (and you care about it) you should ask him/her by placing a comment, not just blindly accept the 1.9 tag edit...
For now, in my opinion, we should just keep on rejecting these edits. By the way, I did go through a user's edit history once to correct all of his edits of this kind ;)
Answer: now days it is necessary to separate question either it belong to magento-1 OR magento-2. as you said it is annoying to see this kind of edit but these are necessary so i approve these kind of edit. That all in my defense.
Comment: Suggestion: **reject and edit** instead and apply the `magento-1` tag instead of `magento-1.9`
Comment: Fabian's point was discussed, Qaisar, not the point in your answer. `magento-1.9` tag is specific to 1.9. `magento-1 tag` is for questions where version of M1 is uncertain or for taxonomy. BTW I edited your comment to make the link work
Comment: @fschmengler i think we already discuss  here: http://meta.magento.stackexchange.com/questions/778/usage-of-the-magento-1-tag-if-no-version-is-specified-in-question/
Answer: Same for https://magento.stackexchange.com/users/18639/umesh-kumar who has done lot of edits last days ... just putting original texts into quotes.
This does not improve anything ...
I accept a lot of minor edits, but this is wasted time.
Rolled back some of this edits, but it has no affect for repution points ... Can someone contact him to stop this?

What can I do to avoid such behavior?


iTextSharp table not displayed in document

Question: I'm trying to create a simple PDF with header and table.
I create the document but the table is not displayed in the document.
I followed:
mikesdotnetting.com
http://www.mikesdotnetting.com/Article/86/iTextSharp-Introducing-Tables
But for some reason it doesn't work for me.
I use the below code:
<code>    Document document = new Document();
    PdfPTable table = new PdfPTable(2);
    MemoryStream ms = new MemoryStream();
    PdfWriter.GetInstance(document, ms);
    Paragraph header = new Paragraph(new Phrase("some title"));
    Phrase phrase = new Phrase("dasdasdas");
    byte[] streamBytes;

    header.Alignment = Element.ALIGN_CENTER;
    document.Open();
    document.Add(header);
    document.Add(table);

    table.AddCell("cell1");
    table.AddCell("cell2");

    document.Close();
</code>
Thanks in advance.
Answer: You are adding the table to the document before adding anything to it. You need to fully create the table and then add it to the document.
<code>table.AddCell("cell1");
table.AddCell("cell2");

document.Add(table);
</code>
Comment: i thought the document constructs itself upon the call to close and thus the order doesn't matter... I was wrong... thanks!


Como eu consigo centralizar meu formulario dentro da minha div?

Question: estou tenndando centralizar meu formulario de pesquisa no centro da minha div mas nao estou conseguindo existe alguma propiedade em css que faca isso sem precisa de ganbiarra? aqui esta a pagina de desenvolvimento
index.html

        <meta charset="UTF-8">
        <title></title>
        <link rel="stylesheet" type="text/css" href="style.css" />

            <form action="search.php" method="GET" class="fc">
                    <input type="text" autocomplete="on" name="q" placeholder="Digite a sua pesquisa" />
                    <input type="submit" value="Tecle Enter" />
            </form>

style.css
* {
    margin: 0;
    padding: 0;
    vertical-align: baseline;
}

html body {
        background-color: ghostwhite;
}

body div {
    display: block;
    width: 90%;
    height: 123px;
    margin: 0 auto;
    border: 2px dashed black;
}

div .fc   {
    margin: 0 auto;
}

vejam eu setei margin: 0 auto; na minha div e na minha class .fc mas nao esta funcinando
resultado n naavegador
Comment: Horizontalmente e verticalmente?
Comment: Para o `margin: 0 auto;` funcionar o seu `form` tem que ter uma largura fixa também!
Comment: nos dois, agora quando eu setei  o width ele ficou tronxo pendendo para a direita
Answer: A única alteração simples que você deve fazer é essa:

Em vez de DISPLAY:BLOCK digite - DISPLAY:FLEX. E mais nada!

TENTE ISSO. É MÁGICO!!!!!!!

* {
    margin: 0;
    padding: 0;
    vertical-align: baseline;
}

html body {
        background-color: ghostwhite;
}

body div {
    display: flex;
    width: 90%;
    height: 123px;
    margin: 0 auto;
    border: 2px dashed black;
}

div .fc   {
    margin: 0 auto;
}

    <form action="search.php" method="GET" class="fc">
      <input type="text" autocomplete="on" name="q" placeholder="Digite a sua pesquisa" />
      <input type="submit" value="Tecle Enter" />
    </form>

Answer: O form por default tem width de 100% do elemento pai. Tem de definir text-align:center

form {
  text-align:center;
  border: 1px solid;
}

            <form action="search.php" method="GET" class="fc">
                    <input type="text" autocomplete="on" name="q" placeholder="Digite a sua pesquisa" />
                    <input type="submit" value="Tecle Enter" />
            </form>

Pode definir também uma width e centrar o form, nota que centrar o form não é a mesma coisa que centrar os elemtos dentro dele, para isso tem o exemplo acima:

form {
    border:1px solid;
  width: 300px;
  margin: 0 auto;
  }

            <form action="search.php" method="GET" class="fc">
                    <input type="text" autocomplete="on" name="q" placeholder="Digite a sua pesquisa" />
                    <input type="submit" value="Tecle Enter" />
            </form>

Comment: valeu cara agora ficou direitinho +1 vlw
Comment: De nada @NikoBellic
Comment: alem de generoso ainda e modesto


How to find number of milliseconds that have elapsed from a timestamp in local/any timezone in java

Question: I know that System.currentTimeMillis() gives the number of milliseconds that have elapsed from epoch(1 Jan 1970, 00:00:00 GMT). But I need a solution to find the number of milliseconds that have elapsed from say 1 Jan 1970, 00:00:00 IST or any other timezone in general.
Comment: May I ask why you want to do this?
Comment: Becoz i need this value in my application to perform some other checks
Comment: It just seems like a very strange thing to want to do.  The number of milliseconds since the epoch is a standard way of measuring a point in time irrespective of time zone.  Using a different epoch could be very confusing to someone looking at your code.
Comment: Yeah, i understand your point @SimonC. I will think of a different way to solve my problem
Answer: Convert the Calendar to the IST timezone and compare the values by calling Calendar.getTimeInMillis();
Please go through the below link for converting to TimeZones.TimeZone Conversion
Comment: Calendar.getTime() will give the local time irrespective of whether we change the Calendar to a different timezone or not. Thats my observation. Pls re check and correct me if im wrong
Comment: I mean to say, it is getTimeInMillis().In general there is no API to compare Date as far as I know.But anyways, I have updated the answer.
Answer: This is a way to get the time for a specific zone:
DateTimeZone tz = DateTimeZone.forID("America/Winnipeg");
DateTime dt = new DateTime(tz);
DateFormat df = DateFormat.getDateInstance();
df.format(dt.getTime());

The IDs can be found on this site.
Comment: I am looking how to find the no. of milliseconds that have elapsed from this new timestamp. Not how to convert to a different timezone