JSON from php into JS file

Question: I'm trying to draw DB tables using php PDO which i successfully made using the following code:
test.php
<?php
$dbh = new PDO("mysql:host=localhost;dbname=test", "root", "");
$statement=$dbh->prepare("SELECT * FROM pdotable");
$statement->execute();
$results=$statement->fetchAll(PDO::FETCH_ASSOC);
$json=json_encode($results);

echo $json;

?>

I need to take the results into a chart JS code which his data array is in this code:
  initCharts: function() {
        if (Morris.EventEmitter) {
            // Use Morris.Area instead of Morris.Line
            dashboardMainChart = Morris.Area({
                element: 'sales_statistics',
                padding: 0,
                behaveLikeLine: false,
                gridEnabled: false,
                gridLineColor: false,
                axes: false,
                fillOpacity: 1,

                data:

                 [{
                    period: '2011',
                    sales: 1400,
                    profit: 400
                }, {
                    period: '2011 Q2',
                    sales: 1100,
                    profit: 600
                }, {
                    period: '2011 Q3',
                    sales: 1600,
                    profit: 500
                }, {
                    period: '2011 Q4',
                    sales: 1200,
                    profit: 400
                }, {
                    period: '2012 Q1',
                    sales: 1550,
                    profit: 5
                }],

                lineColors: ['#399a8c', '#92e9dc'],
                xkey: 'period',
                ykeys: ['sales', 'profit'],
                labels: ['Sales', 'Profit'],
                pointSize: 0,
                lineWidth: 0,
                hideHover: 'auto',
                resize: true
            });

        }
    },

How do i replace the data : [json] with the JSON from the PHP result?
Answer: In php file after closing tag you need write JS part, like it

Be careful with the semicolon. Just need to correct a little JS script, but I think you can do it.
Comment: And how do i import the PHP file into JS? and whats the correct code to replace 'data: [JSON]'?
Comment: @TonalDev Something like this: `function(**data_from_php**) {
if (Morris.EventEmitter) {
dashboardMainChart = Morris.Area({////
data: **data_from_php**,
////});
}
},`
Answer: // test.php
$dbh = new PDO("mysql:host=localhost;dbname=test", "root", "");
$statement=$dbh->prepare("SELECT * FROM pdotable");
$statement->execute();
$results=$statement->fetchAll(PDO::FETCH_ASSOC);
$json=json_encode($results);

header('Content-type: application/json');

echo $json;

//js
...
if (Morris.EventEmitter) {
    //if you use jQuery
    $.get('/test.php', {}, function(result) { //or use $.getJSON
        dashboardMainChart = Morris.Area({
            ...
            data: result,
            ...

xmlhttp if you not use jQuery
Comment: here's the code, is there a problem in the syntax? initCharts: function(data_from_php) {
              $.get('/test.php', {}, function(result) {
                // Use Morris.Area instead of Morris.Line
                dashboardMainChart = Morris.Area({

                     data: result,

                });
              });
            }
        },
Comment: $.get('/test.php', {}, function(result) {  // here in value result will be data from test.php   }
Comment: try to add this code to page
Comment: Nothing basically happens..no alert, tried inserting this in the HTML file and in JS..ill try more tomorrow and come back here to report..thank you
Comment: here example with 2 files http://www.tutorialspoint.com/jquery/ajax-jquery-get.htm
Answer: You would need to echo out the content type, so that browser sees return data and accept as json.
echo ('Content-type: application/json');
$json = json_encode($results);
echo $json;

I usually open up the google console, or firefox's firebug to see the response tab on the network tab you've sent to. From there, you can see what data you are getting back from server to check if you are echoing right data or not.
Furthermore, you can print them in console log by pluggin
console.log(data);

Within your javascript to confirm whether you are getting data in right format.
You might as well look upon some db document to pull out data just what you need.
Instead of SELECT * FROM pdotable, SELECT period, sales, profit would do.
Hope this helps.
Answer: you only need this json_encode function
this accept an array as parameter and return a string json that you can insert inside a js code simple use echo function
the js code must insert in php file and not in external js

JSON from php into JS file


Decomposing interval in $\mathbb{R^n}$

Question: The book I am using states the following result with the respective demonstration:
Let $ \Omega \subset \mathbb{R}^n$ be open set and $E \Subset \Omega$ (Compact embedding). Show that for all $\epsilon > 0$ exist $I_n$, $n=1,2,3...$ open interval such that $I_n \subset \Omega$ for all $n \in \mathbb{N}$, $E \subset \bigcup _{n \in \mathbb{N}} I_n$ and
$$\sum_{n \in \mathbb{N}} \text{Vol}(I_n) < m^*(E)+ \epsilon.
$$
Proof:  Take $\epsilon >0$ exist $I'_n$, $n=1,2,3...$ such that $E \subset \bigcup _{n \in \mathbb{N}} I'_n$ and
$$
\sum_{n \in \mathbb{N}} \text{Vol}(I'_n) < m^*(E)+ \frac{\epsilon}{2}$$
(this goes straight from the definition of external measure by the infimum of volumes). For each $n$ we decompose:
$$\overline{I'_n}= J_{n,1} \cup J_{n,2} \cup...\cup J_{n,k_n}$$
where each $J_{n,k}$ is compact set and
$$\text{Vol}(I'_n)= \text{Vol}(J_{n,1})+...+\text{Vol}(J_{n,k_n}), \text{   diam}(J_{n,k})< \frac{1}{2} d(E,\mathbb{R}^n- \Omega)$$
...
How do I build or show the existence of these $J_{n,k_n}?$
Answer: I will use $d$ for the dimension of the ambient space $\Bbb R^d$. I understand that each $I'_n$ is an interval. Fix an integer $M$ and decompose $I'_n$ into $M^d$ subintervals, where the length of each side of the subinterval is the length of the parallel side of $I'_n$ divided by $M$. The diameter of each subinterval is the diameter of $I'_n$ divided by $M$. Tou can make it as small as you wish by taking $M$ large enough.
Comment: Yes of course, it is an interval with finite volume.
Comment: The same procedure decomposes $\overline{I'_n}$.
Comment: Are you using that $I'_n$ is bounded?
Comment: Very good, but I think wanted to decompose $\overline{I'_n}$. But i understood how it does.

Decomposing interval in $\mathbb{R^n}$


Creating a closed loop with vtkTubeFilter

Question: What I want: Specify a set of points, specify the connectivity using vtkCellArray so that it becomes a closed circle, create a vtkPolyData from it and apply the vtkTubeFilter to give it some volume.
What I get: A loop that doesn't connect the first and last element correctly, despite explicitly defining that. The other confusing thing is, that the tubes seem to get wider approaching the misconnected start/end.

What I did: I started out with this example from the VTK wiki on how to create a polyline. Here's the code (Python 3.4, VTK 7.0):
import vtk

def rendering(mapper):
    """Takes mapper and handles the rendering."""
    actor = vtk.vtkActor()
    actor.SetMapper(mapper)

    # Create a renderer, render window, and interactor
    renderer = vtk.vtkRenderer()
    renderWindow = vtk.vtkRenderWindow()
    renderWindow.AddRenderer(renderer)
    renderWindowInteractor = vtk.vtkRenderWindowInteractor()
    renderWindowInteractor.SetRenderWindow(renderWindow)
    # Add the actors to the scene
    renderer.AddActor(actor)
    # Render and interact
    renderWindow.Render()
    renderWindowInteractor.Start()
    return

pts = vtk.vtkPoints()
pts.SetNumberOfPoints(4)
pts.SetPoint(0, 0.5, 0, 0)
pts.SetPoint(1, 1, 0.5, 0)
pts.SetPoint(2, 0.5, 1, 0)
pts.SetPoint(3, 0, 0.5, 0)

lines = vtk.vtkCellArray()
lines.InsertNextCell(5)
lines.InsertCellPoint(0)
lines.InsertCellPoint(1)
lines.InsertCellPoint(2)
lines.InsertCellPoint(3)
lines.InsertCellPoint(0)

poly = vtk.vtkPolyData()
poly.SetPoints(pts)
poly.SetLines(lines)

tubes = vtk.vtkTubeFilter()
tubes.SetInputData(poly)
tubes.CappingOn()
tubes.SidesShareVerticesOff()
tubes.SetNumberOfSides(4)
tubes.SetRadius(0.1)
tubes.Update()

mapper = vtk.vtkPolyDataMapper()
mapper.SetInputData(tubes.GetOutput())
rendering(mapper)

I hope somebody has any suggestions on how to fix this behaviour. I am thinking of a parameter in either polydata or tubes that I am missing.
Answer: The tube filter does not support generating periodic tubes. You should be able to approximate the effect by making the seam (where tubes meet) collinear. So instead of
pts = vtk.vtkPoints()
pts.SetNumberOfPoints(4)
pts.SetPoint(0, 0.5, 0, 0)
pts.SetPoint(1, 1, 0.5, 0)
pts.SetPoint(2, 0.5, 1, 0)
pts.SetPoint(3, 0, 0.5, 0)

lines = vtk.vtkCellArray()
lines.InsertNextCell(5)
lines.InsertCellPoint(0)
lines.InsertCellPoint(1)
lines.InsertCellPoint(2)
lines.InsertCellPoint(3)
lines.InsertCellPoint(0)

you could do this:
pts = vtk.vtkPoints()
pts.SetNumberOfPoints(5)
pts.SetPoint(0, 0.25, 0.25, 0)
pts.SetPoint(1, 0.5, 0, 0)
pts.SetPoint(2, 1, 0.5, 0)
pts.SetPoint(3, 0.5, 1, 0)
pts.SetPoint(4, 0, 0.5, 0)

lines = vtk.vtkCellArray()
lines.InsertNextCell(6, range(5) + [0,])

You would probably want to turn CappingOff() since the seam will appear in the middle of a segment.
The "bump" will also move to the seam and appears to be a bug in VTK. You should search VTK's bug tracker and file a report if there's not already an issue listed.
Comment: This is the best workaround proposed so far, but [the bump is a problem](https://i.imgur.com/qlOOZxQ.png). VTK is working on their bug tracking system at the time, so I can't submit it.
Comment: I think this answer is as good as it gets, you might as well get the bounty for it!
Answer: Though this is years ago, late comers might want to look at this PR: https://gitlab.kitware.com/vtk/vtk/-/merge_requests/2047
It didn't made into master, but using the changes, you can do this to get a closed tube from a looped polyline:
    // Reset input polydata to a single cell, looped polyline
    double lpt0[3] = { 0.0, 0.0, 0.0 };
    double lpt1[3] = { 1.0, 0.0, 0.0 };
    double lpt2[3] = { 1.0, 1.0, 0.0 };
    double lpt3[3] = { 0.0, 1.0, 0.0 };

    vtkSmartPointer<vtkPoints> pts = vtkSmartPointer<vtkPoints>::New();
    pts->InsertNextPoint(lpt0);
    pts->InsertNextPoint(lpt1);
    pts->InsertNextPoint(lpt2);
    pts->InsertNextPoint(lpt3);

    vtkSmartPointer<vtkPolyLine> line
      = vtkSmartPointer<vtkPolyLine>::New();
    line->GetPointIds()->InsertNextId(0);
    line->GetPointIds()->InsertNextId(1);
    line->GetPointIds()->InsertNextId(2);
    line->GetPointIds()->InsertNextId(3);
    line->GetPointIds()->InsertNextId(0);

    vtkSmartPointer<vtkCellArray> cells
      = vtkSmartPointer<vtkCellArray>::New();
    cells->InsertNextCell(line);

    inputPolyData = vtkSmartPointer<vtkPolyData>::New();
    inputPolyData->SetPoints(pts);
    inputPolyData->SetLines(cells);

    tubeFilter->SetInputData(inputPolyData);
    tubeFilter->LoopedOn();
    tubeFilter->Update();


Minimum for combinatorial sortingproblem

Question: I'm stuck with a combinatorial problem, maybe one of you can help me out, thanks in advance. So heres the problem:
Consider tuples $(i,j)\in \{1,...,N_1\}\times\{1,....,N_2\}=A$. Let $S_1,...,S_x$ be a disjoint decomposition of $A$ which fulfils the following:

$(i,j),(k,l)\in S_m$ then $(i,l),(k,j) \in S_m$.
$(i,j),(k,l)\in S_m$ then: If $i \neq k$ $\forall o \neq m$ and $(i,\ast)\in S_o$ $\Rightarrow (k,\ast')\neq S_o$ ($\ast, \ast'$ arbitrary elements).
$\forall S_m:$ $\vert\{i \in \mathbb{N} \mid (i,\ast) \in S_m\}\vert < n_1$ and $\vert\{i \in \mathbb{N} \mid (\ast,i) \in S_m\}\vert < n_2$.

I'm looking for a minimal bound on $x$ dependent of $N_1, N_2, n_1$ and $n_2$. An upper bound is obviously given by $N_1 \cdot N_2$. For an easy to see lower bound I get $\frac{N_1}{n_1} \cdot \frac{N_2}{n_1}$ by using the third condition. But I think there should be a better lower bound, because I do not use the second condition.
Thank you for any help and sorry for any language mistakes.
Answer: Too long for a comment:
I imagine this is a ridiculously hard problem, since a special case of it relates to finding a large set of mutually orthogonal Latin squares (MOLS), which is an open problem.
Specifically, suppose that $N_1=n_1^2$, and that $n_2=1$. I claim that the lower bound of $(N_1/n_1)\times (N_2/n_2)=n_1\cdot N_2$ can be obtained if and only if there exists an set of $(N_2-2)$ MOLS with dimensions $n_1\times n_1$.
I can illustrate the "only if" part of this claim by showing how to construct your partition given an MOLS of two Latin squares with order $3$:
Given: Two MOLS of order $3$:
Square 1   Square 2
 1 2 3      1 2 3
 2 3 1      3 1 2
 3 1 2      2 3 1

Output: A partition of the $3^2\times (2+2)=9\times 4$ rectangle $A$ into "combinatorial rectangles" of dimensions at most $3\times 1$:
1 1 1 2 2 2 3 3 3   <--- First row is n_1 contiguous blocks
1 2 3 1 2 3 1 2 3   <--- Each group in second row consists of n_1 items, spaced n_1 apart.
1 2 3 2 3 1 3 1 2   <--- The entries of square 1, read left to right, top to bottom
1 2 3 3 1 2 2 3 1   <--- The entries of square 2, read left to right, top to bottom

More generally, I think your problem might relate to "mutually orthogonal Latin rectangles," but this is getting out of my depth.
Comment: I'm not sure if I understand your rectangle in the end correctly. The 1,2 and 3 in each line represent different $S_i$ right? So per line you have 3 sets, which gives you $x=12$ in the end. The MOLS are needed so that per line the $3\times 1$ "rectangles"  are not getting into trouble with the ii) condition. Is this right?

I never considered MOLS until now, so do we know anything about the number of MOLS with dimension $(n\times n)$? Or stated differntly how big has $N_1$ and/or $N_2$ to be to ensure that the lower bound can not be obtained?


Composerise Drupal command syntax and starting location

Question: I have 2 sites a dev site and the production site.  Need to convert them from drupal/drupal composer sites to the newer format allowing continued support for composer.
The syntax is unclear to me as detailed below. From https://github.com/grasmash/composerize-drupal
The 'composer-root' is supposed to be where .git is.  Composer is at /home dir, their is a Git directory a .composer (dir) .gitconfig (file) (I have always ran composer from terminal /opt/lampp/htdocs/d8/web directory)

So is composer-root=./home ??
The Drupal dev site is located in /opt/lampp/htdocs/d8/web where in the web dir the index.php, composer.json, composer.lock, update.php files are.  This is where I ran composer from previously.

So is --drupal-root=./opt/lampp/htdocs/d8/web ??
And I run the command from /opt/lampp/htdocs/d8/web as I have done previously?
If correct: composer composerize-drupal --composer-root=./home --drupal-root=./opt/lampp/htdocs/d8/web --no-update
Many thanks for any suggestions!
Comment: I'm confused by your description of how the project directories are organized. Could you maybe post screenshots of the `ls -la` output of those directories?
Comment: pics of directory structure added
Answer: composer-root should not be your Home directory. It looks like you might have assumed that because there is a directory named git there, but that's not what the installation instructions at https://github.com/grasmash/composerize-drupal are referring to. Those instructions assume that your project files are being tracked as part of a Git repository. You can tell where the root of a Git repository is because it contains a directory called .git (note the dot in front, which the git directory in Home doesn't have).
If your project files are indeed part of a Git repository, the root would be in one of the upward directories from /opt/lampp/htdocs/d8/web.

So is --drupal-root=./opt/lampp/htdocs/d8/web ??

Almost. The drupal root is where the core directory and index.php can be found. But that . in there makes this a relative directory path. You almost certainly mean to use an absolute path there, which would be --drupal-root=/opt/lampp/htdocs/d8/web.

So is composer-root=./home ??

No. Most likely, you will want your composer-root to be one directory up from your project root (where index.php is). So I'd try --composer-root=/opt/lampp/htdocs/d8.


Como Desativar cache em links específicos via htaccess?

Question: Após pesquisar bastante sobre o assunto, vou reformular minha pergunta.
Vai ficar mais claro:
Em meu site Wordpress, eu gostaria de definir a expiração dos arquivos  html apenas na página inicial do site "homepage" para que o browser não salve o cache.
Eu tentei usando o código abaixo para definir a expiração de todos os html para 1 semana, e da homepage "index.html" para 0.
<IfModule mod_expires.c>
# Enable expirations
ExpiresActive On
# Default directive
ExpiresDefault "access plus 1 month"

ExpiresByType text/html "access plus 1 week"
    <FilesMatch "^(index\.html)$">
        ExpiresActive On
        ExpiresByType text/html "access plus 0 seconds"
        Header append Cache-Control "public"
    </FilesMatch>
</IfModule>

Mas apenas a configuração de 1 semana funcionou.
ExpiresByType text/html "access plus 1 week"

A configuração dentro de "FilesMatch" é completamente ignorada.
ExpiresByType text/html "access plus 0 seconds"

Porém, quando eu tento definir a expiração do arquivo styles.css no lugar de index.html, usando a mesma estrutura, o código funciona.
Logo, parece que algo relacionado ao fato de definir a expiração especificamente do arquivo index.html dentro de "FilesMatch" impede o código de funcionar.
Eu sei que o wordpress não possui um arquivo index.html, ele possui um index.php que gera o html. mas provavelmente o nome do arquivo html usado na página home do wordpress não é index. deve ser outra coisa.
Então a pergunta é, como definir a expiração do arquivo html somente na página home?
Answer: Parece que o propósito e o funcionamento de caches não está muito claro para você. O objetivo de se usar cache é para evitar processamentos mais "pesados" e com baixa probabilidade de serem necessários. Deve-se criar mecanismos de cache para processamentos repetitivos e sem alterações durante a existência do cache.
Um exemplo seria uma homepage de um site com muitas visitas. Todos os visitantes devem ver o mesmo conteúdo logo, não é necessário fazer consultas à base de dados para cada visita, por exemplo. Neste caso, pode-se "montar" o html da página uma única vez e utilizá-lo para todos os visitantes.
Uma forma simples e comum de se ter cache em páginas web é com headers específicos que indicam ao navegador do usuário se ele deve aproveitar o conteúdo que já tem armazenado localmente ou se deve fazer uma requisição ao servidor. O header mais comum para isso é o cache-control (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control).
Dependendo de como estiver o header cache-control, em um pedido subsequente de um mesmo usuário, o navegador do próprio usuário vai verificar se já tem o conteúdo localmente e se o mesmo está dentro das regras de expiração definidas. Caso positivo, não é sequer feito um request ao servidor.
Em adição ao header cache-control, pode-se usar o ETag. O objetivo deste é que o navegador identifique se uma URL teve o seu conteúdo alterado depois da expiração do cache.
Pelo que foi descrito, não parece ser o que você precisa. O mais provável é que vc esteja procurando algum cache no servidor para evitar que diferentes utilizadores façam requests que demandem recursos do seu servidor. Para isso, deve-se buscar outras formas de cache, como armazenar o html final gerado pelo seu site. Existem inúmeros plugins de Wordpress que fazem todo este controle automaticamente (e inclusive vão configurar os headers adequadamente para a maioria dos tipos de arquivos como scripts, css e imagens).

Respondendo mais diretamente as perguntas:

Pode fazer algo como

<FilesMatch "<URL>">
    <IfModule mod_headers.c>
        FileETag None
        Header unset ETag
        Header unset Pragma
        Header unset Cache-Control
        Header unset Last-Modified
        Header set Pragma "no-cache"
        Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate"
        Header set Expires "Thu, 1 Jan 1970 00:00:00 GMT"
    </IfModule>
</FilesMatch>

Não há como fazer isso com o Apache/configuração no .htaccess, tão pouco faz sentido a pergunta. O que, de certa forma pode ser feito é caso o cache tenha expirado verificar se há mudanças no conteúdo (para isso pode ser usado o ETag).

Tente antes ler sobre diferentes tipos de caching e onde usá-los. Mas se procura a documentação do .htaccess do Apache, aqui vai: https://httpd.apache.org/docs/2.4/howto/htaccess.html.
Comment: O wordpress salva cache da busca. se não recarregar a página após cada busca, continuará vendo resultados desatualizados.

Preciso desabilitar o cache somente nesta URL de busca.
O código que consegui criar gera um erro 500.
Comment: Veja o meu htaccess:

# ALL - 1 YEAR CACHE

ExpiresActive On
ExpiresDefault "access plus 1 year"

# SEARCH RESULTS - 1 SECOND CACHE

    ExpiresActive On
    ExpiresDefault "access plus 1 second"

Comment: De que cache vc está falando @lgdelai? E para que fazer cache de 1s? Se for uma página com milhares de acessos por segundo até faria sentido, mas ainda assim não seria nas pesquisas.
Comment: E de onde veio "O wordpress salva cache da busca. se não recarregar a página após cada busca, continuará vendo resultados desatualizados."?
Comment: Olá @tvdias, obrigado pela sua ajuda até aqui, eu pesquisei bastante a respeito, e reformulei minha pergunta, acho que agora ficará bem mais claro o que eu desejo. Se puder dê uma olhada, agradeço.


scoring for mlp regressor for pipelines

Question: I'm struggling to figure out what cross_val_score and pipeline.score returns when using MLPRegressor. I was under the impression that it returned r2 score, however when i run sklearn.metrics.r2_score with the data I receive a completely different value.
eg here is my code.
x_train, x_test, y_train, y_test=train_test_split(x,y,test_size=0.2, random_state=i)
scaler=StandardScaler()
x_train=scaler.fit_transform(x_train)
x_test=scaler.transform(x_test)

mlp=MLPRegressor(activation='relu', solver='lbfgs', hidden_layer_sizes=(6,6,6), max_iter=10000, learning_rate_init=0.001)
pipeline=Pipeline([('transformer',scaler),('estimator',mlp)])
pipeline.fit(x,y)
cv=KFold(n_splits=3)
scorescrossval=cross_val_score(pipeline,x_test,y_test,cv=cv)
print('scores cross val',scorescrossval)
print('accuracy: %0.2f (+/- %0.2f)'%(scorescrossval.mean(),scorescrossval.std()*2))
print('pipeline score',pipeline.score(x_test,y_test))

mlp.fit(x_train,y_train)
predictions=mlp.predict(x_test)
RTWO=sklearn.metrics.r2_score(y_test,predictions)

which gives,
scores cross val [0.97435245 0.96969591 0.98333161]
accuracy: 0.98 (+/- 0.01)
pipeline score -0.7028296099215846
RTWO -5.735668230366273

I'm struggling to know which value to trust, and determine what they mean. thanks for your help
Answer: These scores are not comparable because each time a different data is used for learning the model.
Let's break your code in three parts:
1)
pipeline.fit(x,y)
print('pipeline score',pipeline.score(x_test,y_test))

2)
cv=KFold(n_splits=3)
scorescrossval=cross_val_score(pipeline,x_test,y_test,cv=cv)
print('scores cross val',scorescrossval)
print('accuracy: %0.2f (+/- %0.2f)'%(scorescrossval.mean(),scorescrossval.std()*2))

3)
mlp.fit(x_train,y_train)
predictions=mlp.predict(x_test)
RTWO=sklearn.metrics.r2_score(y_test,predictions)

Code 3 is easily separated.
One thing that IMO you are misunderstanding about the cross_val_score
is that it only is testing and not learning from the data, but its not the case.
cross_val_score will split the supplied data according to the supplied cv and train the model those many number of times and report score on the remaining data. So your x_test and y_test will be split into 3 folds and each fold will be tested once based on learning of other folds. So you will get 3 scores.
Also, For code 1 and 2, I can change the order as I did above because cross_val_score will use a clone of pipeline object to learn on the data and report the scores. So your original pipeline is untouched.
Now can you understand the reason behind those different scores?
Update for comment:
In the code 1, you fit the pipeline on whole unscaled data and then test on the scaled part of same data. So the score should be somewhat higher. But you are getting -0.7028.
That's because, x_test is part of x, but scaled already and pipe.score will scale it again which is wrong.
Code 2 is the close to correct. But its only using 0.2 percent of your data and that too already scaled out of the cross_val_score which leads to data leakage. So the scores are high.
Code 3 also seems correct. In this case, the RTWO should match the output from mlp.score(x_test, y_test). Now why its so low is another question. Whats the score on (x_train, y_train)
Comment: Thanks for your answer. I understand how different data is used and how the data is split for cross_val_score, however i dont understand why they are so different. accuracy from cross_val gives 0.98 which suggests a very good model, but r2_score gives -5, suggesting a terrible model. How do i know how to score the model? thank you
Comment: thank you your comment has helped massively. The score on `mlp.score(x_train, y_train)` is 0.992, which is dramatically different to the the score on `sklearn.metrics.r2_score(x_train, y_train)` at -899978!!!

scoring for mlp regressor for pipelines


Dimension of a simple algebra over its center

Question: Let $A$ be a finite dimensional simple algebra over a field $k$. Denote by $K$ the center of $A$. Why the dimension of $A$ over $K$ is a square?
Answer: This follows from the Artin-Wedderburn classification of central simple algebras, together with a sneaky little argument involving tensor products.
First, the big theorem: let $A$ be a finite dimensional simple (associative, unital) $K$-algebra with precise center $K$.  Then $A$ is isomorphic to $M_n(D)$, where $D$ is a
$K$-central division algebra.  Here, for any associative algebra $B$, $M_n(B)$ is the $n \times n$ matrix algebra with entries in $B$.  Clearly $\operatorname{dim}_K M_n(B) = n^2 \operatorname{dim}_K B$, so it is enough to show that the dimension of a $K$-central division algebra $D$ is a square.
For this, tensor from $K$ to the algebraic closure $\overline{K}$.  On the one hand,
$\operatorname{dim}_{\overline{K}} D \otimes \overline{K} = \operatorname{dim}_K D$.  But on the other hand, an algebraically closed field admits no nontrivial finite-dimensional division algebras -- easy exercise -- so $D \otimes \overline{K} \cong M_N(\overline{K})$ and thus $\operatorname{dim}_{\overline{K}} M_N(\overline{K}) = N^2$.
To prove the "easy exercise", let $D$ be a finite-dimensional division algebra over an algebraically closed field $K$, let $x \in D$ and consider the minimal polynomial of
multiplication by $x$, viewed as a $K$-linear endomorphism of $D$...


Convert Object Hierarchy to Object Array

Question: All, I want to create an object array foo[], where the constructor for Foo is
public Foo(string name, string discription){}

I have a database object which has a structure (not incuding stored procedures, functions or views for simplicity) like
public class Database
{
    public string name { get; set; }
    public string filename { get; set; }
    public List
 tables { get; set; }

    public Database(string name, string filename)
    {
        this.name = name;
        this.filename = filename;
    }
}

protected internal class Table
{
    public string name { get; set; }
    public List<Column> columns { get; set;}

    public Table(string name, List<Column> columns)
    {
        this.name = name;
        this.columns = columns;
    }
}

protected internal class Column
{
    public string name { get; set; }
    public string type { get; set; }

    public Column(string name, string type, int maxLength,
                  bool isNullable)
    {
        this.name = name;
        this.type = type;
    }
}

I would like to know the quickest way to add Column and Table information to the Foo[] object array?
Clearly I can do
List<Foo> fooList = new List<Foo>();
foreach (Table t in database.tables)
{
    fooList.Add(new Foo(t.Name, "Some Description"));
    foreach (Column c in t.columns)
        fooList.Add(new Foo(c.Name, "Some Description"));
}
Foo[] fooArr = fooList.ToArray<Foo>();

But is there a quicker way? Clearly LINQ is likely to be slower for a query that does a simalar operation, but I care allot about speed here so any advice would be appreciated. Perhaps the use of a HashSet would be the way to go as there will not be duplicate entries...
Thanks for your time.
Comment: Think about removing the setters from your List<> properties. You don't really need them (Removing the setter won't make the object read only... only the reference to the object).
Comment: What makes you think LINQ will be slower?
Comment: Finally, why do you need an array at all? Why, exactly, is a List not good enough? The answer to this question may influence suggestions for how to solve your question.
Comment: @JoelCoehoorn Thanks for the 'setters' comment, you are right. Now the others; LINQ, under the hood is basically doing loops itself. It also has overheads ascociated with building the actual loops from the fluent/query syntax. The question is, is the loop I have the best possible. The reason for the List -> array is that I am working with an external library that takes `Foo[]` not `List`. Thanks for your time.
Comment: linq does use loops under the hood, but why are those loops any slower than the loop you write yourself? The extra overhead is born mainly by the compiler, and often what linq really does is help you write clearer code that is easier to optimize... resulting eventually in _better_ performance. Yes, you _could_ have written the same code without linq... but too often that doesn't happen. Also, beware the premature optimization.
Answer: You could initialize the array with the correct size and only use it without a backing list:
int size = db.tables.Sum(t => t.columns.Count + 1);
Foo[] fooArr = new Foo[size];
int currentSize = 0;
foreach (var tbl in db.tables)
{
    fooArr[currentSize++] = new Foo(tbl.Name, "Some Discription");
    foreach(var c in tbl.columns)
        fooArr[currentSize++] = new Foo(c.Name, "Some Discription");
}

Comment: @Killercam: You will at least have a good chance to reduce memory consumption and a possible `OutOfMemoryException`. Your list has already doubled the amount of used memory. Also remember that `List` uses a doubling algorithm; if you are feeding items via `Add` (without allocating all the space first) it is going to try to double the capacity always. If the list is already large(say 1 GB), the next `Add` that exceeds the internal capacity of the backing array would allocate 2GB.
Comment: Thanks, this is a good idea. I suppose I will have to run test to actually see the what sort of benifits I _will_ recive. Thanks for your time...
Answer: I would say change your foreach loop to for loop, as discussed here In .NET, which loop runs faster, 'for' or 'foreach'?
Datastructure wise, you do need mutable structure, unless you know exactly how many records you will be inserting into fooList, then you can use Array instead of list. According to the answer of the foreach vs for-loop question, assuming it's correct, for loops on List are a bit more than 2 times cheaper than foreach loops on List, and Looping on array is around 2 times cheaper than looping on List.
So 2 improvement would be:

change foreach to for
use linq to compute the length for the array
as per @Tim Schmelter, and change List to Array
Comment: Thanks for the answer. Switching to a for loop is a good idea. Thanks for the link... All the best.


How do I extend the xmlsitemap module to include the google news / images and videos formats

Question: The xmlsitemap module covers the standard google xml site map. However, https://support.google.com/webmasters/answer/178636?hl=en is the information for the google image search and the specific markup for creating a sitemap for image content. Likewise https://support.google.com/webmasters/answer/80472?hl=en is for video content. I don't think that the xmlsitemap module supports these additional types.  (Google news sitemap is another one). Can anyone offer any more information about extending the module or if there are any patches that will achieve the above.
Answer: For news there's the Google News sitemap module

The Google News module generates a Google news compatible sitemap feed from your Drupal site's content.

I don't think there are existing modules for images/video (I've looked for the same thing before myself and ended up hacking something together).
There's a feature request in the XMLSitemap module's issue queue for exactly this, but it's currently postponed. There are a couple of links on that thread pointing to how others have achieved it, those might be worth a look.


Evaluating $\sum_{k=0}^{\infty}4^{-k}\left | \Gamma(-k+\frac{i}{2}) \right |^2$

Question: I've come across a certain hypergeometric series and have tried to express it in a different way.
So far I've got the following sum:
$$\sum_{k=0}^{\infty}4^{-k}\left | \Gamma(-k+\frac{i}{2}) \right |^2$$
Which just screams Parseval's identity, but it runs from $0$ to $\infty$ instead of from $-\infty$ to $\infty$, so I am a bit stuck and don't really know where to go from here.
To clarify: I know it can be expressed using a Hypergeometric function, but I am looking for a different way of evaluating it.
Comment: I think summation $0$ to $\infty$ need not be a problem.  But do you know some function where the Fourier series involves $\Gamma(-k+i/2)$ ??  That is the reason I doubt Parseval will help .
Answer: The integral representation (which might have something to do with Bessel functions)
$$
\sum_{k=0}^\infty 4^{-k}\left|\Gamma\left(-k+\frac i2\right)\right|^2
=\color{blue}{\frac\pi{\sinh^2(\pi/2)}\int_{-\pi/2}^{\pi/2}e^{\pm x}\cosh\cos x\,dx}
$$
is easy to obtain from the known integral (for $a,b\in\mathbb{C}$ with $\Re a>-1$):
$$
\int_{-\pi/2}^{\pi/2}e^{ibx}\cos^a x\,dx=\frac\pi{2^a}\frac{\Gamma(1+a)}{\Gamma\left(1+\frac{a+b}2\right)\Gamma\left(1+\frac{a-b}2\right)};
$$
using the reflection formula for $\Gamma$, at $a=2k$ and $b=\pm i$ we get
\begin{align*}
4^{-k}\left|\Gamma\left(-k+\frac i2\right)\right|^2
&=\frac{\pi^2}{\sinh^2(\pi/2)}\frac1{2^{2k}\Gamma\left(1+k+\frac i2\right)\Gamma\left(1+k-\frac i2\right)}
\\&=\frac\pi{\sinh^2(\pi/2)}\frac1{(2k)!}\int_{-\pi/2}^{\pi/2}e^{\pm x}\cos^{2k}x\,dx,
\end{align*}
giving the announced result after summation over $k$.


Why $_2F_1(\Delta/2 - 1/8,\Delta/2 + 7/8;\Delta;1)$ is divergent?

Question: I need to evaluate a residue and this term: $$_2F_1\left(\frac{\Delta}{2} - \frac{1}{8},\frac{\Delta}{2} + \frac{7}{8};\Delta;1\right),$$ which appears inside that expression. This makes the residue divergent. I checked using mathematica this gives complexinfinity independent of the value of $\Delta$.
I want to know how this term diverges. Tried to apply several available identities without any success.
Edit : Similarly, $$_3F_2\left(\frac{\Delta }{2}-\frac{9}{8},\frac{\Delta }{2}+\frac{7}{8},\frac{\Delta }{2}+\frac{7}{8};\frac{\Delta }{2}-\frac{1}{8},\Delta ;1\right)$$ diverges for any value of $\Delta$. There should be some identity that's applicable to any $\, _{p+1}F_p$ function, it seems.
Comment: See below Eq.8 here [ http://mathworld.wolfram.com/HypergeometricFunction.html ] which discusses convergence properties (for your series $z=1$ and $R[c-a-b] = -1 < 0$)
Answer: The fundamental reason is that the hypergeometric differential equation for $_2F_1(a,b,c,z)$ is characterized by the Riemann symbol
$$\left\{ \begin{array}{ccc}0 & 1 & \infty \\ 0 & 0 & a \\
1-c & c-a-b & b\end{array}\right\}.$$
This means, in particular, that (for sufficiently generic parameters) $_2F_1(a,b,c,z)$ can be represented as a linear combination of two functions of the form $$f(z),(1-z)^{c-a-b}g(z),$$
where $f(z)$, $g(z)$ are holomorphic in the neighborhood of $z=1$ and can be normalized as $f(0)=g(0)=1$. In your case $c-a-b=-1$, which produces divergence. Similar reasoning can also be applied to $_{p+1}F_p$.


How to solve this differential equation with the initial condition?

Question:
How to solve this differential equation with the initial condition
$$\begin{cases} x(0)=1\\ x'(t) \cos t - 2x \sin t = 1 \end{cases}$$ for $t \in (\frac{-\pi}{2}, \frac{\pi}{2})$?

I tried to use $z(t)=x(t)\cos(t), z'(t)=x'(t)\cos(t) -x(t) \sin(t)$, but then I have $z(t)-x(t)\sin t =1$, so there is still a problem to be solved.
Answer: Hint: $x' - (2\tan t)x = \sec t$. Can you take it from here as you have a linear $1$st order DE.


On groups with none of their quotient groups divisible

Question: Does there exist a group $G$ that satisfies the following conditions:

Any proper subgroup of $G$ is contained in a maximal subgroup.
There is some $N\unlhd G$ such that $\frac{G}{N}$ is divisible.

?
How about if $G$ is abelian?
My definition of divisibility is the same as what is used in this post: https://math.stackexchange.com/a/419101/138171
Comment: trying to find a counterexample?
Comment: What is it that makes you think it is false?
Comment: If second condition is not satisfied, the first one is satisfied. I'm asking for the converse.
Answer: There does not exist an abelian group with the prescribed properties.
Assume contrariwise that $G$ is such a group and that $N\lhd G$ gives rise to a divisible quotient group. The first assumption implies that $N$ is contained in a maximal subgroup $M$. If $G/M$ is infinite, then it has non-trivial subgroups, so correpondence principle tells us that there are subgroups properly between $M$ and $G$. So $G/M$ is finite, say $n=[G:M]$.
This implies that for all $x\in G$ we have $x^n=M$. So if $yN\in G/N$ is chosen in such a way that $y\notin M$, then $(zN)^n\neq yN$ for all $z\in G$, because $(zN)^n\in M/N$.

If we look for a non-abelian example, the same argument does not work, because $M$ is not necessarily normal. The argument only disallows those pairs $(G,N)$ such that $N$ is contained in a normal subgroup of a finite index.
Comment: @user It is a standard result: act on the cosets of your subgroup by left multiplication, and take the kernel of this action.
Comment: So the equation $\theta ^n = yN$ has no solution for $\theta$ in $G\over N$ which is a contradiction because $G\over N$ is divisible by hypothesis. thanks. I could accept this answer if the group was assumed to be abelian.
Comment: if $x\in G$, then $Mx^n=M$ so $x^n\in M$.
Comment: There is some $y\in G\setminus M$. for any $z\in G$, if $Nz^n=Ny$ then $yz^{-n}\in N\subseteq M$ and so $My=Mz^n=M$ which implies $y\in M$. So $Nz^n\ne Ny$.
Comment: A subgroup of index $n$ contains a normal subgroup of index $\le n!$, so in such a group $G$ there has to be a maximal subgroup of infinite index.
Comment: Do you have a reference for this proposition?


Projective Co-ordinate Geometry

Question: I am learning projective geometry in my computer vision course. So, we represent a co-ordinate point in an image as a homogeneous co-ordinate as $(x,y,1)$.
My professor says that if we are given two such points, $P(x_1,y_1,1)$ and $Q(x_2,y_2,1)$ then $P \times Q$ (vector product) represent the equation of this line.
Any explanation to this?
Answer: In affine geometry you can express a line as
$$\left\{(x,y)\;\middle|\;ax+by+c=0\right\}$$
In homogeneous coordinates this becomes
$$\left\{(x,y,z)\;\middle|\;ax+by+cz=0\right\}$$
Now the coordinates of the line are this vector $(a,b,c)$. And the equation on the right side of the above set notation is a dot product between line $(a,b,c)$ and point $(x,y,z)$. So a point lies on a line if the dot product is zero. The dot product is zero if the vectors are perpendicular in $\mathbb R^3$.
The cross product in $\mathbb R^3$ is perpendicular to both its arguments. Therefore the resulting vector will represent a line which is incident to both the points from which you computed it.

Projective Co-ordinate Geometry


How to change Haskell code formatting rules in VS Code

Question: I'm using VS Code with the Haskell extension v2.2.1 and the Haskell Syntax Highlighting extension v3.6.0.
When I write:
data Name = Name
  { first :: String
  , last :: String }

And I invoke formats election on it I end up with
data Name = Name
  { first :: String,
    last :: String
  }

How can I change this behaviour, so that it keeps the first way of formatting (with leading commas)?
Answer: The default formatter for the Haskell extension is ormolu which always uses trailing commas.
In the VSCode settings for the Haskell extension you can choose a different formatter. For example fourmolu which is a fork of ormolu with more configuration options. It has a setting for the comma style:

comma-style |   leading, trailing | Where to place commas in lists, tuples, etc.
Comment: Thanks. Works perfectly.

Surprisingly I didn't have to install `fourmolu`! -- And even more surprisingly it is not on my path (which includes ~/.ghcup/bin)


$f=0$ $a.e$ implies that $f\equiv 0$

Question: Let $U\subseteq \mathbb{R}^n$ be a connected open set, and let $f:U\to\mathbb{R}$ be a function of class $\mathcal{C}^1(U)$ such that $f=0$ $a.e$ (almost everywhere). How does one prove that $f\equiv0$ ?
Any hints would be appreciated.
Comment: Try to show that $f= 0$ on a dense subset of $U$, then use continuity (differentiability is not needed)
Answer: Assume that $f$ is not equal to $0$ everywhere. Therefore, there exists some $x_0$ such that $f(x_0) > 0$ (without loss of generality).
Now, try and find some ball around $x_0$ on which $f$ is not equal to $0$. Continuity of $f$ should be enough for this. Since the ball has a nonzero measure, $f$ is not $0$ a.e.
Comment: Why is connectedness needed?
Comment: @learnmore I don't see why it would be needed.
Comment: okay good work +1
Comment: I think connectedness is needed. for example let $U=N^n\subset R^n$ and $f(x)=1$ then $f$ is continuous
Comment: @RaziehNoori Your example is not an open set. I think connectedness is not needed, because once we have $x_0$, we can simply take a ball $B(x_0, \delta)$ for some small enough $\delta$ such that the ball is a subset of $U$. Then we only focus on the ball and the behaviour of the function in the ball, in which we find a smaller ball on which $f$ has no roots.
Comment: quite right, thank you very much

$f=0$ $a.e$ implies that $f\equiv 0$


Retornando número semana do mês

Question: No mês de Janeiro deste ano (2018) eu tenho 5 semanas no mesmo, sendo elas:
1 semana: 1 a 7;
2 semana: 8 a 14;
3 semana: 15 a 21;
4 semana: 22 a 28 e
5 semana: 29 a 31 (ou para outros de 29 a 04 de Fevereiro).
Ao utilizar a função WeekOfTheYeara aplicação me retorna o dia 29 como sendo semana 1, o que não é verdade.
Alguma solução para este problema?
Answer: Talvez isso possa te ajudar:
weekNum: Integer;
weekNum:=(StrToInt(FormatDateTime('dd',Date)) - 1) div 7 + 1 ;

Comment: Perfeito, obrigado!

Retornando número semana do mês


To prove that the division of two entire functions is rational

Question: If f and g are entire functions such that $|f(z)|\lt|g(z)|$ for $|z|\gt1$, then show that $f(z)/g(z)$ is a rational function.
I am thinking that if I can just prove that $f/g$ is a meromorphic function on the extended plane. And then maybe show that $g(z)$ can't be zero at some point in $|z|\gt1$ so that $f/g$ is rational? But I am stuck at here. Can someone help? Thanks.
Comment: Sounds like GAGA theorem.
Comment: @JackyChong What's that?
Comment: Since $|g(z)| > |f(z)| \ge 0$ you see that $g$ has no zeros in $|z|>1$.
Answer: Let's relax the inequality to $|f|\le |g|$ on the open set $U=\{z:|z|>1\}.$ (This allows $f=g=0$ everywhere. Is $0/0$ rational? Probably not. Let's forget about that case and assume $g$ is nonconstant.)
Since $g$ is nonconstant, it only has isolated zeros in $U.$ Let $a$ be one of them. Then $f/g$ is bounded in a deleted neighborhood of $a,$ hence has a removable singularity there. We can thus regard $f/g$ as holomorphic in $U.$ Note that $|f/g|\le 1$ in $U.$
Let $a_1,\dots,a_n$ be the zeros of $g,$ counted by multiplicity, in $\{|z|\le 1\}.$ Then
$$\tag 1 h(z)=\frac{f(z)(z-a_1)\cdots (z-a_n)}{g(z)}$$
extends to be entire. For $z\in U$ we have
$$|h(z)| \le |z-a_1|\cdots |z-a_n|.$$
Thus $h$ grows "like a polynomial". An entire function that grows like a polynomial is a polynomial. Thus we have
$$\frac{f(z)}{g(z)}=\frac{h(z)}{(z-a_1)\cdots (z-a_n)}.$$
Therefore $f/g$ is rational.
Comment: So I get the basic idea, we can prove that $p(z)f(z)/g(z)$ is rational and because $p(z)$ is a polynomial, so that $f(z)/g(z)$ is rational? But how to prove that $p(z)f(z)/g(z)$ is rational?
Comment: And when z is equal to $z_1, ... or z_n$, the whole thing is going to be zero. How does that help?
Comment: Well $pf/g$ is entire, and you need to spend more time than a few minutes thinking about this.
Comment: It is entire and bounded by $|p(z)|$ for large $z$. That restricts the set of functions.
Comment: But how to connect this fact with rationality?
Comment: Have you seen the result that an entire function that "grows like a polynomial" is a polynomial?
Comment: Nope? Can I use that directly?
Comment: The result is: If $f$ is entire and $|f(z)| \le A + B|z|^k,$ then $f$ is a polynomial of degree no more than $k.$
Comment: Okay, so we can use the fact above to prove that f is a polynomial. And because g cannot be zero, so it is also a polynomial. As a result, $f/g$ is rational?
Comment: @TrashFailure I changed my hint to a more complete answer, allowing for $\le$ in the given inequality.
Comment: Thanks. From above, I see the argument if $h(z)$ is entire, but how do we deal with the case for zeros of $g$ outside the disk?
Comment: @TrashFailure You don't need to worry about them. The zeros of $f$ kill them off. Remember, $f/g$ is holomorphic in $U.$
Comment: Could you elaborate more about how does zeros of $f$ would kill them off? Edit: NVM


Finitely generated projective resolution

Question: Let $K$ be a field, $A$ be a finite dimensional $K$-algebra and $M$ be a finitely generated $A$-module. Is it true that $M$ admits a projective resolution by finitely generated projective $A$-modules?
Answer: As $A$ is a finite algebra over $K$ it is noetherian. As $M$ is finitely generated there is a surjection $A^{\oplus n} \longrightarrow M$. $A^{\oplus n}$ is noetherian as $A$ is. Let $N$ be the kernel of this map. By noetherianness, it is finitely generated, so there is a surjection $A^{\oplus m} \longrightarrow N$ and hence an exact sequence $A^{\oplus m} \longrightarrow A^{\oplus n} \longrightarrow M \longrightarrow 0$. Repeat this process to get a projective (in fact free) resolution by finitely generated modules. Note that we didn't need the full strength of the assumption that $A$ is finite over $K$ - only that it was noetherian.
Comment: How easy it is depends on your background knowledge. Here are some facts which will prove that $A$ is noetherian. 1. If $R$ is noetherian then the polynomial ring $R[x]$ is noetherian (this is called the Hilbert basis theorem and the proof is not that easy to come up with on your own). 2. If $R$ is noetherian and $I$ is an ideal, $R/I$ is noetherian. 3. Fields are noetherian. 4. A finite dimensional algebra $A$ over a field $K$ is finitely generated as an algebra, i.e. there is a surjection $k[x_1, \dots, x_n] \longrightarrow A$ for some $n$.
Comment: Is it easy to show that $A$ is noetherian?


Is there a statistical significance in my paired sample data after performing Wilcoxon signed rank test?

Question: I have a paired data consisting of $N$ = 421 samples. I would like to find out if there is statistical significance between the paired samples. Since, I do not know the underlying data distribution I select the Wilcoxon Signed Rank test for my task. I use the SciPy stats module in Python to do the job for me [0]. This function returns the $z$ statistic (under the large-sample approximation that the signed-rank statistic is normally distributed) and $p$ value for the two-sided test. I received the following values as output

$z = 4788.5$
$p = 1.00530788183* 10^{-8}$

The Null Hypothesis $H_o$ Wilcoxon Signed Rank test [1,2,3] states that there is no difference between the paired samples. Since, the $p \leq 0.99$ , I reject $H_0$ as my null hypothesis at 1% confidence interval and conclude that the difference between the two paired samples is statistically significant.
I have two questions -
$Q1$ Is my choice of Wilcoxon Signed Rank test a correct choice. If not, what should I use?
$Q2$ Are my conclusions from the significance test correct? If not, can you please explain why they are not correct?
[0] http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html#scipy.stats.wilcoxon
[1] http://blog.excelmasterseries.com/2010/10/wilcoxon-signed-rank-test-in-excel-for_13.html
[2] Please help me interpret these Wilcoxon signed-rank test results
[3] http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test
Comment: Since $p\le 0.99$, you reject $H_0$ at the **1%** level of confidence, not the 99% level!  (But since $p$ is actually much smaller than $0.99$, and indeed very small indeed, it's safe to reject $H_0$ anyway.)
Comment: @whuber Thanks for the reply. Indeed I should reject $H_0$ at the **1%** level confidence. I am confused though - at what level is rejecting better 99% or 1%?
Answer: Common practise is to compare p-value with three levels - 0.05, 0.01 and 0.001. Since your p-value is less than each of them, you have to choose the smallest one, so you should conclude that differences are significant and p<0.001. Roughly speaking: The smaller the p-value, the more significant differences are.
Since we do not know distribution of your data, we do not also know which test should you use. But you have quite large sample, so there is high chance that parametric test can be appropriate (t-test for paired data).
Comment: Thanks! I do not know the distribution of my data either and therefore I chose to use the Wilcoxon Signed Rank Test as it makes no assumption on the underlying data sample distribution.


Programs can not "see" network drive until I manually browse to it

Question: We have a small peer-to-peer network with all Windows XP computers.
One computer is designated as a Server and has shared several folders. (And it's set to share again automatically on restart, which it does. Other computers on the network hvae a local drive  (s:) mapped to that Server.
Other computers on the network have programs which execute  from (i.e., the .exe is located on) those Server\ShraedFolders.  However, after a fresh restart those apps don't run.  If I manuall browse to the S: drive on the local computer then I can see the Server files. After that I can then start the program(s) that are located on the S:\drive.
These computers are all Windows XP SP2
Answer: Did you have an account on both (client/server) that have the SAME name and also the SAME password?
In a simple P2P network like yours, the easiest way to get things working is to have a commum account between the server and the client.
Answer: I was turning on the Server and Client PCs at the same time. When I booted up the Sever and waited 2 or 3 minutes before booting the client it works fine.
So... it seems that if the Client boots up and the Server isn't fully booted up then the Client diesn't "see" the shared drives until you manually browse to them.
SOLUTION: make sure the Server is fully booted up before starting the client.
Answer: if you have a local login with one password and a network login with the same name but a different password you will see this behavior.


Card Game Probability 13 Card Hand

Question: Me and my friends play a four person poker style card game. Each person is dealt 13 cards, and it is a standard trump card game. Now, as the standard, a five card flush beats a five card straight, but it seems, and we all seem to agree, that we get far more flushes than straights.
The rest of the hands seem to occur an appropriate number of times... So, I am wondering if someone knows, or could help me figure out if the probability of being dealt a straight in a 13 card hand is higher than a flush.
Comment: With flush, do you mean that your entire hand is of the same suit, or that there are $5$ cards of the same suit in your hand? Same question for straight.
Comment: Sorry, I should have been more specific. I will post an edit. But yes, 5 cards.
Comment: http://en.wikipedia.org/wiki/List_of_poker_hands. A flush beats a straight. Also the chance for a straight is higher than for a flush. For usual poker at least.
Answer: I'll post the solution for a flush now. @Awkward came up with a solution for the straight.
(The probability for a straight is about 0.38)
Probability for Flush (including royal flush and straight flush):
Let $p_i$ be the probability that you get a flush in $i$ given suits. Note that $p_3=p_4=0$.
By the inclusion-exclusion principle:
$$\mathbb{P}(\text{Flush})=4p_1-6p_2+4p_3-p_4=4p_1-6p_2$$
Now let's calculate $p_1$:
$$p_1=1-\mathbb{P}(\text{4 or less hearts})$$$$=1-\frac{\binom{13}{0}\cdot 3^{13}+\binom{13}{1}\cdot 3^{12}+\binom{13}{2}\cdot 3^{11}+\binom{13}{3}\cdot 3^{10}+\binom{13}{4}\cdot 3^{9}}{4^{13}}$$$$=1-\frac{3^9}{4^{13}}\cdot(81+351+702+858+715)=1-\frac{2707\cdot 3^9}{4^{13}}=\frac{13826983}{4^{13}}$$
And $p_2$:
$$p_2=\frac{1}{4^{13}}\sum_{i=5}^{8}\sum_{j=5}^{8}\left [1[i+j\le13]\cdot\binom{13}{i}\binom{13-i}{j}\cdot 2^{13-i-j}\right ]$$
$$=\frac{13!}{4^{13}}\cdot(\frac{1\cdot 2^{3}}{5!5!3!}+\frac{2\cdot 2^{2}}{5!6!2!}+\frac{2\cdot 2^{1}}{5!7!1!}+\frac{2\cdot 2^{0}}{5!8!0!}+\frac{1\cdot 2^{1}}{6!6!1!}+\frac{2\cdot 2^{0}}{6!7!0!})=\frac{936078}{4^{13}}$$
So $\mathbb{P}(\text{Flush})=\frac{6211433}{8388608}\approx0.74$.
Comment: Your answer seems to assume cards are drawn with replacement, which is not the usual way to deal cards.  E.g. P(no hearts) should be  ${39 \choose 13} / {52 \choose 13}$ instead of  $({3 \over 4})^{13}$.  That most likely explains why your answer of 0.74 does not match published values on websites dedicated to Bridge (the card game) where suit patterns are studied.  See my answer and sources.
Answer: Ok after trying to tackle this problem for a while, let's just go with the easy 'intuitive' way since we're just comparing probabilities, keep in mind this is probably wrong, and I gave up about halfway through
So both involve 5 cards.
Let's assume they both start with an ace of spades
For a flush, there are 12 options for card 2, 11 options for card 3, 10 options for card 4 and 9 options for card 5 = 11880
For a straight, there are 32 options for card 2, 12+16+20+24/4 options for card 3 (depending on how close card 2 is to card 1), at least 8 options for card 4 and 4 options for card 5. = at least 18432
So according to this shifty research a straight is at least 55% more likely than a flush. But feel free to rip this up and suggest your own way.
tbh i see many errors with this method.
Answer: There are some tricky aspects to this problem.  For example, it's possible for a single thirteen-card hand to contain both a straight and a full house.  In that case, according to the rules of poker, it should be counted as a full house, not a straight.  It's also possible for a single thirteen-card hand to contain two flushes, which makes it easy to accidentally overcount.
So rather than attempting an analytic solution, here are the results of a Monte Carlo simulation.  I simulated 10^6 hands of thirteen cards each and classified the hands according to the highest-ranking five card subset.  Here are the results:
$$\begin{align}
0 & \qquad\text{high card (i.e., nothing)}\\
0 & \qquad\text{1 pair}\\
56,031 & \qquad\text{2 pairs}\\
31 & \qquad\text{3 of a kind}\\
110,243 & \qquad\text{straight}\\
338,621 & \qquad\text{flush}\\
444,528 & \qquad\text{full house}\\
33,794 & \qquad\text{4 of a kind}\\
16,752 & \qquad\text{straight flush}
\end{align}$$
Divide each of these numbers by 10^6 for an approximate probability.
Evidently, flushes are more common than straights.
Answer: A flush is dealt in a 13-card hand iff some suit has length 5 or more.  I.e. a flush is not dealt iff the longest suit has length 4.  This is something that Bridge players care about, and the probabilities are widely available.  E.g. according to:
http://web2.acbl.org/documentLibrary/teachers/celebritylessons/handpatternsrevised.pdf
https://www.bridgehands.com/P/Probability_Hand_Distribution.htm
http://www.bridgeguys.com/MGlossary/mathematical_tables.html
The probability that the longest suit is length 4 (patterns 4-4-3-2, 4-3-3-3, 4-4-4-1 - arbitrary ordering onto the 4 suits are already included) is $\approx 0.35$, and so $P(flush) \approx 1 - 0.35 = 0.65$.
Not sure how they got those numbers but since multiple websites agree, and Bridge is a pretty popular game, and acbl = American Contract Bridge League = the governing body, I would tentatively trust the numbers.
Obviously this doesn't answer the question of how often a flush is dealt and it is actually the best poker hand available among the 13 cards (i.e. there are no full house, etc).

Card Game Probability 13 Card Hand


Inserir varios dados em uma coluna mysql

Question: Boa tarde Galera, estou quebrando a cabeça de como faço essa inserçao no banco, é mais ou menos assim...
Eu tenho um produto, que precisa ter preço de mais de um fornecedor, para comparar os preços. Mas a minha quantidade de fornecedor depende da quantidade cadastrada, nao tenho um numero fixo.
Até o momento eu tenho duas tabelas, uma tabale fornecedor, aonde eu cadastro os fornecedores, e uma tabela produtos, aonde cadastro os produtos e relaciono com a tabela fornecedor, atraves da chave estrangeira id_fornecedor.
Agradeço desde ja!.
Answer: Você vai precisar de uma tabela associativa.
Uma vez que um produto precisa se relacionar com muitos fornecedores e um fornecedor pode ter muitos produtos.
Nessa tabela associativa você pode ter um as colunas ID, ID_PROD, ID_FORN, PRECO.
Dessa forma, você consegue resolver seu problema.
Pra cada preço oferecido por cada fornecedor, você pode preencher essa tabela com o ID do produto, o ID do fornecedor e qual o preço daquele item.
DICA: Normalmente se cria a tabela de item_produto.
Neste item, pode ter inclusive a quantidade de produto, mas não sei se é interessante pra você no momento.
Comment: Andrew Ribeiro, deixa eu ver se entendi, no caso, eu criaria a tabela item_produto, com os campos id, id_prod, id_forn, preco. Aonde id_prod se relaciona com o campo id da minha tabela produtos, e o id_forn se relaciona com o campo id da minha tabela fornecedores. Aí eu teria que excluir a minha chave estrangeira que ja tenho na tabela produtos, certo?
Comment: agora está assim...

minha tabela item_produtos está com _id_produto_ relacionando a tabela de produtos e _id_fornecedor_ relacionando a tabela fornecedor.

coloquei um inner join para mostrar os dados no php, mas nao esta funcionando..
`

$query = "SELECT * FROM `item_produtos`
INNER JOIN `fornecedor` ON `fornecedor.id` =`item_produtos`.`id_fornecedor`
INNER JOIN `materiais` ON `materiais.id` = `item_produtos`.`id_produtos`";
$dados = mysqli_query($con, $query) or die(mysqli_error());
$linha = mysqli_fetch_assoc($dados);
$total = mysqli_num_rows($dados);
`
Comment: É exatamente isso.
Você vai excluir o chave estrangeira da tabela produto por que agora você tem uma tabela associativa.
Dai o select ficaria mais ou menos assim:
`select * from item_produto join produto on produto.id = item_produto.id_produto join fornecedor on fornecedor.id = item_produto.id_fornecedor;`
Tente executar esse SQL diretamente no console do banco de dados, deverá funcionar.
Caso seu erro persista, crie outra pergunta especificamente sobre a linguagem, no caso PHP.


Python producing two instances of data set

Question: New to programming and this seems like a basic problem but I cannot figure it out. The code below creates a .txt file that gives two instances of the last data set.
Could someone help/ explain why this code produces double of the last dataset?
thanks,
import urllib
import re
##NL East stats.
teamstate = ["wsh","phi","nym","mia","atl"]
teamnamelist = ["washington-nationals","philadelphia-phillies","new-york-mets","miami-    marlins","atlanta-braves"]
teamlist = ["Washington Nationals","Philadelphia Phillies","New York Mets","Miami Marlins","Atlanta Braves"]

j=0
i=0
while (i<len(teamnamelist)) and (j<len(teamstate)):
    url = "http://espn.go.com/mlb/team/_/name/" + teamstate[j] + "/" +teamnamelist[i]
    htmlfile = urllib.urlopen(url)
    htmltext = htmlfile.read()
    regex = '(.+?)'
    pattern = re.compile(regex)
    price = re.findall(pattern,htmltext)
    print "the batting average of the",teamlist[i]," is: " ,price
    i+=1
    j+=1

text_file = open("statstest.txt", "a")
text_file.write("averages: {0}\n".format(price))
text_file.close()

Comment: Using a `for x, y, z in zip(teamstate, teamnamelist, teamlist)` seems prettier than a `while` approach.
Comment: Could you double check your indentation... it doesn't look like it survived the copy/paste
Comment: `i` and `j` always have exactly the same value, so you can get rid of one of them for starters.
Comment: Also, you have a lot more than just one element matching your regex.
Comment: as you can see, my skillset is really nonexistent.
Comment: While it's not my place to hijack your approach, are you open to using third-party libraries for this? :)
Comment: I could, but I am more confused as to why this code produces a multiple at the end.
Comment: You mean a list is produced at the end of every line, is that it? :)
Comment: probably. ha, that could be it.
Comment: See my answer below. :)
Answer: A few things:

Use zip on the lists. This pretty much combines them into one list composed of tuples with their elements matching up. Since you've got your elements ordered properly, this will work painlessly.
If you check the page, there are about 7 or 8 elements matching your regex. Using re.findall will already return a list, so some transformation is needed here if you want to properly get the batting average (second in the list).

Number 2 above is largely why your code is returning the following:
the batting average of the Washington Nationals  is:  ['22', '.304', '.362', '.530', '3.21', '2', '0.93', '.179']
the batting average of the Philadelphia Phillies  is:  ['19', '.306', '.364', '.468', '5.96', '2', '1.75', '.311']
the batting average of the New York Mets  is:  ['10', '.179', '.243', '.337', '6.75', '2', '1.64', '.304']
the batting average of the Miami Marlins  is:  ['27', '.301', '.358', '.451', '3.00', '2', '1.31', '.268']
the batting average of the Atlanta Braves  is:  ['6', '.179', '.225', '.337', '1.38', '3', '0.85', '.184']
[Finished in 19.0s]

Varying your approach a little:
import urllib
import re
##NL East stats.
teamstate = ["wsh","phi","nym","mia","atl"]
teamnamelist = ["washington-nationals","philadelphia-phillies","new-york-mets","miami-marlins","atlanta-braves"]
teamlist = ["Washington Nationals","Philadelphia Phillies","New York Mets","Miami Marlins","Atlanta Braves"]

for x, y, z in zip(teamstate, teamnamelist, teamlist):
    url = "http://espn.go.com/mlb/team/_/name/%s/%s" % (x, y)
    htmlfile = urllib.urlopen(url)
    htmltext = htmlfile.read()
    regex = '(.+?)'
    pattern = re.compile(regex)
    val = re.findall(pattern,htmltext)[1]
    print "The batting average of the %s is %s." % (z, str(val))

Result:
The batting average of the Washington Nationals is .304.
The batting average of the Philadelphia Phillies is .306.
The batting average of the New York Mets is .179.
The batting average of the Miami Marlins is .301.
The batting average of the Atlanta Braves is .179.
[Finished in 22.5s]

Using lxml and requests (because it's faster in the long run):
import requests as rq
from lxml import html

teamstate = ["wsh","phi","nym","mia","atl"]
teamnamelist = ["washington-nationals","philadelphia-phillies","new-york-mets","miami-marlins","atlanta-braves"]
teamlist = ["Washington Nationals","Philadelphia Phillies","New York Mets","Miami Marlins","Atlanta Braves"]

for x, y, z in zip(teamstate, teamnamelist, teamlist):
    url = "http://espn.go.com/mlb/team/_/name/%s/%s" % (x, y)
    r = rq.get(url)
    tree = html.fromstring(r.text)
    val = tree.xpath("//span[@class='stat']/text()")[1]
    print "The batting average of the %s is %s." % (z, str(val))

Result:
The batting average of the Washington Nationals is .304.
The batting average of the Philadelphia Phillies is .306.
The batting average of the New York Mets is .179.
The batting average of the Miami Marlins is .301.
The batting average of the Atlanta Braves is .179.
[Finished in 10.6s]

Let us know if this helps.
Comment: Kindly accept the answer by clicking on the green check icon near the top of my post if you found it helpful.


Use Keras TimeseriesGenerator function to generate squence Group By some ID

Question: I need to use TimeseriesGenerator to feed data into a LSTM network. Situation is that, the sequences generated by TimeseriesGenerator function needs to consider the values only within the IDs. For eg, consider the following data:
dt = pd.DataFrame({'ids':[1,1,1,1,1,1,1,2,2,2,2,2,2,2,3,3,3,3,3,3,3], 'values':[1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7]})

For sequence length is 3, I need following output:
[[[1., 1.], [1., 2.], [1., 3.]]
,[[1., 2.], [1., 3.], [1., 4.]]
,[[1., 3.], [1., 4.], [1., 5.]]
,[[1., 4.], [1., 5.], [1., 6.]]
,[[1., 5.], [1., 6.], [1., 7.]]
,[[2., 1.], [2., 2.], [2., 3.]]
,[[2., 2.], [2., 3.], [2., 4.]]
 ...
 ]

However, when I run the below:
X = TimeseriesGenerator(dt.values,dt.values,3,batch_size = dt.shape[0])

I get 2 additional undesired elements as below before the sequence for ID = 2 starts:
[[1., 6.], [1., 7.],[2., 1.]],[[1., 7.],[2., 1.],[2., 2.]]

I know that looping through all IDs is a solution, but that takes massive time when I am doing it in real life, Is there any shortcut to achieve that?
Comment: In fact, `TimeseriesGenerator` source code also uses loop to get data. If you want to save time,you need to rewrite your own `TimeseriesGenerator` class. Other methods can only reduce code without saving time.
Comment: Seems that the source at github is not full. It has only a pass statement
Comment: You can refer `TimeseriesGenerator` source code in here https://github.com/keras-team/keras-preprocessing/blob/master/keras_preprocessing/sequence.py.
Answer: You have to write custom data generator. Below code should fit your custom ids based batch generation which you can use as a  baseline and coustomize it as as required.
import keras
import numpy as np
class DataGenerator(keras.utils.Sequence):
    def __init__(self, dt, length, batch_size=128):
        self.tgs = list()
        for i in range(dt['ids'].min(),dt['ids'].max()+1):
            adf = dt[dt['ids'] == i]
            self.tgs.append(TimeseriesGenerator(adf.values,adf.values,length,batch_size =batch_size))
        self.len = sum([len(tg) for tg in self.tgs])
        self.idx_i = list()
        self.idx_j = list()

        for i, tg in enumerate(self.tgs):
            self.idx_i.extend(list(range(len(tg))))
            self.idx_j.extend([i]*len(tg))
        #print ( self.idx_i,  self.idx_j)

    def __len__(self):
        return self.len

    def __getitem__(self, index):
        return self.tgs[self.idx_j[index]][self.idx_i[index]]

# Test
g = DataGenerator(dt,3,3)
for i in range(len(g)):
    print (g[i])

I am creating a generator for each id and storing it in tgs
The no:of batches are the sum of batches in each generator
idx_j maps a index to corresponding generator
idx_i maps a index to a batch in the the generator mapped by idx_j

Output:
(array([[[1., 1.], [1., 2.], [1., 3.]],
        [[1., 2.], [1., 3.], [1., 4.]],
        [[1., 3.], [1., 4.], [1., 5.]]]),
        array([[1., 4.], [1., 5.], [1., 6.]]))
(array([[[2., 1.], [2., 2.], [2., 3.]],
       [[2., 2.], [2., 3.], [2., 4.]],
       [[2., 3.], [2., 4.], [2., 5.]]]),
       array([[2., 4.], [2., 5.], [2., 6.]]))
(array([[[3., 1.], [3., 2.],[3., 3.]],
       [[3., 2.], [3., 3.], [3., 4.]],
       [[3., 3.], [3., 4.], [3., 5.]]]),
       array([[3., 4.], [3., 5.], [3., 6.]]))


Estilização no Bootstrap

Question: Estou com uma dúvida de iniciante.
Sempre fiz interface web na mão (usando HTML e CSS puro, sem utilizar framework.) eu acabei de estudar um curso que ensinava a utilizar o Bootstrap, pelo oque eu vi no curso, o bootstrap agiliza na produção de interface com seus componentes já prontos, só precisando usar suas classes no html.
Minha Dúvida é:
1- Utilizando o Bootstrap, ainda posso criar meu próprios arquivos css e estilizar alguns componentes ?
2- E se no caso possa utilizar bootstrap junto com arquivos css criados por mim, oque posso estilizar ? por que no caso o framework ajuda muito, mas não quero utilizar os componentes padrões do bootstrap, deixando as minhas páginas com uma cara só.
Answer: Respondendo a 1ª questão:
Sim, você pode criar seus próprios arquivos css, estilizar da maneira que desejar qualquer componente que quiser.
Respondendo a 2ª questão:
O Bootstrap te ajuda a desenvolver de maneira responsiva, sem ter que se preocupar muito com CSS p/ tudo isso acontecer, como tu mesmo citou, ele possui inúmeros componentes, que também podem ser personalizados, mas caso você não queira utilizar estes componentes não tem problema, você pode criar os teus próprios, só tem que tomar cuidado com os possíveis conflitos que isso pode ocasionar.
Comment: Mas geralmente quem já sabe muito sobre bootstrap, eles tentam estilizar a página com seus próprios arquivos css ? ou utilizando tudo que o bootstrap tem, sem necessidade de criar arquivos css? @Mathias
Comment: @RodrigoJacinto utilizar os componentes do bootstrap não quer dizer que a tua página não vai ficar do teu estilo, o bootstrap só tenta facilitar toda a questão da parte responsiva, óbvio ele tem estilos prontos, mas tudo pode ser manipulado, mantendo a responsividade que geralmente é o que da mais trabalho. Não quer dizer que quem tem mais experiencia ou não vai usar estilos externos, varia muito quando se trata de estilo.
Comment: Só uma dica, para complementar... caso vá fazer alterações, verifique qual propriedade não lhe agradou e crie uma outra classe que altera essa propriedade com o `! important` em vez de modificar o as classes originais do bootstrap, pois assim garante a funcionalidade padrão em outras views, quando desejar.

Estilização no Bootstrap


Pandas get rows by its values from dataframes

Question: I have a reference dataframe:
ex:
  time latitude longtitude pm2.5
  0 .  0        0          0
  1 .  0        5          1

  ......

And I have a query with
ex:
  time latitude longtitude
  0 .  1        3
  1 .  0        5

  .......

I want to get the pm2.5 which matches the rows in query.
I have used the iteration of rows but it seems very slow.
predications_phy = []
for index, row in X_test.iterrows():
    Y = phyDf[(phyDf["time"] == row["time"]) & (phyDf["latitude"] == row["latitude"]) & (phyDf["longtitude"] == row["longtitude"])]
    predications_phy.append(Y)

What is the efficient and correct way to get the rows?
Answer: Given reference dataframe df1 and query dataframe df2, you can perform a left merge to extract your result:
res = df2.merge(df1, how='left')

print(res)

#    time  latitude  longtitude  pm2.5
# 0     0         1           3    NaN
# 1     1         0           5    1.0

Loops are highly discouraged unless your operation cannot be vectorised.


Arithmetic Sequences; Finding a term given it's value.

Question: I must have missed this topic in my notes but my assignment is asking me to find which term has a value of 38 in the arithmetic sequence of 4, 5.7, 7.4, and I do not know the formula or method of doing so.
Answer: To find any term of an arithmetic sequence:
$$a_n=a_1+(n-1)d$$
where $a_1$ is the first term of the sequence and $d$ is the common difference , $n$ is the number of the term to find.
Comment: You know, this should have came to mind but I guess I just wasn't thinking clear.


Equal in distribution but unequal almost everywhere?

Question: If this question has been asked, I apologize but I could not find it.
I was wondering if it was possible construct $X$, $Y$ two iid rv's such that they equal in distribution, i.e. $P_X(B) = P(X^{-1}(B)) = P(Y^{-1}(B)) = P_Y(B), B \in Borel(R)$, but differ every a.s., so that $X(\omega) \neq Y(\omega), \forall \omega \in \Omega$, except on a set of measure zero. I realize that likely one has to use the Axiom of Choice, but I am unsure about how to proceed.
I see here " If X and Y are equal almost surely, then they have the same distribution, but the reverse direction is not correct " that equality in distribution does not mean that $P(X = Y) = 1$ even if $X,Y$ are iid rv's.
Any advice/links would be greatly appreciated!
Answer: If rv $U$ is uniformly distributed on $(-1,1)$ then so is $-U$.
They have the same distribution but $U=-U\iff U=0$ and $P(U=0)=0$.
If you want this under the extra condition that the random variables are independent, then let $(X,Y)$ be uniformly distributed on $(0,1)^2$. Then $X$ and $Y$ are independent and both uniformly distributed on $(0,1)$ (so they are iid). However: $P(X=Y)=0$.
Answer: Without independence: Flip a fair coin. $1_H$ and $1_T$ have the same distribution but are never equal.
With independence: Pick any identically distributed continuous RVs $X$ and $Y$. $P(X=Y)=0$, independent or not. Consider in Excel/Google Sheets:
A1=RAND()
B1=RAND()
A1 will never be equal to B1.
However if C1=A1
then
C1 will always be equal to A1 by definition.


Is it true that $L^2$ is compactly embedded in $(W^{1,2}_{0})^{\ast}$?

Question: Is it true that $L^{2}(\mathbb R^{n})$ is compactly embedded in $(W^{1,2}_{0}(\mathbb R^{n}))^{\ast}$?
If so, how can I prove it?
Context
I've just started to study Functional Analysis. I tried to read some articles about embeddings in Sobolev spaces, but all I can find are the standard inequalities (like Sobolev inequality etc).
Comment: In which dimension are we working? On which set do we consider $W_0^{1,2}$?
Comment: @ Tomás: I've just started to study Functional Analysis, so I'm actually not sure how to deal with it. I try to read some articles about embeddings in Sobolev spaces, but all I can find are the standard inequalities (like Sobolev inequality etc).

@ PhoemueX: I consider $R^{n}$. I'll edit it.
Answer: No, not on $\mathbb R^n$. Take a smooth function $\phi$ with support contained in the unit ball. Consider the sequence $f_n(x) = \phi(x-3ne_1)$ where $e_1$ is a basis vector. This is a sequence of bumps going into infinity. Translation by $3ne_1$ does not change either  $L^2$ norm or $W^{-1,2}$  norm (I write $W^{-1,2}$ for $(W_0^{1,2})^*$ here). Since the supports are separated, for $n\ne m$
$$\|f_n-f_m\|_{W^{-1,2}}\ge \frac{\langle f_n-f_m, f_n\rangle}{\|f_n\|_{W^{1,2}}}
= \frac{\langle f_n , f_n\rangle}{\|f_n\|_{W^{1,2}}} = c
$$
where $c>0$ does not depend on $n$, due  to translation invariance. Thus, $(f_n)$ has no convergent subsequence in $W^{-1,2}$.
Remarks
The argument applies to other Sobolev-type spaces with translation-invariant norms. Pretty much nothing is compactly embedded on $\mathbb R^n$, unless you use a weight decaying at infinity.
On domains where the Rellich-Kondrachov theorem applies, you get compact embedding by dualizing the embedding of $W_0^{1,2}$ into $L^2$. The adjoint of a compact operator is compact.
Comment: It should be worth to mentioning that [Adams book](http://www.amazon.com/Sobolev-Spaces-Edition-Applied-Mathematics/dp/0120441438), has an characterization for the domains where Rellich-Kondrachov theorem does work. I think it is in chapter 6.


How to get all roles granted to a user in Trino/Presto?

Question: basically what is the equivalent of hive query in trino/presto.
show role grant user <username>;

Comment: Are you looking for something like this https://prestodb.io/docs/current/sql/show-grants.html ?
Answer: we can get the information from information_schema it is self explanatory but i cannot find document about this system defined schema.
SELECT *
FROM information_schema.role_authorization_descriptors
WHERE grantee = 'a_user';

a more useful query is check whether a user has access to a table
SELECT *
FROM information_schema.table_privileges
WHERE table_schema = 'a_schema'
AND   table_name = 'a_table'
AND   privilege_type = 'SELECT'
AND   (grantee = 'a_user' OR grantee IN (SELECT role_name
                                      FROM information_schema.role_authorization_descriptors
                                      WHERE grantee = 'a_user'));


ValueError: not enough values to unpack (expected 2, got 1) BUT i provide 2 values

Question: Im working on this code that simply adds new data into a text file, and i had it working but then i ran it once more and it produced the errer "ValueError: not enough values to unpack (expected 2, got 1)". I know what this error means its just that i seem to provide two values? i must be missing something thats wrong?
The code where it produces an error:
schoolnames=[]
scores=[]
with open("Cdrive.txt","r")as a_file:
     for x in a_file.readlines():
         data=x.strip()   #Im aware i could just use data, but i use X to try fix the error, of course this didnt work
         print(data)
         school,score=data.split(",") #The error is produced here
         schoolnames.append(school)
         scores.append(score)

My text file looks like this (there are no lines between each bit of data in the text file):
sjp,15
chelmer,10
newhall,20
Comment: Your code runs fine for me when I use those three lines you provided. Most likely you have a blank line somewhere in your text file. Try printing a loop index along with data to track what line you are on before you get the error.
Comment: can you please indent your code?
Comment: Sounds like you have an empty line somewhere in your file.
Answer: You may have blank lines, you can make it work just add one condition checking:
if data would be false if you apply strip() on a blank line. you can check whether comma in the stripped line
 for x in a_file.readlines():
     data=x.strip()
     if data and "," in data:
         school,score=data.split(",")

Comment: yep, I had an empty line. Never would have checked, thanks.


AngularJS + Grunt ngAnnotate and single quote

Question: I'm trying to minify my app but can't go over Grunt ngAnnotate as it throws an error

error: couldn't process source due to parse error
    Error parsing regular expression: Invalid regular expression: /[A-Za-z0-9!#_\?:&.- ']/: Range out of order in character class (3:22)

on a directive which looks like
myApp.directive('allowedChars', function() {

  var EMAIL_REGEXP = /[A-Za-z0-9!#_\?:&\.- ']/;

  return {
    require: 'ngModel',
    restrict: '',
    link: function(scope, elm, attrs, ctrl) {
      // only apply the validator if ngModel is present and Angular has added the allowed characters validator
      if (ctrl && ctrl.$validators.chars) {

        // this will overwrite the default Angular email validator
        ctrl.$validators.email = function(modelValue) {
          return ctrl.$isEmpty(modelValue) || EMAIL_REGEXP.test(modelValue);
        };
      }
    }
  };
});

Gruntfile.js looks like
module.exports = function(grunt) {
    //grunt wrapper function
    grunt.initConfig({
        pkg: grunt.file.readJSON('package.json'),
        ngAnnotate: {
            options: {
                singleQuotes: true
            },
            app: {
          files: [
            {
              expand: true,     // Enable dynamic expansion.
              cwd: './app/js/controllers/',      // Src matches are relative to this path.
              src: ['**/*.js'], // Actual pattern(s) to match.
              dest: './app/js/build/controllers',   // Destination path prefix.
              ext: '.min.js',   // Dest filepaths will have this extension.
              extDot: 'first'   // Extensions in filenames begin after the first dot
            },
            {
              expand: true,     // Enable dynamic expansion.
              cwd: './app/js/services/',      // Src matches are relative to this path.
              src: ['**/*.js'], // Actual pattern(s) to match.
              dest: './app/js/build/services',   // Destination path prefix.
              ext: '.min.js',   // Dest filepaths will have this extension.
              extDot: 'first'   // Extensions in filenames begin after the first dot
            },
            {
              expand: true,     // Enable dynamic expansion.
              cwd: './app/js/directives/',      // Src matches are relative to this path.
              src: ['**/*.js'], // Actual pattern(s) to match.
              dest: './app/js/build/directives',   // Destination path prefix.
              ext: '.min.js',   // Dest filepaths will have this extension.
              extDot: 'first'   // Extensions in filenames begin after the first dot
            },
          ],
            }
        },
    });

    //load grunt tasks
    grunt.loadNpmTasks('grunt-contrib-concat');
    grunt.loadNpmTasks('grunt-contrib-uglify');
    grunt.loadNpmTasks('grunt-ng-annotate');

    //register grunt default task
    grunt.registerTask('default', ['ngAnnotate', 'concat', 'uglify']);
}

Answer: As the error suggests, your regular expression is invalid. Try escaping some of the special characters. Like this:
var EMAIL_REGEXP = /[A-Za-z0-9\!\#\_\?\:\&\.\- \']/;

That should allow ngAnnotate to compile.


Monte Carlo Integration Results in Heavy Tailed Distribution

Question: I am running a Monte Carlo simulation that results in an heavy-tailed distribution. The image below shows the distribution of 1,200 runs of the Monte Carlo simulation, where each run consists of integrating over $M$ = 12,000 randomly drawn paths of $ \mathcal{X}_m = \left\{X_{n,m},s_{n,m}\right\}_{n=1}^N$.

The quantity I am simulation is an expectation of a definite sum of exponentials, where I know the sum converges as $N \rightarrow \infty$.
$$\mathbb{E} \left[ S\left(\mathcal{X}\right) \middle| X_1, s_1 \right] = \mathbb{E} \left[\exp\left( X_1\right) + \exp\left(X_1 + X_2\right) + \cdots + \exp\left(X_1 + X_2 + \cdots + X_N\right) \middle| X_1, s_1 \right].  $$
$X_n$ is a Markov-switching Autoregressive process with Gaussian errors:
$$X_n = \alpha_{s_n} + \rho_{s_n} X_{n-1}  + \sigma_{s_n} \epsilon_n,$$
where $\epsilon_n \sim N(0,1)$, $s_n \in \{1,2\} \sim \Pi$, and $\Pi$ is a transition matrix.
For a single realization of what is presented in the histogram, I compute $S(\mathcal{X}_m)$ for each of the $M$ simulated paths. And then I simply take an average over the $M$ realizations,
$$\mathbb{E}\left[S\left(\mathcal{X}\right) \middle| X_1, s_1 \right] \approx \frac{\sum_{m=1}^M S(\mathcal{X}_m)}{M}.$$
I believe Monte Carlo should result asymptotically in a normal distribution, but this resembles a log-normal distribution. How would I diagnose this issue? How should I change my simulation strategy?
I've proved the sum converges. The proof boils down to the unconditional mean of $X_n <0$, and sum inside the exponential goes to negative infinity faster than one-half the unconditional variance.
The instability occurs when the sum inside the exponential terms is greater than 0 for a few periods (the process is persistent). Even though it will eventually converge to -$\infty$, and can blow up temporarily.
Comment: @hipHop What's the distribution of the $X_i$ here? Are they identically distributed? Are they independent or dependent? How do you know the sum converges?
Comment: I don't understand the current simulation scheme, 12000 random paths of what? And what do you mean by a run of monte Carlo?
Comment: @Glen_b updated
Comment: @svendvn updated
Comment: @hipHopMetropolisHastings, I still don't fully understand what you are simulating. You write that you are simulating 1200*12000 realizations S({X_t}_{t=1}^N), but for which values of N? And how does one run differ from another run?
Comment: @svendvn Sorry for the confusion. The histogram above plots 1,200 results of Monte Carlo Integration performed on the expectation. For a single realization of what is displayed in the histogram, I do the following: (1) simulation $M$ = 12,000 paths of $\mathcal{X}_m = \{X_n,s_n\}_{n=1}^N$, where each path is indexed by $m$. Then for each path I compute the sum of exponential inside the expectations operator. I then average over the $M$ paths to get an estimate of the expectation. The histogram displays what I just described, repeated 12,000 times for different simulations.
Comment: @hipHopMetropolisHastings It would be helpful if $N$ was defined, $s_{n,m}$ was defined, and $X_{n,m}$ was defined.
Comment: @NicolasBourbaki $N$ is the length of the path, also the number of terms in the sum and. It is defined implicitly.
Comment: *"the sum of exponential inside the expectations operator"* this is not very clear
Comment: Could you give the values of $\alpha$, $\rho$ and $\sigma$. And does your proof depend on them?
Answer: I am still not sure I understand what you are trying to simulate, but below I have tried to answer the question to how I think it works.
Simulation setup
The simulations depends on three quantities:

$K$, the number of simulations of the $(X_1, s_1)$. That is $(x_1,\tilde s_1)\sim P_{X_1, s_1}$
$M$, the number of simulations of $S_N(X)|(X_1,s_1)=(x_1,\tilde s_1)$ per simulated value $(x_1,\tilde s_1)$.
$N$, the length of the chain $X_1, \dots, X_N$.

And results in $K$ realizations of the conditional expectation (as a random variable), $E[S_N(X)|X_1,s_1]$. Let $\hat P_{K,M,N}$ denote the actual distribution of the simulations and let $P$ be the hypothesized asymptotic distribution. Let $d$ be a favorite distance measure between distributions. The goal of simulations is to determine if $d(\hat P_{K,M,N},P)$ can be made infinitesimally small by increasing $K,M$ and $N$. That can be a daunting task, so let's break it down.
Size of $K$
For a given size of $K$, it is known how similar the simulated distribution should be to the desired asymptotic distribution. One can simply simulate from the desired asymptotic distribution $K$ times, obtaining a empirical distribution $\hat P_K$ and compute $d(\hat P_K, P)$.
Size of $M$
The size of $M$ should be so big that the variance of $S_N(X)|(X_1,s_1)$ becomes negligible. I don't know if it actually has a finite variance, but if one could get some bound on it, $L$, it would be known that the variance of $\hat E_M[S_N(X)|(X_1,s_1)]$ is less than $L/M$. Then, for a given $K$ and $M$, we can simulate $K$ realizations of the desired asymptotic distribution and add $N(0,L/M)$ noise, obtaining an empirical distribution $\hat P_{K,M}$ and then compute $d(\hat P_{K,M}, P)$.
Size of $N$
For a given $K$ and $M$, I suggest that simulations are used for increasing $N$ to show that $d(\hat P_{K,M,N}, P)$ can be made as small as $d(\hat P_{K,M},P)$. The simulations for different values of $N$ doesn't need to be independent to make a convincing argument, so one can simply continue the same $K\cdot M$ chains.
In practice one may want to simulate a distribution for $d(\hat P_{K,M}, P)$ instead of just a single number. $K$ and $M$ should be chosen such that $d(\hat P_{K,M},P)$ becomes so small that it is convincing when it is shown that $d(\hat P_{K,M,N}, P)$ is of similar size.
Comment: Thank you for your thoughtful answer. I believe the issue is the conditional moment I am trying to simulate is either infinite, or the variance is so large that its infeasible to simulate. IE, there are some sample paths with very small probability weight where the expectations goes to infinity. Hence why I see the far right tail.

I'm marking your answer as correct, because it addressed my question which I think was ill-posed.


Definition of a division ring in category theory

Question: I'm wondering how one can define a division ring in category theory. More precisely, is there a well-defined concept of "division ring object" such that a division ring object in the category of sets is a division ring in the standard sense ?
The fact that division is not defined everywhere seems to be a problem for me, so how can we categorically define division rings (or fields) ? And by "categorically define", I mean "define by using categorical tools like universal arrows and properties".
Edit :
I thought of a possible answer, although it probably isn't very satisfying.
Let's start with the concept of a ring object : it is the data of an object and several arrows satisfying some conditions. In particular, we have the arrows $0:1\rightarrow R$, $e:1\rightarrow R$ and $m:R\times R\rightarrow R$ corresponding to the additive identity, the multiplicative identity and the multiplication on $R$.
Now if we suppose that the category has image factorization and is boolean (it's a strong assumption, I know, but we can try to weaken it afterwards), then we have a subobject of $R$ given by $\text{im}\,0\rightarrow R$, and we can take its complement in the lattice of subobjects of $R$, here denoted by $i:R^\times\rightarrow R$ ($i$ is monic). Then we can add the requirement (to get a division ring) that we have an arrow $(\ )^{-1}:R^\times\rightarrow R^\times$ such that :
$$
m\circ\langle i, i\circ (\ )^{-1}\rangle = e\,\circ\,! = m\circ\langle i\circ (\ )^{-1}, i\rangle
$$
where $!$ is the unique arrow $R^\times\rightarrow 1$. (A diagram would be clearer but I don't know how to draw it here.) Of course, this just means that for every non-zero $a\in R$, there exists $a^{-1}$ such that $aa^{-1}=e=a^{-1}a$.
Here I lazily supposed that the category is boolean so I could form the subobject $R^\times$ of non-zero elements of $R$, but I imagine that we could define that complement by other means (as shown in Kevin Carlson's answer) and still keep the rest of the definition.
Do you think this is a viable definition of a division ring object ?
Comment: @Chessanator A topos is way more than you'd need. Sephi: You should be able to formulate the axioms of a division ring in, at least, [coherent logic](https://ncatlab.org/nlab/show/coherent+logic) (e.g. as $\forall x.x=0\lor\exists y.xy=1\land yx=1$). This would allow defining the notion of division ring in any [coherent category](https://ncatlab.org/nlab/show/coherent+category). But beware that there are [multiple distinct notions that arise](https://ncatlab.org/nlab/show/field#constructive), and I suggested just one.
Comment: You might find useful the concept of [Ring object](https://ncatlab.org/nlab/show/ring+object) Since a division ring is a unital ring where each nonzero element has an inverse, you can play around with the categorical definition to get that idea.
Comment: Yes, but I can't see how we can add the "each nonzero element has an inverse"-property to the concept of ring object to obtain a division ring object. That's precisely my point.
Comment: You're probably going to need the category to be a topos to define the 'every element but 0' properties.
Answer: WARNING: A few details below only work for fields, or for right division rings, but no substantial changes are necessary, so I'm probably going to leave fixing this to the reader.
The idea is roughly this. Work in some category $\mathcal C$, to which we'll add properties as we need them. One way to express the notion of division ring was given in the comments by Derek: a division ring is a ring $\mathcal R$ (that is, a ring object: an object $\mathcal R$ equipped with maps $\times,+:\mathcal R\times \mathcal R\to \mathcal R,0,1:*\to \mathcal R$ satisfying the usual axioms) in which the union of the subobjects $0$ (a map out of a terminal object is always a monomorphism) and $\mathcal R^\times$ is all of $\mathcal R$. So, we have to have unions of subobjects available in $\mathcal C$.
We also have to be able to define $\mathcal R^\times$, which is supposed to satisfy some sentence like $\mathcal R^\times=\{x\in \mathcal R:\exists x'\in \mathcal R:xx'=1\}$. This isn't too hard. We have a subobject $H\subset \mathcal R\times\mathcal  R$ which should be defined as $\{(x,x')\in \mathcal R\times \mathcal R:xx'=1\}$. To define this categorically, we simply take the pullback of the multiplication map $\times:\mathcal R\times \mathcal R\to \mathcal R$ along the unit map $1:*\to \mathcal R$. Having defined $H$, we can construct $\mathcal R^\times$ as the image of the first projection map $H\to \mathcal R$. So, in $\mathcal C$ we need every morphism to have an image which is a subobject.
However, this definition is a bit strong. For instance, if your category is the topos of sheaves on an interesting topological space $X$, and $\mathcal R$ is the sheaf of continuous functions valued in some topological ring $R$, then $\mathcal R^\times(U)$ will be the elements of $\mathcal R(U)$ which are everywhere nonzero in $U$. So it'll be almost impossible to make $\mathcal R$ a division ring in the above sense: you'd be asking that every section of the sheaf be either everywhere or nowhere zero. This is the reason for the other notions of field in the nLab article linked by Derek.
A particularly nice alternative is to say that the complement of $\mathcal R^\times$ is precisely $0$. This sounds like the same thing, of course, but logic internal to a category is rarely classical, so it's actually not. More formally, we define
the complement $\left(\mathcal R^\times\right)'$ to be the largest subobject of $\mathcal R$ whose intersection with $\mathcal R^\times$ is empty (initial.) To define the complement directly in this way, $\mathcal C$ needs to admit arbitrary unions of subobjects; it's possible to get away with less than this by characterizing the complement differently, but in so we need $\mathcal C$ to be a "Heyting" category. In the example of sheaves on $X$ valued in a topological ring $R$, this is the subsheaf of functions $f:U\to R$ such that $f^{-1}((R^\times)')$ is dense: $f$ is almost everywhere valued in nonunits. It's much easier to make $\mathcal R$ a division ring under this definition! We merely need to know that a continuous $R$-valued function which is a non-unit on a dense subset is actually everywhere zero. For this it suffices that $R$ itself be a division ring in which $0$ is a closed subset, as is usually the case.
It may be worth observing that in our example, the union of $\mathcal R^\times$ with its complement is not all of $\mathcal R$: it doesn't include those $R$-valued functions whose zeroset is nonempty but nowhere dense. This non-Boolean aspect of the logic is, again, characteristic of most categories. Unfortunately, it means that even though there are many "division rings" in many categories, not all theorems of elementary linear algebra hold. For instance, we cannot prove that every finitely generated $\mathcal R$-module admits a basis, which would imply in particular that every finite-dimensional vector bundle (viewed as a $C^0(\mathbb{R})$-module) was a direct sum of line bundles.
Comment: Thank you for your answer. I added a few thoughts on my first post.
Comment: @Sephi Your definition is equivalent to mine, whenever the relevant complements exist (it's equivalent to the stronger version under Booleanness, and the weaker version under Heytingness.) In particular, your functional definition of inversion always follows from my existential description since inverses are unique when they exist.
Comment: Great, now things are much clearer for me, thanks !


LINQ to SQL - Using TransactionScope for SELECT statement - is call to Complete() necessary?

Question: I'm using a LINQ to SQL class to define a query, and then calling the ToList() method, which executes the query. The ToList() call is wrapped in a 'using TransactionScope' block to achieve the 'NOLOCK' hint. The query is read-only, so I'm not sure if the call to TransactionScope.Complete() is necessary in this case. Here is the example I followed, which does not make this call: Getting LINQ to SQL and Entities to use NOLOCK
Answer: I'm thinking no.  Complete() in C# is the same as calling Commit in SQL, but since this is a select statement there is nothing to commit.  Any other residual constructs resulting from the TransactionScope will be disposed of at the end of the using { } block so I don't think Complete() is necessary in your case.
Comment: That's the confirmation I was looking for. Thank you


Finite fields of complex numbers

Question:
I don't understand why $(1-i)$ is in the set of element of $R_2$ in the solution.
My understanding is that $\mathbb{Z}/2\mathbb{Z} = \{0,1\}$, endowed with addition and multiplication mod 2. So wouldn't the elements of $R_2$ be $\{0,1,i,1+i\}$? And shouldn't the multiplication table of $R_2$ be something like this?
Comment: $\Bbb Z/2\Bbb Z$ is the quotient of $\Bbb Z$ under the equivalence $a\sim b$ iff $a\equiv b$ (mod $2$). Its elements are equivalence classes of this relation: $[0]$ the set of even numbers and $[1]$ the set of odd numbers. Then $[-1]=[1]$ as $-1\sim 1$. Writing lots of brackets is tiresome, so we abuse notation by writing $1$ for $[1]$ and $-1$ for $[-1]$ etc., so that $-1=1$ insider $R_2$.
Comment: Use your table, but also take into account that $i^2=-1$ and $2i=0$.
Comment: The $p=2$ case is answered in [MSE question 2777107](https://math.stackexchange.com/q/2777107) "Addition and multiplication in F_4".
Answer: Yes, you are right, the elements of $R_2$ are $0$, $1$, $i$, and $1+i$. And since, in $\mathbb{Z}/2\mathbb{Z}$, $1=-1$, $1+i=1-i$ there.
Comment: @ensbana In this case, it is not a field, since $(23+i)(23-i)=23^2+1=530=0$.
Comment: @ensbana $7+2i=7-2i\iff2i=-2i\iff4i=0$, which is false.
Comment: My last question was actually not very well though-out. Sorry! What I meant to ask is that whether there is a general way to do this sort of manipulation? For example, I'm considering $\mathbb{Z}/53\mathbb{Z}$, and want to know whether $7-2i$ is a member of the field, and if it is, how I can rewrite it in the standard form as defined in the definition of $\mathbb{Z}/53\mathbb{Z}$.
Comment: Thanks. But the hint for this exercise is $53 = 7^2 + 2^2$, so I thought I might be able to use an idea similar to the previous exercise, i.e. seeing whether $(7-ki)(7+ki) = 0$ has a positive integer solution k. But this require knowing if $7 - ki$ is a member of the field, given that $7 + ki$ is one.

Finite fields of complex numbers