Linear algebra: cross product property

Question: I am probably a bit lost but I found the following 'cross-product' property in a paper :
$(Ma) \times (Mb) = \frac{1}{det([a,b]^{-1})} ( M_1 \times M_2 ) $
With $M$ a $3\times2$ Matrix, $a$ and $b$ two $2\times1$ vector. $[a,b]$ is a $2\times2$ matrix obtained by concatenating (horizontaly)the two vector a and b and $M_1$, $M_2$ correspond to the 1st and 2nd column of $M$.
I am currently struggling a lot to prove this formula and would like to know if someone can help me. I m sorry for this question which is probably quite trivial but I really don t see where to start.
Thank you very much for your help.
F
Answer: Let the entries of $a$ be $a_1$ and $a_2$ and those of $b$ be $b_1$ and $b_2$. Then the left hand side reads
$$(a_1M_1 + a_2M_2) \times (b_1M_1 + b_2M_2).$$
Using the distributive property we know from ordinary multiplication, this expands to
$$a_1b_1 (M_1 \times M_1) + a_1b_2 (M_1 \times M_2) + a_2b_2 (M_2 \times M_2) + a_2b_1 (M_2 \times M_1)$$
Ignoring terms of the form $X \times X$ (because they are zero) and using the fact that $M_2 \times M_1 = - M_1 \times M_2$ we end up with:
$$(a_1b_2 - a_2b_1)(M_1 \times M_2)$$
This is already almost what we wanted: obviously the scalar $(a_1b_2 - a_2b_1)$ appearing here equals $\det[a, b]$ so all we need is the general fact that $\det A = \frac{1}{\det A^{-1}}$ that holds for all invertible matrices $A$.
This answers the question.
Notice that I tried to avoid actually computing any cross products and worked with their properties (bilinearity, anti-symmetry) instead. This is often a good idea when cross products are involved.
Comment: Thank you very much. Now I get it. I indeed tried to work directly with the cross product and was a bit lost even if I obviously find that left and right part of the equality leads to the same results I did not get the reasoning behind it. That's really helpful thank you very much.
Answer: There is an obvious simplification in your formula:$$\frac1{\det\bigl([a,b]^{-1}\bigr)}=\det\bigl([a,b]\bigr).$$After this, write $M$ as$$\begin{bmatrix}m_{11}&m_{12}\\m_{21}&m_{22}\\m_{31}&m_{32}\end{bmatrix},$$write $a$ as $(a_1,a_2)$, write $b$ as $(b_1,b_2)$ and compute both$$(Ma)\times(Mb)\text{ and }\det\bigl([a,b]\bigr)\bigl((m_{11},m_{21},m_{31})\times(m_{12},m_{22},m_{32})\bigr).$$You will see that they are equal.
Comment: Thank you for your answer. I was trying to understand this equation starting from the left part of the equality but both Vincent and you gave me the reasoning behind it. Thank you again.

Linear algebra: cross product property


Recommend way and maximum byte size for requests in a data-intensive Django application

Question: I am working on an application that handles datasets with varyiing sizes.
For POST requests I do a check on the attached .csv-File for byte size like and if its over 10mb, I return a BAD_REQUEST response:
MAX_FILESIZE_BYTES = 10485760 # 10mb

csvFile = request.data["file"]
if(csvFile.size > MAX_FILESIZE_BYTES):
    raise Exception("File size over "+str(MAX_FILESIZE_BYTES))

except Exception as e:
    return Response("Uploaded File bigger than maximum filesize", status=BAD_REQUEST, exception=True)

Is this the recommend way of handling files too big? And is 10mb a reasonable size, or could my application handle much bigger sizes?
I use rest_framework, atomic transactions with postgresql and I dont expect to have more much than 50 users at the same time.
Comment: 10 megabytes is relatively tiny these days, unless you've deployed your app to run on a toaster. Why do you (think you) need a limit for the upload size?
Comment: What _are_ your "heavy data processing operations"? Remember that a short video taken with a phone camera can be 500 megabytes these days, and those can be processed just fine.
Comment: btw valohai looks like a super cool project.
Comment: Well, if those operations start taking too much time, you can offload them outside the request/response cycle with worker solutions like huey/django-rq/celery/...; if they start taking too much memory, you can optimize them or run your app on a bigger machine.
Comment: I thought it would be a good idea to limit the file size to avoid slow down of the system. I use quite heavy operations and processing and datasets too big would quickly result in long answering times.
Comment: Well, what happens when your user actually does need to upload a 10.1 megabyte file?
Comment: the user shouldnt do that and in my case probably wont. Anyways, we are talking about about application requirements.
My question was more of a technical nature... What size is generally feasible for data-intensive Django applications? You said 10mb is relatively, small. Even if I have heavy data processing operations?
Comment: The data is processed for predictions/forecasts, normalization and more. I implemented everything in pandas.
Comment: I read about celery and I will consider that. As I have kept performance in mind at all time and tried to use principles like the vectorized operations in pandas as often as possible, I dont think it will be neccessary any time soon. But thanks for the hint.
Answer:
Is this the recommend way of handling files too big?

Well, it seems a bit weird that you'd raise an exception just to catch it.
MAX_FILESIZE_BYTES = 10 * 1024 * 1024  # 10 MiB

if request.data["file"].size > MAX_FILESIZE_BYTES:
    return Response(
        "Uploaded file bigger than maximum filesize",
        status=BAD_REQUEST,
        exception=True,
    )

is simpler.

And is 10mb a reasonable size, or could my application handle much bigger sizes?

As I said in my comment, 10 mb is relatively tiny these days, so unless you're running on a very small machine, or your "quite heavy operations" are very heavy indeed, it's a pretty low limit.
Also, consider the case where your user actually needs to upload a file that's, say, 10.1 megabytes in size – they'll just complain that your app doesn't allow them to do that.

Simply stated: You will either need zero of a thing, one of a thing, or an arbitrary number of the thing.
Programmers and architects ignore this at their own peril. Arbitrary fixed limits are a Code Smell.

– Zero-One-Infinity Rule
Sure, if your app allows just anyone to start uploading a gigabyte of data, you might want to set a very high arbitrary "safety" limit, but if it's e.g. an internal tool of some sort, I'd just get rid of the limit altogether. If your user tries to upload a large enough file, the "quite heavy" operation will likely fail, and they'll complain about that, not that your app doesn't let them do their work.


Campos não são 'mascarados' com jQuery Masked Plugin

Question: Boa tarde,
@EDIT
Os campos agora estão acessíveis. Porém, o que é digitado não é mascarado conforme foi predefinido no script.
Plugin: http://digitalbush.com/projects/masked-input-plugin/
Código:
<!DOCTYPE html>

<meta charset="UTF-8">
<title>Cadastro</title>

<form action="">

        Campo 1:
        <input type="text">

        Campo 2:
        <input type="text">

        Campo 3:
        <input type="text">

        Campo 4:
        <input type="text">

        Campo 5:
        <input type="text">

</form>

Como podem ver não entendo quase nada ainda. Estou começando a aprender.
Obrigado desde já!
Comment: O que queres dizer com _"impossíveis de serem acessados numa página PHP"_? queres dizer que não consegues receber o valor do input no servidor? ou que o input não funciona e o utilizador não consegue escrever nele? Podes colocar o HTML que tens?
Comment: @Sergio, não se consegue digitar nada no input, você clica dentro dele e tenta digitar mas nada é inserido.
Comment: Podes [edit] a pergunta e colocar o teu HTML? Se fizeres um jsFiddle é ideal...
Comment: Entao com certeza o problema não esta relacionado ao php. Você pode editar sua pergunta e colocar o codigo do formulario com os inputs também? Talvez ajude a encontrar o erro
Comment: Sergio e Andrei Coelho, editei.
Answer: Estás a aplicar o .mask() ao elemento div. Tens de aplicar ao elemento input. Assim o teu selector tem de ser ".classe_da_div input".
Assim funciona:
 jQuery(function ($) {
     $(".data input").mask("99/99/9999");
     $(".telefone input").mask("9999-9999");
 });

Exemplo: http://jsfiddle.net/x94dzw92/
Comment: funcionou! Obrigado pela ajuda!!
Comment: Gostei deste Plugin... Vou usar também! obrigado
Comment: @Luan ótimo! Bom que pude ajudar.
Answer: O código acima faz funcionar, mas para utilizar os dados do fomulário precisará colocar os atributos name e id dos campos input.

    Campo 1:
    <input name="campo1" id="campo1" class="maskdata" type="text">

Na chamada função da máscara ainda pode fazer por herança ".data input" ou aplicação de classes com nomes à sua escolha como ".maskdata" ou ".masktel".
jQuery(function ($) {
     $(".data input, .maskdata").mask("99/99/9999");
     $(".telefone input, .masktel").mask("9999-9999");
 });

Segue exemplo rodando.
http://jsfiddle.net/s3vq3s9z/1/


Puppeteer, looping through links selected by xpath

Question: I'm new to puppeteer (and not so great at javascript in general) and am trying to write some basic functionality to:

Get all the links from an XPath
Loop through and click those links
Screenshot and save the HTML of the page
Go back, screenshot and save the HTML of the records page to be save in the same directory of the others and start the process over

The error I get is:

Evaluation failed: DOMException: Failed to execute 'querySelector' on 'Document': '0' is not a valid selector

Here's the code I have:

I'm fairly confident all the code works, except for the issues i'm having getting the right things to click on with my XPath. The website I'm getting these from is:
https://hrlb.oregon.gov/bspa/licenseelookup/searchdir.asp?searchby=lastname&searchfor=a&stateselect=none&Submit=Search
code:
const records = await page.$x('//table[2]//tr[td[a]]//td[1]/a');
let int = 0;
for (let record in records) {
    await Promise.all([
        page.waitForNavigation(),
        page.click(record)
    ]);

    await Promise.all([makeDirectory('screenshots/item'+int), makeDirectory('screenshots/item'+int+'/base'), makeDirectory('screenshots/item'+int+'/record')]);
    let recordPath = "screenshots/item"+int+"/record/record.html";
    let basePath = "screenshots/item"+int+"/base/base.html";

    page.screenshot({path: "screenshots/item"+int+"/record/record.png", fullPage: true});
    let recordBody = await page.evaluate(() => document.body.innerHTML);
    await saveHtml(recordPath, recordBody);

    await Promise.all([
        page.waitForNavigation(),
        page.goBack()
    ]);

    await page.screenshot({path: "screenshots/item"+int+"/base/base.png", fullPage: true});
    let baseBody = await page.evaluate(() => document.body.innerHTML);
    await saveHtml(basePath, baseBody);

    int++;
    console.log(record);
}

async function makeDirectory(path) {
    mkdirp(path, function(err) {
        if (err) throw err;
    });
};

async function saveHtml(path, html) {
    await fs.writeFile(path, html, (err) => {
        if (err) throw err;
    });
};

Note: I'm required to use XPath :(
Updated 6/25/18
This now gives me all the links from the xpath selector. Which i then iterate over and just use a page.goto, to go to the correct site.
const linksXPath = '//table[2]//tr[td[a]]//td[1]/a';
const links = await page.evaluate((selector) => {
    let results = [];
let query = document.evaluate(selector,
  document,
  null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
  for (let i=0, length=query.snapshotLength; i<length; ++i) {
    results.push(query.snapshotItem(i).href);
  }
    return results;
}, linksXPath);

Comment: @Vaviloff thanks, here you go.
Comment: Please prefer inserting your code into the question as text rather than an image, it makes testing your code possible.
Comment: Did you fix this issue using xpath?
Answer: I think it is your selector that is the issue.
I believe your selector for the table should be:
"body > table > tbody > tr:nth-child(2) > td > table > tbody > tr:nth-child(1) > td > table.bodytext > tbody"

The easiest way to get the right selector for a page is to use the Chrome Dev Tools.
Inspect the page, and go to the "Elements" tab. From there, you should see all of the HTML elements. Right click on the one you want (I went for
 so you can iterate through the
 elements.) and choose copy > copy selector.
Comment: I would use that but I'm required to use XPath
Comment: Ah, sorry - I was actually under the impression XPath was only used to work with XML documents. Is there an actual reason for wanting to use XPath, or is it a purely academic requirement?
Answer: My code is now doing what I need it to do, however I wish there was a simpler way of doing this. Also, you'll see as I iterate over the links, i'm using a page.goto function to go there. I still don't know of a way to use page.click. I would have to use the xpath to get all the td's and then click on them, but i was never able to get that to work. So here is the working product:
const puppeteer = require('puppeteer');
const fs = require('fs');
const mkdirp = require('mkdirp');

async function run() {
    const pageToClick = 'body > table > tbody > tr:nth-child(3) > td > table > tbody > tr > td > form > table > tbody > tr:nth-child(3) > td > div > input[type="submit"]';
    const select = 'body > table > tbody > tr:nth-child(3) > td > table > tbody > tr > td > form > table > tbody > tr:nth-child(1) > td:nth-child(2) > select';
    const inputField = 'body > table > tbody > tr:nth-child(3) > td > table > tbody > tr > td > form > table > tbody > tr:nth-child(2) > td:nth-child(2) > input[type="text"]:nth-child(1)';
    const linksXPath = '//table[2]//tr[td[a]]//td[1]/a';
    const browser = await puppeteer.launch({
        headless: true
    });
    const page = await browser.newPage();
    await page.goto('https://hrlb.oregon.gov/bspa/licenseelookup/');
    await page.select(select, 'lastname');
    await page.focus(inputField);
    await page.keyboard.type('a');
    await Promise.all([
        page.waitForNavigation(),
        page.click(pageToClick)
    ]);

    const links = await page.evaluate((selector) => {
        let results = [];
        let query = document.evaluate(selector,
            document,
            null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
        for (let i=0, length=query.snapshotLength; i<length; ++i) {
            results.push(query.snapshotItem(i).href);
        }
        return results;
    }, linksXPath);
    const basePic = await page.screenshot({fullPage: true});
    let baseBody = await page.evaluate(() => document.body.innerHTML);
    let int = 0;
    for (i = 0; i < links.length; i++) {
        await Promise.all([
            page.waitForNavigation(),
            page.goto(links[i])
        ]);

        await Promise.all([makeDirectory('screenshots/item'+int), makeDirectory('screenshots/item'+int+'/base'), makeDirectory('screenshots/item'+int+'/record')]);
        let recordPath = "screenshots/item"+int+"/record/record.html";
        let basePath = "screenshots/item"+int+"/base/base.html";
        let basePicPath = "screenshots/item"+int+"/base/base.png";

        await page.screenshot({path: "screenshots/item"+int+"/record/record.png", fullPage: true});
        let recordBody = await page.evaluate(() => document.body.innerHTML);
        await saveFile(recordPath, recordBody);

        await Promise.all([
            page.waitForNavigation(),
            page.goBack()
        ]);

        await saveFile(basePath, baseBody);
        await saveFile(basePicPath, basePic);

        int++;
    }
    await page.close();
    await browser.close();
}

async function makeDirectory(path) {
    mkdirp(path, function(err) {
        if (err) throw err;
    });
};

async function saveFile(path, html) {
    await fs.writeFile(path, html, (err) => {
        if (err) throw err;
    });
};

run();


How and why has figure skating scoring changed?

Question: I remember that figure skating used to be scored on a 6.0 scale.  After the skater was done with his or her routine, the judges would each rate the routine from 0.0 - 6.0, and the results from the judges were averaged together.
Watching the Sochi 2014 Men's Short program last night, I see that the scoring has completely changed.  The commentators mentioned that Yuzuru Hanyu set a new world record with his short program score of 101.45.
Why was the scoring changed for figure skating, and how is the new system different from the old system?
Answer: The rules changed after there was a judging scandal at the 2002 Olympics in Salt Lake City in the pairs event. Following this, it was decided that there needs to be more transparency and clarity in judging.
The technical score because the element score where all elements (jumps, spins, step sequences, dance segments) are scored separately. Each element is assigned a level of difficulty (jumps are exception). For instance a level two spin will have 8 revolutions in a difficult position while a level one spin will only have one of these feature. This is called by a technical panel of three people. Each called element is then judged on execution by the main panel of judges. For instance, an exceptionally high jump with a difficult entry and exit would bring more points than a jump landed on two legs from a standard entry.
The second score are the components. These are judged on a scale of 0-10. There are 5 components (skating skills, transitions, performance, choreography, interpretation). This works like the old 6.0 system in that most judges probably use it as a ranking device (although there are guidelines on how to mark components). These are later multiplied by a weight (factor) and added to the element score. In men free skate at the Olympics all components have weight of 2 which means that skaters could obtain a maximum of 100 points here. In the short programme the weight was 1 so the maximum was 50. Ladies have lower weights. These weights were chosen such that skaters should care not only about doing the elements but also about the overall quality. In dance the weights are different for different components.


Plotly python bubble chart - add text

Question: I need to add text to a plotly bubble chart with python. I can only get the text attribute to take 1 value per datapoint. However, for each datapoint I need to show two values on the tooltip: its size, and another value. I'm using a list of lists, loosely based on an example here.
This is my code:
bubbletext = dfenv[['Max_FZ','Argmax_FZ']] # dataframe with the 2 cols I need for text
bubbletext = bubbletext.values.tolist() # puts the df into list of lists

trace1 = Scatter(
         x=dfenv['X (ft)'],
         y=dfenv['Y (ft)'],
         text=bubbletext,  # here's the problem
         mode='markers',
         marker=Marker(
            size=dfenv['Max_FZ'],
            sizeref= dfenv['Max_FZ'].max() / 1e2**2,
            sizemode='area'
            )
         )
data = Data([trace1])
layout = Layout(showlegend=False)
fig = Figure(data=data)#, layout=layout)
plot_url = py.plot(fig, filename='Envelope_MaxBubblechart')

Answer: Actually, you can concatenate a new string column for the text of the tooltip as follows:
dfenv['text'] = dfenv['Max_FZ'].round(1).astype(str) + 'units' + '
at: ' + dfenv['Argmax_FZ']

trace1 = Scatter(
     x=dfenv['X (ft)'],
     y=dfenv['Y (ft)'],
     text=dfenv['text'], #string df column here with line breaks if needed
     mode='markers',
     marker=Marker(
        size=dfenv['Max_FZ'],
        sizeref= dfenv['Max_FZ'].max() / 1e2**2,
        sizemode='area'
        )
     )
data = Data([trace1])
fig = Figure(data=data)
plot_url = py.plot(fig, filename='Envelope_MaxBubblechart')

Plotly python bubble chart - add text


MySQL remote access on Ubuntu 16.0.4

Question: The followings are commands which I have executed on Ubuntu.
$ mysql -u root -p

mysql>GRANT ALL ON my_database.* TO user@'MY_REAL_IP' IDENTIFIED BY 'passwd';

$ sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

 #bind-address           = 127.0.0.1

$ sudo ufw enable

$ sudo ufw allow 3306

$ service mysql restart

But when I use another computer on Ubuntu to access mysql on MY_REAL_IP type:
$ mysql -u user -p -h MY_REAL_IP

Can't connect to MySQL server on 'MY_REAL_IP'

how to fix it? thx
Comment: Please check `telnet  3306` ?
Comment: `telnet: Unable to connect to remote host: Connection refused
`
Comment: This is the root cause, at first you need to establish network connectivity.
Answer: I guess bind-address should be 0.0.0.0. If you would like to access this server with real IP address. Or, you can access MySQL with simply localhost.
Would you please change bind-address = 0.0.0.0, and restart MySQL ?
EDIT
It looks there is no network connectivity. Please find the way to connect server with port 3306. How to make sure that firewall is off?
Comment: I turn `#bind-address           = 127.0.0.1` into `bind-address = 0.0.0.0` And the message was same
Answer: You may misunderstand the grand command, user@host means the user can login from host, which in your case, should be another computer's ip, or just use * for all ips
Comment: Is that you mean change into like this? `GRANT ALL ON my_database .* TO user@'*' IDENTIFIED BY 'passwd'`  And the message was same
Comment: @JonathanCheng Yes, and please make sure your.mysql server is up running and listening port 3306. Lots of reasons could lead to your issue, first of all you should understand every commands or options you are using, e.g. 'bind address'
Answer: Exact path on Ubuntu 16.04
/etc/mysql/mysql.conf.d/my.cnf
bind-address = 0.0.0.0
service mysql restart

MySQL remote access on Ubuntu 16.0.4


What is the use of taking $t\in (0,1)$ in proving monotone convergence theorem?

Question: Let $(X, S, \mu)$ be a measure space.Let $0\le f_1\le...$ be an increasing sequence of $S$ measurable functions. Define $f:X\to [0,\infty]$ by $f=\lim_n f_n$.
Then $\lim_{k\to \infty} \int f_k = \int f.$
Proof: $f$ is measurable as it is limit of measurable functions. $\forall x\in X, f_k(x)\le f(x)\implies \int f_k\le \int f$.
Now the proof goes along the following lines: Choose $c_i\in [0,\infty)$ and disjoint sets $A_i$, $i=1,2,...,m$ in $S$ such that $f(x)\ge \sum_{i=1}^m c_i \chi_{A_i}(x)$.
Take any $t\in (0,1)$ (I don't understand why this is required. I think one could take $t=1$ at this stage.). Define for every $k\in \mathbb N$, $E_k=\{x\in X: f_k(x)\ge t\sum_{i=1}^m c_i \chi_{A_i}(x)\}.$
Now, $E_k$'s are increasing and $\bigcup_k E_k=X$; so for every $j$, we have $\lim_k \mu(A_j\cap E_k)=\mu(A_j)$. $f_k(x)\ge t\sum_{i=1}^m c_i \chi_{A_i\bigcap E_k}(x)$ for every $x\in X$.
Then it follows that $\int f_k\ge t\sum_{i=1}^m c_i \mu(A_i\bigcap E_k)\implies \lim_k\int f_k\ge t\sum_{i=1}^m c_i\mu(A_i).$ Taking sup over the disjoint sets $A_i$ and $c_i$'s we get $\lim_k\int f_k\le t\int f$. Letting $t\to 1$ proves the statement.
Suppose that I take $t=1$ in the beginning itself.  Then also the proof works as we get $\cup_k E_k=X$ (pf: Take any $x\in X$. If $x\in A_i$ for some $i$, then $f(x)\ge c_i$. Hence, there exists some $k$ such that $f_k(x)\ge c_i\implies x\in E_k$, where $E_k=\{x\in X: f_k(x)\ge \sum_{i=1}^m c_i \chi_{A_i}(x)\}$ ($t=1$). So $x\in \bigcup_k E_k$) And $E_i$'s are increasing as before. Rest of the steps are word for word with $t=1$.
So my question is: why was $t\in (0,1)$ taken in the beginning and then $t\to 1$ was done later?
Please help me understand this. Thanks.
Answer: '$f(x)\ge c_i$. Hence, there exists some $k$ such that $f_k(x)\ge c_i$':
This is wrong. If $f(x)=c_i$ we can have $f_k(x) <c_i$ for every $k$ but still have $f_k(x) \to f(x)$. (For example, $1-\frac 1 n <1$ for all $n$ but $\lim (1-\frac 1n )=1$ (so $\lim (1-\frac 1n )\geq 1$).
Comment: I was thinking: suppose $x_n\to L$. $L>0\implies x_n>0$ for some $n$. Proof: Suppose on the contrary $x_n\le 0$  for all $n$. Then, $L\le 0$, a contradiction. I see that I need strict inequality to pass that claim. But still, I don't understand how introducing $t$ helps there.
Comment: The point is you only have $f(x) \geq c_i$. If you had $f(x) >c_i$ that would work To get strict inequality you have to use $f(x) >tc_i$ where $0<t<1$ @Koro
Comment: I think you meant $f_k$ instead of $f$.
Comment: I understood now why. So take any $x\in X$. If $x\in A_i$ for some $i$, then $f(x)\ge c_i$. Suppose on the contrary that $f_k(x)< tc_i$ for all $k$, then $f(x)\le tc_i<c_i$, a contradiction. So this is what $t$ is for. Thank you so much.


R-squared from conditional mean equation in GARCH models (rugarch package)

Question: I was recently asked to report the r-squared statistics together with the estimations of GARCH models with exogenous regressors on the conditional mean equation. However, there is no function to get these statistics automatically using the rugarch package.
My question is straightforward:
Can I use the residuals of the fitted model (uGARCHfit class object) to calculate the r-squared and the adjusted r-squared manually or should I use the estimated sigma from the fitted model to weight an lm object and extract the r-squared from there?
I did some research but I couldn't find anything in the vignette of the package.
Two exchanges on cross-validated touched the issue:
How to measure the goodness of fit of a GARCH model?
How to extract the adjusted r squared from uGARCHfit class data?
But they don't provide a definitive answer.
Here is a reproducible example of what I want to do:
library(rugarch)
library(zoo)
library(dplyr)
library(purrr)
library(tidyr)
library(fredr)

# Get and clean the data

fredr_set_key("your_key_here") # fredr package requires a key to download the data.

df = map_dfr(c('DEXBZUS', 'DCOILWTICO'), fredr) %>%
  filter(date > '2010-01-01')

df1 = df %>%
  spread(series_id, value) %>%
  set_names('date', 'oil', 'brl') %>%
  mutate_at(vars(-date), list(~ log(.) - lag(log(.)))) %>%
  drop_na

oil = as.matrix(zoo(df1$oil, df1$date))
brl = as.matrix(zoo(df1$brl, df1$date))

# Estimate the model
m1_spec  = ugarchspec(variance.model = list(model = 'eGARCH',
                                        garchOrder = c(1,1)),
                  mean.model = list(armaOrder = c(1,0),
                                    external.regressors = oil),
                  distribution.model = 'norm')

m1_fit = ugarchfit(spec = m1_spec, data = brl,
                 solver.control = list(trace = 1))

# Calculate the r-squared statistics
r2 = 1 - (sum(residuals(m1_fit)^2) / sum((brl - mean(brl))^2))
r2a = 1 - ((1 - r2) * (length(brl) - 1) / (length(brl) - 3))
```

Comment: My answer in the thread [How to measure the goodness of fit of a GARCH model?](https://stats.stackexchange.com/questions/154066) that you linked above focuses specifically on fitting the conditional variance, not the remaining aspects of the model (conditional mean, other distributional properties); perhaps I should clarify that there. If you focus on the conditional mean (which is one of the equations that constitute a GARCH model), the usual $R^2$ should be fine.
Answer: Yes, you can use the residuals of the fitted model (uGARCHfit class object) to calculate the $R^2$ and the $R^2_{adj.}$ manually. Note: you need raw residuals, not standardized ones.
You do not use the estimated sigma from the fitted model to weight an lm object and extract the $R^2$ from there; the classical definition of $R^2$ does not involve any weighting.
The last line of your code for calculating $R^2$ appears to be correct.


Where is finitely generated used in the proof below.

Question: This is regarding Proposition 3.14 of Atiyah and Macdonald. It says that if $M$ is a finite $R$ module then the formation of localization and annihilators commute. This is well known, and the proof is clear. But I cannot find where the mistake in the proof below is and the proof below does not require $M$ to be finitely generated, so there is an error.
Let $r/s \in Ann(S^{-1}M)$. Thus $r/s.m/t=rm/st=0$, which implies that $urm=0$ for some $u\in S$. So $ru\in Ann(M)$. Hence $r/s=ru/su\in S^{-1}(Ann M)$, which shows that $Ann(S^{-1}M)\subseteq S^{-1}(Ann M)$.
Conversely, let $r/s\in S^{-1}(Ann M)$, where $r\in Ann M$. Consider $m/t\in S^{-1}M$. Note that $r/s.m/t=rm/st=0$. Hence $r/s\in Ann(S^{-1}M)$.
Where is the error? Thanks.
Comment: The $u $ might depend on the $m $. Being explicit about where you introduce your variables, what they are, and where their scope ends would have helped you avoid this.
Comment: you are right, thanks.
Answer: Just to remove this from the unanswered list, here I write down the proof suggested by Darij Grinberg.
Let $M$ be generated by $m_1,\ldots, m_t$. By the first part of the above faulty proof, there is a $u_i\in S$ such that $u_irm_i=0$ for all $1\leq i\leq t$. Thus $ru_i\in Ann(m_i)$ for all $1\leq i\leq t$. Set $u=u_1\cdots u_t$. Note that $ru\in \bigcap_{i=1}^t Ann\; m_i=Ann M$. Note that $r/s=ru/su\in S^{-1}(Ann M)$.


htaccess - how to redirect /q/ to /#/q/?

Question: Here is my Rewrite Rule :
RewriteEngine on

RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
RewriteRule ^(.*)$ https://%1/$1 [R=301,L]

RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

RewriteRule "^/q/(a-zA-Z0-9)*/?$" "/#/q/$1"

Something does not work, I don't understand what.
I want to redirect /q/xxx to /#/q/xxx
How can I do it ?
Answer: here is the solution: RewriteRule ^q/([a-zA-Z0-9-]+)/?$ /#/q/$1 [L,NC,R=301,NE]
The flags were missing


microsoft virtual server multiple cpu configuration

Question: How can i configure multiple virtual cpu in microsoft virtual server 2005R2. If there is any other virtual server which supports multiple virtual CPUs please let me know.
Comment: Can you be more specific please, what are you trying to achieve - there's bound to be lots of great options for you, VS2k5 is WAY old and everyone's years ahead of it, but let's pick you the right product by understanding your requirements fully first.
Answer: You can't.  The product doesn't support multiple virtual CPUs within a single VM.
Since you asked, almost any virtualization product issued in the last few years supports multiple virtual CPUs.  Since you're already using Microsoft's virtualization, you should probably look at Hyper-V.
Answer: From Microsoft "Virtual Machine Emulated Hardware"

If Virtual Server is run on a multiprocessor hardware system, each
  hosted virtual machine session will only have a single CPU available
  to it in its emulated hardware environment.
Virtual Server can
  distribute virtual machine sessions amongst multiple processors if
  available, but each session will only have a single emulated processor
  available to it.

You can read it here:
Microsoft Virtual Server 2005 Virtual Machine Emulated Hardware


Good machine learning models for confusable categories

Question: I'm using the word confusable to represent similar looking glyphs in text.
I'm building an optical character recognition tool with the primary goal of experimenting with machine learning – especially neural nets, which I already have some knowledge about. There is one issue that I just can't get my head around: Representing confusables.
As an example I (capital I) and l (small L) looks the same in sans-serif fonts. If you extend it to the complete Unicode range, there are a lot of glyphs that look more or less the same. If I'm using supervised learning naïvely, I'm afraid that the training algorithm won't be effective because of these confusables.
Maybe I need unsupervised learning. I could use pretraining on RBMs or stacked auto-encoders, but I'm not sure you can assume that all confusables are close in the output space. They might be. The task still remains to detect these areas and make a good representation.
I'm basically looking for a good model to cluster confusables, which I can use to create an improved supervised learning model. I expect to do some post-analyses on the output of the classifier in order to resolve each confusable to a real Unicode character.
Answer: I don't think the "confusables" problem is as big as you think it is. Supervized learning algorithms are designed to find the differences between classes, even if the classes have resembling symbols. For instance, the MNIST data, a standard optical recognition dataset used to test many techniques, contains handwritten digits, and to my knowledge none of the standard methods (random forests, boosting, SVM, MLP, deep networks...)  require any specific pre-treatment because of the fact that handwritten "6" resemble "8". Or "1" and "7", for that matter.
You might get a problem if your classes are of uneven sizes : if you have much more "7" than "1" in your training data, the model might be tempted to class all 7s and 1s as "7" to reduce error. But that is a different question, not directly related to the problem you raised.
Comment: I think the difference to MNIST is scaling up the number of classes. Unicode does have a lot of confusables. I was just googling and [found this table of Unicode confusables](http://www.unicode.org/Public/security/latest/confusables.txt). I could possibly use this list to create the categories for supervised learning. But maybe, as you suggest, it is not a big problem in the end. I will have to try it out. Regarding training data, I'm creating it automatically. I want to solve OCR of screen dumps, and it is possible to auto-generate a lot of training/validation data.
Answer: This problem can be solved on the higher level - when you move from symbols to the words and pick not the all most probable symbols, but the most probable symbol combination based on vocabulary and context.
Comment: I totally agree that it can be solved on a higher level. I want to get the best data for this level. As an example, my classifier could output: `l` (48%), `I` (47%), `1` (5%). And the higher level would have a better chance at getting it right.


Chi-square test with replicate nested

Question: I have a question about how to analyse count data with replicates nested in each treatment. For example, imagine temperature can influence the sex ratio of mosquito larva emerged from eggs. I have two temperatures, 27 degree and 37 degree. For each temperature treatment, two replicates are used, eg. in temperature 27, there are 32 male larva and 18 female larva for the first replicate; and 30 male/ 20 female for the second replicate. My main purpose is to determine if there is significant difference of sex ratio between the two temperature regimes.
Clearly, if there is only one replicate for each treatment, chi-square test would be the best choice to analyse the data. But with the two replicates, I don't know what should do? Merge two replicates in each treatment as one? I don't think so.
Answer: I don't know what software you are using, but I will be using R to show how to perform a 2x2 table proportion test. First recreate your table, merge the sexes in each temperature
dat=matrix(c(62,38,46,54),nrow=2,byrow=T)
dat
     [,1] [,2]
[1,]   62   38
[2,]   46   54

And the test
prop.test(dat)

    2-sample test for equality of proportions with continuity correction

data:  dat
X-squared = 4.529, df = 1, p-value = 0.03333
alternative hypothesis: two.sided
95 percent confidence interval:
 0.01364509 0.30635491
sample estimates:
prop 1 prop 2
  0.62   0.46

Comment: I am also using R. My problem is how to deal with the two replications in each treatment. In your reply, you simply add them together to create one 2X2 table. I am wondering if there are other more reasonable method .
Comment: @EvenGuan if you are worried about the replicates, you could use generalized linear models with replicates as random effects to account for the replicate variablity.

Chi-square test with replicate nested


What is good software for learning how to sight-read?

Question: I'd like to learn how to sight-read on my own, but practicing the same song over and over again just causes me to memorize it, and filling out worksheets is boring.
Is there any good software for learning how to sight-read?  I'm envisioning something that just drills you with note after note, then later chord after chord, until you can react instantly to seeing it in sheet music.
Bonus points if it supports a MIDI-interface, so you can play with an actual instrument while you learn.  It would also be nice if it had ear training, but perhaps that's another question...
Comment: Also, learning from the PC (besides easy practice) additionally gave me one extra benefit: I noticed that I immediately gave up looking at the keys when practising with a program! This is clearly a double-win approach. (I've tried *Jalmus* from this list.)
Comment: Please, if you want to answer this question (which is borderline and might be closed), just **one software** proposal by answer, and be very careful not to enter duplicates. If someone has already suggested your favorite software, vote it up, and add comments if needed.
Comment: One thing to note is that if you want to learn how t sight read fluidly, it's important to be able to read the intervals between notes, not just the notes themselves. Emphasis on just quickly reading each note sells you short.
Answer: Jalmus provides what you are looking for and it has a MIDI interface. It is also free open source, is cross platform (written in Java, it works on Windows, Linux, and Mac), and is available in English, French, Spanish, Italian, Danish and German.
From the website:

Jalmus is a free, open source music
  education software helping the
  musicians, especially pianists, to
  improve their sight-reading. You can
  train to read music with both
  exercises on notes or rhythms.

Update development of Jalamus stopped at version 2.3, but the software is still available for download at sourceforge
Comment: Just noticed that Jalmus has been discontinued: http://www.jalmus.net/News,1183.html
Comment: There are lots of good answers, and I haven't had a chance to try them all yet (I'll mark my favorite as accepted, after I've tried them all).  However, I am giving this one the bounty because, though it is very ugly, it not only looks like it does everything I'd like, but it's open-source!  I may even consider contributing to this if I like it enough...
Comment: @evanrmurphy not "discontinued", but "fixed" at version 2.3; development has stopped but the software is still stable and available.
Comment: @David their news feed says "discontinued", which is why I used that word. Point taken though that version 2.3 is still available.
Answer: Are you sure you want a software solution at all? An alternative is a large supply of small pieces, like Bartok's Mikrocosmos. Just keep playing different ones. One level of that will keep you entertained for quite a while. I particularly like the song about the foxes and the chickens.
Sight-reading is not just about connecting your eyes and your fingers. It's not just about translating notation into notes. It's also about reading the idea of some music -- learning to anticipate a bit (when the composer is not planning to pull the rug out from under you in the next measure.) Software is not going to help so much with that, unless it passes a sort of musical Turing test.
Comment: @11684 If the player can stay within the scale (particularly easy, for, say, a beginner pianist playing in C major), there's not much of a way to tell. Unless of course, your aim is to play what YOU think sounds nice, rather than what's written.
Comment: Playing small pieces is fine - but to learn sight reading, you need feedback to tell you if you've played the wrong note or rhythm. A teacher can do this, but having software do it would be convenient and maybe cheaper.
Comment: @11684 If you've never heard the tune before, and just have a score to read, how do you know how it should sound? Only through skill at sight reading.
Comment: I hope you can hear you played the wrong notes. At least if you're playing tonal music... @slim
Comment: Nope, you can hear a wrong note in scales, or tonal chords, even if you don't see and know the score (okay, not always). At least I can.
Answer: One option is to use (and abuse) Impro-Visor. Impro-Visor is a jazz improvisation trainer, but it has built into it a "lick generator", which builds melodies over existing chord changes using grammar rules. (The program is of course also capable of generating chords.) By putting in various grammar rules you should be able to adapt the program to also work for other styles of music that is less directly related to Jazz.
(And yes, it supports MIDI interface.)
Comment: Heh, funny; I actually looked into those before for something [completely separate](http://stackoverflow.com/questions/5525959/are-there-any-music-notation-rendering-libraries-for-java), but never put any thought into it for this.  I'll take a look at it - thanks!
Answer: I was looking for something like this once too.
One I've used and liked because first I can put the note names to be Do, Re, Mi, Fa... or A, B, C, D... and it's free if you used it on the website is this one: http://www.emusictheory.com/practice.html
They drill you with note after note and you need to say which it is. It also has the average time it takes you to identify each note. You can set a goal to lower it. No MIDI though.
They have a bunch of "drills" as they call them and are free as long as you use it them in the site.
Personally I wouldn't use something like this with the instrument necessarily, but I guess it depends on what you play just because being able to read the music and playing them on the instrument are different things. You can know the name of the note you are reading and even the fingering, but have bad intonation. You can also have great intonation and not be able to read music. They are separate skills.

Another (free) site that allows customization of exercises: http://www.musictheory.net/exercises/
  I like this one because it allows you to work out acute kinks one by one (through customization) and then broaden your fluency.
Comment: It's not free to hook up to a midi keyboard.  However, this is exactly what I was looking for, so if I can't find anything better, I think I will have to pay the $12.
Answer: The best one I've found so far is Noteable.
It drills you on different notes, allowing you to choose which notes and/or key signature(s) you want to practice, and even learns which notes you're suffering on and drills you on those ones more.  It supports MIDI interface.
As far as I'm aware, though, there's no support for practicing chords, and it's not free.
Answer:
Is there any good software for learning how to sight-read?

I've spent the last year working on a software product called SightReadingMastery to help people with sight reading. A few things about it to note:

It's web-based, so you don't have to download or install anything
It's not free, but there's a 7-day free trial, so you can try it out and see how you like it without paying
It works by giving you real pieces of music (I team up with professional composers) at the difficulty level you choose; you sight read these from the comfort of your instrument
Each piece comes with a correct performance in high-quality audio; you're encouraged to listen to this after sight reading and use your ear to check if you played the piece correctly, rather than relying on any kind of MIDI-based evaluation system to tell you how you performed

Feel free to comment here or email me at firstname.lastname@example.com if anyone has questions. Thanks for considering my product.
Answer: Musition 4.0 is available for Mac and Windows and is very broad in its coverage of theory and solfège, so it exceed certainly what you are looking for.
About note reading, it covers among other things individual notes, chords, all kind of scales, both letter and solfège name for notes and interactivity with a midi keyboard.
Compared to previous version, there are more elementary lessons, more progressive on most topics.
It is not free, but you can download a functional demo.
Answer: A free  online site that supports MIDI is Sight Reading Practice, set it up and pick MIDI See-and-play.
It supports a lot of configuration settings so you can practice under different settings...
Answer:
Is there any good software for learning how to sight-read? I'm envisioning something that just drills you with note after note, then later chord after chord, until you can react instantly to seeing it in sheet music.

I have a program that does this.  http://PianoCheetah.com

It displays midi files in piano roll format and sheet music format
displays the notes you play (via MIDI) -on top- of the sheet music you're playing
It can also pause on any note that you miss until you play it.
for "drilling", if you load a song called RandomSong, it will auto-generate a random song of chord progression in certain arrangements that are always playable with 2 hands.

(no ear training stuff, though, really)
(My original answer seems to have been turned into a comment, and then deleted for a reason of "see the FAQ"...  grumble)
Answer: If you have an iPad, try sightread4piano by Wessar.
You can buy individual grades from your music authority of choice, or if you really fancy it,  you can buy over 1,100 pieces of music covering all the grades from some of the World's top music examination boards.
The concept is amazing and it really does teach you how to sight read.
Answer: I have answer this also in another topic. for quick reference you can find some info from the screenshots.
Answer: A website that my band teacher uses is Sight Reading Factory. I think that it is a very nice piece of software. It has a lot of good features

It allows you to choose almost any instrument, including vocals.
You can choose a difficulty from one to 6
You are able to choose any of the major or minor key signatures
Can choose from 4 different time signatures.

This does cost money, but there is a trial version, where they give you 5 free samples for every set of samples you do. So if you only change one thing, such as the key signature, you get five more free pieces. But if you want to buy it, it is $30/year
Answer: I use my mobile phone with Solfa. Mobile device allows me to learn on way to and from work (train, bus) and also somehow feels more convenient and "ready" than a desktop computer or laptop. You cannot take your piano on a bus.
This program allows to configure the level of difficulty (which clefs, should notes above/below the clef be included, should sharps and flats be included) and uses either blank or letter-labelled on screen piano keyboard to enter answers. This keyboard is playable (not a great sound but seems correct pitch) so the tool also trains ear in some degree.
Nothing bad can be said about this program really; it does all (apart MIDI), costs like a cup of coffee  and runs on a wide variety of smartphones (Android, iPhone, Windows Phone, Amazon Android, etc).
Answer: Another one I just found:  Finale.  It claims to have flash cards and "music-themed puzzles and games" to help learn various aspects of music playing and theory.  However, I've never tried it.
It supports MIDI interface, and is not free.
Comment: Finale is a bit feature rich for this purpose and not exactly intuitive.
Answer: Adventus Piano Suite Premier Piano (course-based), eMedia Piano & Keyboard Method (feedback-based) both allow you to learn sight-reading and supports MIDI, they are however not free. There are tips to improve your sight-reading on their introducary page...
Comment: I've seen *"eMedia Piano & Keyboard Method"* in the past and was not impressed.
Answer: Here's another one:  PrestoKeys.
The free version has pretty much all the features of Noteable, but with nicer graphics.  It also supports accidentals, and a timed scrolling mode.  and supports practicing songs (stored as MIDI files) rather than just random notes.
However, it doesn't support MIDI input.
The good news is that there's a paid version which does support MIDI input.  It also allows you to practice chords and key-signatures.
The paid version is $30.
Answer: I personally use Band-in-the-Box for the purpose of practicing (or better say upkeep) my sight-reading.

It can generate endless songs in all styles You can dream of
You can slow it down to match you reading skills
It generates, chords, melodies and solos so You can practice different qualities of sight-reading
Those are songs - so You learning in real musical context no just exercise for exercise
You can loop generated songs. (You can choose keys, styles, postions and so one - nor to songs will be the same)

Of course this is not a learning software, so maybe You will need some preparatory exercises done elsewhere. it's not quite cheap (regarding only this feature)
But regarding all other great feature BIAB sports for musician this cool another use case.
Answer: You may also want to consider Piano Marvel, which does not randomly generate notes and chords like Jalmus; it is more thorough than the any of the other programs listed thus far for developing functional sight-reading skills.  With it you sight-read actual compositions, more of them than you could ever dream of buying or borrowing--it includes a large library of classical pieces updated daily, ranging widely in scope and difficulty among the traditional repertoire, IE, from a Schubert Ecossaise, to J.S. Bach inventions, to Debussy preludes, and even Chopin's etudes and some gospel.  They are decently prepared editions including fingerings, dynamics, and tempo markings.  It runs very smoothly and relatively bug free as far as I can tell, and it does score you.  It is free for the first month and costs $15 a month after that, but the quality is on par with the ear trainers Auralia, Ear Master Pro, and the theory trainer Musition.


How to enable and disable a CRM 2011 Organization using C#/VB.Net and WCF Deployment Service or CRM 2011 SDK?

Question: I want to be able to enable/disable a CRM 2011 Organization using Deployment Service (Deployment.Svc) or using CRM 2011 SDK (Microsoft.Xrm.Sdk.Deployment Namespace) using C# or VB.Net application.
I am using some code like this:
<code> Dim StateReq As CRM2011DeploymentSvc.DeleteRequest
                Dim StateResp As CRM2011DeploymentSvc.DeleteResponse

                StateReq.EntityType = DeploymentEntityType.Organization
                StateReq.InstanceTag.Id = FoundOrganization.OrganizationId
                StateReq.InstanceTag.Name = FoundOrganization.UniqueName

                StateResp = CType(_CrmDeployService.Execute(StateReq), DeleteResponse)
</code>
1. Q1: This code is OK as far as deletion of entities(Organization, accounts etc) is concerned. but i am more interested in Enabling and disabling the organization (occasionally) along deleting (when required).
2. Q2: Lets say if i Delete an Organization, will i be able to undo delete or recover it. If yes then how and if not what else is the alternative?
3. Q3: In CRM 4.0 we can disable/Enable the organization using
<code>**SetStateOrganizationRequest**

**SetStateOrganizationResponse**
</code>
Classes using the Deployment web service but in CRM 2011, the Deployment WCF Service does not contain any such method. What should i do to enable or disable the organization?
Replies are much appreciated.
Thank you.
Answer: ok i got the solution courtesy of ResultOnDemand
The following code worked for me
<code>EntityInstanceId i = new EntityInstanceId();
i.Id = OrganisationId; //Organisation Id

DeploymentService.Organization organization = (DeploymentService.Organization)Provider.deploymentservice.Retrieve(DeploymentEntityType.Organization, i);

//Update status to disabled
organization.State = OrganizationState.Disabled;

DeploymentService.UpdateRequest updateRequest = new UpdateRequest();
updateRequest.Entity = organization;

//update status
Provider.deploymentservice.Execute(updateRequest);
</code>


Union of all sets such that $\left\{ c-b , b-a \right\}=\left\{ 1789 , 1830\right\}$

Question: If we consider all sets $\left\{ a;b;c\right\}$ such that $a;b;c\ge0$ and $a<b<c$ and
$\left\{ c-b;b-a\right\}=\left\{ 1789;1830\right\}$. Can we say that the set of all positive integers is the union of such disjointed sets ?
I was thinking about doing an induction, starting by $0$ we must have $\left\{ 0;1789;1789+1830\right\}$ or $\left\{ 0;1830;1789+1830\right\}$. Then we must have $\left\{ 1;1790;1790+1830\right\}$ or $\left\{ 1;1831;1789+1831\right\}$. And so on but i do not know if this works. Thank you for helping me.
Comment: A revolutionary question...
Comment: @JeanMarie, only for the French :)
Answer: So, you're asking about the set of all positive integers and then write that $a,b,c$ can be equal to $0$, and start your first triple with $0$. It doesn't really matter since the distance between numbers is what matters, not their actual values. I will also start with $0$.
Assume that the set of all admissible numbers is the disjoint union of such triples. We will call a number "$a$" if it's the smallest in its triple, "$b$" if it's the middle one, or "$c$" otherwise.
All numbers from $0$ to $1788$ are undoubtedly "$a$"s. We don't know quite how their "$b$"s are looking yet, but what we know for sure is that all numbers from $1789+1830$ to $1789+1830+1788$ are their corresponding "$c$"s. There are $1830$ numbers in between $1788$ and $1789+1830$. $1789$ of them are the "$b$"s for $0,…,1788$. The rest $1830-1789=41$ are "$a$"s.
Where are the $41$ "$b$"s of those "$a$"s? Consider the smallest of these "$a$"s. It is $41+1789=1830$ numbers away from the nearest free number. So clearly the last $41$ of those $1830$ numbers are "$a$"s for some new triples. (And the first $1789$ of those $1830$ numbers are "$b$"s for the numbers $0,…,1788$ in the natural order. The number $1789$ can be a "$b$" only for "$0$", $1790$ - only for $1$, etc. I believe, we don't really need this fact though.)
So what did we get yet? $1789+1830+1788$ first numbers are distributed, then $41$ "$b$" go which are of the from $a+1830$. It means that there are $41$ "$c$"s there later in the numbers row of the from $b+1789$. There are $1748$ numbers in between those $41$ "$a$" and "$b$"s. They are all clearly "$a$"s.
Their "$c$"s are later in the numbers row. They are equal to $a+1789+1830$ for the corresponding "$a$"s. There are $41+1830$ numbers between these "$a$"s and "$c$"s. $41$ of them are "$c$"s for the earlier $41$ triples. $1830-1748=82$ are "$a$"s for some new triples.
Their "$b$"s are after all the numbers which are already distributed, and they are of the from $a+1830$. The pattern repeats itself fully. The only thing that changes is the number of these "$b$"s with which we start. They went from $0$ to $41$ to $82$. All multiples of $41$. So in order to succeed it must be that $1789$ is divisible by $41$ but it's a prime number.
We've got a contradiction so the positive integers aren't the disjointed union of these triples.
Comment: i do not clearly see how did you manage to prove that the smallest of these "a" is $1830$. For me the smallest of these "a" could clearly be "1789 ? For exemple is you take $1830$ for $a=0$ then $1831$ for $a=1$ and so on your $41$"a"  will be integers from $1789$ to $1829$ ?


In the Visual Studio SQL editor, how do I get rid of the boxes?

Question: I usually create my SQL tables and stored procedures by writing a script inside Visual Studio. This works really well for me except for one simple annoyance: VS puts blue boxes around all the SQL queries and data-manipulation commands. The purpose of these boxes is to draw undue attention to the fact that VS thinks the query can be edited in "Query Builder."
I don't want to use Query Builder. I just want a nice, clean script that reflects my fantastic vision of what the DB engine should do. Blast it, Jim, I'm a programmer not a Microsoft Access hobbyist!
I do, however, like the syntax highlighting and source-control integration that VS provides.
So my question is this: How do I turn off the annoying blue boxes?
Answer: Open Visual Studio Tools --> options --> Database Tools --> General. Uncheck "Enable DML Markers"
Comment: Note that this option is not applied to currently open files.
Answer: Set the color of the SQL query outline to the same color as the background.


Standard definition of subnets.

Question: Reading Willard's General Topology, I found the following definition of a subnet of a net $f:D\to X$:

$g:E\to X$ is a subnet of $f$ if there exists an increasing $\varphi:E\to D$ such that $g=f\circ\varphi$ and $\forall \alpha\in D, \exists \beta \in E, \varphi(\beta)\ge\alpha$;

while, reading Kelley's General Topology, I found the following definition of a subnet of net $f:D\to X$:

$g:E\to X$ is a subnet of $f$ if there exists $\varphi:E\to D$ such that $g=f\circ\varphi$ and$\forall \alpha\in D, \exists \beta_0 \in E, \forall\beta \ge\beta_0,  \varphi(\beta)\ge\alpha.$

Clearly, Willard's definition is strictly stronger than Kelley's definition (i.e. every subnet according Willard is a subnet according Kelley but in general not the converse). Obviously we can use both definitions to obtain the standard theorems about subnets we were looking for, so it seems just a matter of tastes which definition we decide to use. However:

In the literature, which one of the two definitions has become standard? Willard's or Kelley's?
Comment: You may be interested in [this question](https://math.stackexchange.com/questions/1126609/different-definitions-of-subnet)
Comment: Thx for the comment... at this point I see that I have problems in searching topics in this site...
Comment: For what it's worth, so do I. I Googled it instead, found this question there. I think it's very difficult to search this site.
Answer: This is indeed mostly a matter of taste; Willard's definition is more in the style of subsequences (maybe easier to visualise), while Kelley's is more general (so easier to fulfil, which can be easier to construct in proofs e.g.) and more in the generality spirit of directed sets, as it were. A subnet in Kelley's sense need not be one in Willard's sense, but the difference is quiet subtle. In terms of translating to filters a third notion (AA-subnet) is also sometimes used. See the discussion in this thread e.g. Most papers I've seen tend to use filters or use Kelley's definition (but I haven't done a count of popularity).
Comment: A subnet of the net f from D into X is f restricted to a cofinal subset of D.  Is not that another equivalent definition?
Comment: @WilliamElliot no, that's much too restrictive; you don't get the theorem that a space is compact iff every net has a convergent subnet, which is one of the main reasons for defining subnet in the first place. A sequence in your proposal only subnets that are subsequences while there are quite a few compact but not sequentially compact spaces...

Standard definition of subnets.


Proving independence of $N(\mu,\sigma^2)$ sample mean $\bar X$ and variance $S^2$ by change of variables

Question:
Let $X_1,...,X_n$ are iid sample from $N(\mu,\sigma^2)$. Then $\bar X$ and $S^2$ are independent.

I was stuck on proving above statement.
The joint PDF of $(X_1, ... ,X_n)$ is given by
$$f(x_1,...,x_n)=\frac{1}{\sqrt {2\pi\sigma^2}}exp \bigg[-\frac{\sum_{i=1}^{n}(x_i-\mu)^2}{2\sigma^2}\bigg]$$
$$=\frac{1}{\sqrt {2\pi\sigma^2}}exp\biggl[-\frac{1}{2\sigma^2}\biggl\{\sum_{i=1}^{n}(x_i-\bar x_n)^2+n(\mu-\bar x_n)^2\biggl\}\biggl] $$
Now, consider the following transformation
$y_i=\bar x_n$ and $ y_i=x_i-\bar x_n, i=2,3,...,n$
then $x_1-\bar x_n = -\sum_{i=1}^{n}(x_i-\bar x_n)=-\sum_{i=1}^{n}y_i$
Thus $\sum_{i=1}^{n}(x_i-\bar x_n)^2=\biggl(-\sum_{i=1}^{n}y_i\biggr)^2+\sum_{i=1}^{n}y_i^2$
The joint PDF of $y_1,...,y_n$ is given by
$$f(y_1,...,y_n)=J\Biggl(\frac{1}{\sqrt {2\pi\sigma^2}}\Biggr)^n exp\Biggl[\frac{1}{2\sigma^2}\Biggl\{\Biggl(\sum_{i=1}^{n}y_i\Biggr)^2+\sum_{i=1}^{n}y_i^2+n(y_1-\mu)^2\Biggr\}\Biggr]$$
$$=g(y_2,..,y_n)h(y_1)$$
,where $J$ denotes the Jacobian, $g(y_2,..,y_n)$ is joint PDF of $y2,...,y_n$ and $h(y_1)$is marginal PDF of $Y_1$
I don't understand how the joint PDF of $y_1,...y_n$ could be broken into such two part. I guess $E(Y_1)=\mu, Var(y_1)=\sigma^2$ such that $Y_1$ follows $N(0,\sigma^2)$. So, I guess rear part of exponential, $\frac{J}{\sqrt {2\pi\sigma^2}} exp\Biggl[\frac{n(y_1-\mu)^2}{2\sigma^2}\Biggl]$, means $h$. But, I'm not sure because of multiple of $n$. Further, I don't know how $g$ could be derived from  that front part of the exponential. Please give me a hint!
Comment: Consider accepting answers to your questions if you feel they have fully answered your queries. I see you haven't marked any of your questions as answered.
Comment: This [note](https://jekyll.math.byuh.edu/courses/m321/handouts/mean_var_indep.pdf) has a detailed proof. Also https://math.stackexchange.com/questions/47350/proof-of-the-independence-of-the-sample-mean-and-sample-variance?noredirect=1&lq=1.
Answer: If you are trying to prove the independence of $\overline X$ and $S^2$ via change of variables, suggest you using an orthogonal transformation for ease of calculation. The result can be proved without finding the joint pdf of $(\overline X,S^2)$ and showing they are independent because the joint pdf factors as the product of two marginals.
Consider the transformation $$(X_1,X_2,\ldots,X_n)\to(Y_1,Y_2,\ldots,Y_n)$$ such that $$\begin{pmatrix}Y_1\\Y_2\\\vdots\\Y_n\end{pmatrix}=Q\begin{pmatrix}X_1\\X_2\\\vdots\\X_n\end{pmatrix}$$
, where $Q$ is an $n\times n$ orthogonal matrix with the first row $$\left(\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}},\ldots,\frac{1}{\sqrt{n}}\right)$$
Then, $$Y_1=\frac{1}{\sqrt{n}}\sum_{i=1}^n X_i=\sqrt{n}\overline X \quad\text{ and }\quad\sum_{i=1}^n Y_i^2=\sum_{i=1}^n X_i^2$$
Clearly, $$(X_1,X_2,\ldots,X_n)\in\mathbb R^n\implies (Y_1,Y_2,\ldots,Y_n)\in\mathbb R^n $$
The absolute value of the Jacobian determinant is $$|J|=\frac{1}{|\det Q|}=1$$
Further,
\begin{align}
\sum_{i=1}^n (x_i-\mu)^2&=\sum_{i=1}^n x_i^2-2n\bar x\mu+n\mu^2
\\&=\sum_{i=1}^n y_i^2-2\sqrt{n}y_1\mu+n\mu^2
\\&=(y_1-\sqrt{n}\mu)^2+\sum_{i=2}^n y_i^2
\end{align}
So joint pdf of $(Y_1,Y_2,\ldots,Y_n)$ is of the form
\begin{align}
f_{Y_1,\ldots,Y_n}(y_1,\ldots,y_n)&=\frac{1}{(\sigma\sqrt{2\pi})^n}\exp\left[-\frac{1}{2\sigma^2}\left\{(y_1-\sqrt{n}\mu)^2+\sum_{i=2}^n y_i^2\right\}\right]
\\&=\frac{1}{\sigma\sqrt{2\pi}}\exp\left[-\frac{1}{2\sigma^2}(y_1-\sqrt{n}\mu)^2\right]\,\prod_{j=2}^n \left\{ \frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{y_j^2}{2\sigma^2}\right)\right\}
\end{align}
It is now clear that $Y_1,Y_2,\ldots,Y_n$ are independently distributed with
$$Y_1\sim\mathcal N(\sqrt{n}\mu,\sigma^2)\quad \text{ and }\quad Y_j\sim\mathcal N(0,\sigma^2)\,,\quad j=2,3,\ldots,n$$
While we get the distribution of $\overline X$ from $Y_1$, we get the distribution of $S^2$ from $Y_2,Y_3,\ldots,Y_n$.

$\overline X$ and $S^2$ are independently distributed precisely because $Y_1$ is independent of $Y_2,\ldots,Y_n$.

Noting that
\begin{align}
\sum_{i=2}^n Y_i^2&=\sum_{i=1}^n Y_i^2-Y_1^2
\\&=\sum_{i=1}^n X_i^2-n\overline X^2
\\&=\sum_{i=1}^n (X_i-\overline X)^2
\\&=(n-1)S^2
\end{align}
, we have $$\frac{(n-1)S^2}{\sigma^2}\sim \chi^2_{n-1}$$
And from $Y_1$ we already had $$\overline X\sim \mathcal N\left(\mu,\frac{\sigma^2}{n}\right)$$
Answer: Actually Var$(Y_1)=\text{Var}(\overline X_n)=\sigma^2/n$ which explains the multiple of $n$.
Comment: Ah I understand, I have a mistake. Then could you explain front part?
Answer: Your confusion stems from some typographical errors. The transformation from $(x_1\ldots,x_n)$ to $(y_1,\ldots,y_n)$ that you are using is fine. But your derivation should read:
$$x_1-\bar x_n = -\sum_{i=2}^{n}(x_i-\bar x_n)=-\sum_{i=2}^{n}y_i$$
which implies
$$\sum_{i=1}^{n}(x_i-\bar x_n)^2=\biggl(-\sum_{i=2}^{n}y_i\biggr)^2+\sum_{i=2}^{n}y_i^2$$
so the joint PDF of $(Y_1,\ldots,Y_n)$ is
$$
f(y_1,...,y_n)=J\Biggl(\frac{1}{\sqrt {2\pi\sigma^2}}\Biggr)^n \exp\Biggl[-\frac{1}{2\sigma^2}\Biggl\{\Biggl(\sum_{i=2}^{n}y_i\Biggr)^2+\sum_{i=2}^{n}y_i^2+n(y_1-\mu)^2\Biggr\}\Biggr].
$$
Since the Jacobian is free of $y$, the PDF clearly factors into the form $g(y_2,..,y_n)h(y_1)$. The point of this factorization is to show that $Y_1$ is independent of $(Y_2,\ldots,Y_n)$. Since $\bar X= Y_1$ and each $X_i-\bar X$ can be reconstructed from $(Y_2,\ldots,Y_n)$, conclude that $\bar X$ is independent of $\sum_{i=1}^n(X_i-\bar X)^2$.


Can an umpire refuse a challenge as being in bad faith?

Question: There's a lot of talk about how badminton umpires must make sure there is no undue delay in play, in order to make the sport more audience-friendly. Umpires are expected to keep players on a fairly tight leash and dish out warnings when they take too long to get ready to play the next point.
When players are in trouble or tired and their game is not working for them, we frequently see them doing their best to try to circumvent this by going for long walks around the court and trying to find ways to take little breaks to cut off bad rhythms/streaks. Most of these are things the umpire can approve or deny at their discretion, and we often see the umpire telling the player they can't take a break, hurrying them to get ready.
If the umpire is already on a player for being too slow and delaying play, we sometimes see them instead take a challenge just get a break, even if the shuttle was not just out, but very obviously out.
Challenges, however, are not generally at the umpire's discretion in the same way that towelling off, changing the shuttle or wiping the court is. Looking through the BWF Umpire & Service Judge Instructions, there doesn't seem to be a lot of specific rules about handling challenges, except that players must challenge immediately. If they're too slow in calling the challenge, the umpire can refuse the call – and what constitutes 'too slow' is entirely the umpire's call. But there's nothing in there, as far as I can tell, about other venues for the umpire to refuse a challenge.
Famously, in a round-of-16 game at the 2016 China Open, umpire Cai Fengjie incorrectly denied a Japanese set-point challenge. To all appearances, he gave no reason at all for doing so, but his refusal stood, and the Japanese duo lost the set. So at least in that situation, the umpire did seem to have pretty much full autonomy over allowing or disallowing a challenge (although this was a Chinese umpire, presiding over a game with a Chinese pair playing, in a Chinese tournament… so perhaps not quite the norm).
So my question is essentially two-fold:

In general: does the umpire have the right to allow or deny a challenge call for any reason they see fit (not just the call being too slow)?
More specifically: can the umpire decide that a challenge call is just a ploy to get a break and not made in good faith (i.e., there's no way the player actually thinks the shuttle was in), and refuse to allow the challenge on this basis?
Answer: It's hard to prove a negative, but neither the Laws, nor the Instructions to Technical Officials, nor the BWF umpire manuals mention any right of the umpire to refuse a challenge.
The only limitation on challenges follows from §5.8.5 ITTO:

(...) The player must clearly say 'Challenge' to the Umpire and/or
make a clear signal by raising the hand. Any such challenge must be made
by the player immediately after the player has observed the call made by
the Umpire or Line Judge. (...)

So a player cannot wait, and then challenge.
If the player challenges timely and clearly, and has at least one challenge left, then the umpire must allow it.
Since challenges are limited to two incorrect challenges per game and players probably want to keep at least one option, delaying with challenges is rather limited in the first place. Also, unlike other ways of delay, it is upon the BWF and its technical providers to speed up challenges.
Comment: @Nij In badminton, the time available to call a challenge is indeed up to the umpire, but it is much shorter than your description makes it seem for cricket – note the word "immediately" in the quote. Usually umpires will refuse a call if the player hasn't called it within about two to three seconds, so a system like the one in cricket isn't really feasible.
Comment: Yeah, it's mostly seen later on in sets when they're clearly worried about getting carded if they take more walks around the court but still need a bit of a break. It's almost a shame the umpire seemingly can't refuse frivolous challenges. But it's interesting that the ITTO specifies "immediately after _the player has observed the call made_", since I've seen multiple instances of challenges being denied for being too late because the players _had not seen_ that a call had been changed, and thought they'd won the point. So really those challenges should have been allowed as well.
Comment: From this description, it sounds like there is far too much subjectivity in how challenges work. Compare cricket's UDRS for example, the umpire must make a clear signal only after reaching a decision, not changing it without review, and the players then have a defined time in which to decide on using a challenge, and will be told when that time is almost run out in case they are still thinking.


Solos on the 12 bar blues in A

Question: When I'm soloing over the 12 bar blues in A (A7 D9 E9) I only use notes from the minor pentatonic scale in A.
I know it should be possible to use the major pentatonic and even other scales. But I have no idea what scales and when they are appropriate to use.
All the videos and lessons I find never provide any answer to these questions.
Is there any reference material or guidelines that I can follow to spice up my licks?
Answer: There's one major concept missing from this discussion: What are your ears hearing you play over this blues progression? When you play/practice this blues are you able to hear a melody or theme that you would like to be able to play from your instrument? If not, who do you listen to that would play what you want to hear? Understanding scales and their relationship to the chords is very important but if you focus on the mechanics too much you forget that what you are trying to do is play music.
As far as the mechanics are concerned, assuming you understand basic music theory, you should be able to analyze a chord progression and determine which scale fits each chord. That having been said, your original question was about playing over a blues progression so let's assume you have analyzed the 12 bar blues. Notice the I7 chord (A7 in your example) is really a V7 chord meaning the actual scale it was derived from (again using A7 for this example) is the D Major (ionian mode) scale. A7 is the diatonic V7 chord in D major and therefore would use the D major scale as it's chord scale. We know that the mode of the V7 chord is Mixolydian and if you understand modes and their structure you will know that the D Major / Ionian mode is the same set of 7 notes as the A Mixolydian mode.
In your blues, the next chord after the I7 is the IV7 chord (in the A blues the IV7 chord is D7). Again, if you know that D7 is borrowed from the key of G Major (D7 is the V chord in G, G Major/ Ionian mode so the D Mixolydian mode would be the appropriate chord scale in this case. Your blues then goes back to the I7 chord for 2 bars, the IV7 chord gets 2 bars, then the I7 chord gets 2 more bars.
We are now approaching the last 4 bars of a 12 bar blues, the critical point! Some call it a turn-around, but it takes you to the V7 chord which is E7. E is the V chord in A major therefore the A major scale/Ionian mode and E Mixolydian would be the scale to use. You mention using a pentatonic scale so you should know that in most configurations of the pentatonic scale the 5 notes fall within the Mixolydian mode as well.
Notice that this whole structure is based on a V7 to I resolution. All V7 chords want to resolve to the I major 7 chord by nature and if you construct a melody that targets this resolution somehow, you might actually be playing music. Listen to Stevie Ray Vaughn or Buddy Guy for guitar blues. Charlie Parker and Sonny Rollins use the same 12 bar blues structure but in a bebop style. Doesn't matter. 12 bar blues is 12 bar blues whether it swings or rocks. After all, there are only 12 notes, what you do with them will define your relationship with music.
Comment: "After all, there are only 12 notes" – I rather disagree. Blues guitar uses microtonality all over the place.
Comment: I like this idea of mixolydian of... But don't forget the other blues notes that are used - very often!! b3 and b5 don't feature in mixolydian of..., but they do feature  - lots - in blues.  +1.
Comment: When you are using the mixolydian mode,  notes not included in the mode are generally passing tones or approach notes. The b3 is not part of the mixolydian mode by definition but is often used. This is an example of modal interchange where you are borrowing from another mode relative to the key you are playing in. Theoretically you can play all 12 notes in the chromatic scale in any situation whether it be C major or Gb diminished. There are no rules saying it's illegal but you have to consider whether or not you like the way it sounds.
Answer: Use the notes from A pent min. and A pent maj. while the progression is on A. You may even use the min. and maj. blues scale notes - one extra for each. These obviously work over D and E (you already use them!) but better to use the pents or blues of D over the D bars and E over the E bars. We have discussed similar in other answers. Basically there are about 3 notes which don't theoretically fit , (left from the 12 chromatics), but they still get used, mainly as passing notes.
Answer: As you've already found out, the A minor pentatonic scale - or, if you add the flat 5 = Eb, the A blues scale - fits all three chords. But your solos can indeed be made more interesting if you add more notes. You can use chord tones to spice up your solo and to incorporate the changes into your melodies.
Below are the chord tones that are not already part of the A blues scale and that can be added when the corresponding chord is played:
A7: C#
 D7: F#
 E7: B, G#
Of course, all chords can be played as ninth chords, which gives you even more options: B for A9, and F# for E9 (the E of D9 is already part of the original A blues scale).
This is a first step which will help you to make your solos more interesting, and which will make your melodies reflect the underlying progression.
One problem I've noticed with my students is that when using chord tones, you always need to know/hear where exactly in the progression you are. E.g., the note C# can be played as a target note over the A7 chord, but not (as a target note) over D7, because then it would clash with the 7th of the chord (the C). (By the way, note that any note is fine as a passing tone). This means that now you need to listen even more carefully and you need to be fully aware of the changes. This is a very good thing anyway, because it will result in more musical solos. However, since exclusively using the blues scale doesn't force you to exactly hear the changes (because all notes fit more or less all the time), many beginners are not very much aware of the changes.
You can use many more extra notes and scales, but I think the above is a very good starting point. Using the major pentatonic scale results in almost the same. Also here you can't use one scale over all three chords, for the same reasons as above. Over A7 you can use the A major pentatonic scale, but it contains the C#, which, as mentioned above, will clash with the note C of the D7 chord. One way to use the major pentatonic scale over a blues is to switch scales for every chord, i.e. use A major pent. over A7, D major pent. over D7, and E major pent. over E7. But this is just one option and not the only possibility.
Only if you feel comfortable with all of the above, you might want to try other things; they will generally sound more jazzy. One option would be the diminished scale (see this answer), or any other scale that can be used over dominant seventh chords.
To hear some weird sounds over the blues, you can check out this instructive example by guitarist Oz Noy. He demonstrates different scales such as the standard minor and major pentatonic scales and the mixolydian scale, but also more oddly sounding ones like the whole tone, the diminished and the altered scale.
Comment: Lots of folk seem to call the minor blues notes 'the blues scale', but the major blues notes, as you alluded, add some more options. Using the notes from both, over A, gives 9 note choices! There are only 3 'wrong' notes, which can still get used as passing notes
Comment: Out of time! Using the two pents only, gives 8 notes to choose from, losing the Eb (b5).
Comment: @Tim: Yes, if you combine the major and minor pentatonic scales you basically end up with the 7 notes of the mixolydian scale plus the b3, which I'd call a blue note.
Answer: Two good references:
Blues Scales: Essential Tools for Jazz Improvisation. by Dan Greenblatt
Tony Monaco's online "Blues" video lessons.
A very important thing for you to realize is: pick any scale (pentatonic or otherwise, blues or otherwise). As long as you create a logical, meaningful solo in that scale, it will work, and the scale doesn't have to exactly fit the key you're in. It doesn't need to match the background chords. It just needs to stand as a good solo on its own.
If you listen to some organ work by Jimmy Smith or Jimmy McDuff, sometimes their solos are pretty simple blues stuff. But, they will choose a different blues scale. So on the first pass, they might do a F minor blues. But on the second pass they might do an F major blues, even though the chords haven't changed. They'll pick one and stick with it, so that the solo sounds logical.

Solos on the 12 bar blues in A


Ways to implement auto voting feature on a content type?

Question: I have a content type called "articles".  A user adds a new article and other users on the site can vote for the article.  I am using the Drupal default poll module.  In order for this to work, I have to do the below steps:

Create new poll.
Link submitted article to newly created poll.
Publish article to open it up for voting.

I am trying to eliminate step 1 and step 2.  In other words, the poll would immediately be registered into the newly submitted article.  Aside from writing my own custom module, is there another way?
Answer: The Rate might be useful for what you are trying to accomplish.

This module provides flexible voting widgets for nodes and comments.
Administrators can add multiple widgets. By default, there are 8 widget types to choose from:

Thumbs up
Thumbs up / down
Number up / down
Fivestar
Emotion (this makes me mad, angry...)
Yes/no
Slider (Drupal 6 only)
Custom
Comment: To Elaborate on aerozeppelin's answer. You can use the Rate module to attach a voting widget to your content type. The rate module has a nice integration with views, so you can create a page view that pulls all of the articles for voting onto one page and allows user to vote.
Comment: Excellent!  This is exactly what I was looking for! Thank you, aerozeppelin and nicoz


executar lista de arquivos python com script shell

Question: Meu código:
#!/bin/bash

scriptspy =(
    '/scs/sp1.py',
    '/scs/sp2.py',
    '/scs/sp3.py',
    '/scs/sp4.py',
    '/scs/spweb.py',
    '/scs/sp11.py',
    '/scs/spservice.py',
    ....
)

for i in scriptspy;
  do python3.7 $i;
done

Meu objetivo é executar uma lista de arquivos que possuem python no diretório e não precisam executar um por um individualmente com python3.7 sp1.py
Obs: Não posso executar todos os arquivos .py no diretório, preciso passar uma lista para o loop com o nome deles, ou uma lista que não podem ser executados, tenho ao todo 28 quero executar e 3 que não quero executar.
Answer: Seu código estava quase certo.
#!/bin/bash

scriptspy=(
    '/scs/sp1.py',
    '/scs/sp2.py',
    '/scs/sp3.py',
    '/scs/sp4.py',
    '/scs/spweb.py',
    '/scs/sp11.py',
    '/scs/spservice.py'
)

for i in ${scriptspy[*]};
  do python3.7 $i;
done


is this the correct way to rewrite calculus limit?

Question: Can
$$\lim_{x \to 0} \frac{x^3-7x}{x^3}$$
be rewritten as
$$\lim_{x \to 0} \frac{x^3(1-7x^{-2})}{x^3}?$$
These two seem to give different answers. Please help me I'm really confused :S
Answer: Your rewrite is correct. The limit of the expression, in either form, does not exist.  Alternately, if you allow $\infty$ or $-\infty$ as possible answers to a limit question, the limit is $-\infty$.
A direct way to calculate the limit is to note that if $x\ne 0$, then
$$\frac{x^3-7x}{x^3}=1-\frac{7}{x^2}.$$
Comment: Maybe it is worth to mention something related to "punctured neighborhood" in a more pedagogical manner, to explain why $$\mathop {\lim }\limits_{x \to 0} \frac{{{x^3} - 7x}}{{{x^3}}} = \mathop {\lim }\limits_{x \to 0} \left( {1 - \frac{7}{{{x^2}}}} \right)?$$
Comment: @PeterTamaroff: Thanks for the suggestion, I added something along the lines you mentioned, but without using the phrase "punctured neighborhood."
Comment: Yes. Using the phrase "punctured neighborhood" wouldn't really help.
Comment: @PeterTamaroff thanks for the term "punctured neighborhood", it gave me something to look up and "try" to understand.
Answer: They are indeed the same:
$$\lim_{x\to 0}\frac{x^3-7x}{x^3}=\lim_{x\to 0}\frac{x^3(1-7x^{-2})}{x^3}=\lim_{x\to 0}\left(1-\frac7{x^2}\right)\;,$$ which does not exist. However, it fails to exist in a relatively nice way: as $x\to 0$, the expression in parentheses gets increasingly negative without any bound, so we often write
$$\lim_{x\to 0}\left(1-\frac7{x^2}\right)=-\infty\;.$$
Comment: Could you drop by the chat?
Answer: What you did is fine.  The next step is to cancel the $x^3$ terms to get
$$\lim_{x \to 0} \frac{x^3 - 7x}{x^3} = \lim_{x \to 0} \frac{x^3 (1 - 7x^{-2})}{x^3} = \lim_{x \to 0} 1 - 7x^{-2}$$
Now, we have a polynomial over a polynomial and we get division by 0 but the numerator of the fraction is not 0.  If it were a left or right hand limit, this would tell us the answer is $+\infty$ or $-\infty$ and we would only need to figure out which one it is.  If it's a limit (not right or left hand), there is the possibility that the left and right hand limits differ (one is $+\infty$ and the other is $-\infty$) so the limit could not exist.  But, as $x \to 0$, the whole thing $1 - 7x^{-2}$ is going to be negative no matter if you approach from the left or right.  Therefore, the left and right hand limits are both $-\infty$, so that the limit itself is $-\infty$.


React.JS, pass data between components

Question: very new to react. you can say I have not yet started to think like React.
here is the problem:

    <DropDown>    </DropDown>
    <Panel>    </Panel>

In the dropdown, I select a value. Store it in state, as something as , currentLocation.
Then I go to Panel, hit a button, and I want to open a modal. When i open a modal, I need to pass the currentLocation to that model.
I can pass in arbitrary value to modal, but I cannot figure out a way to get the currently selected item from DropDown.
How do I get the value of the currently selected item to the Panel?
Am I even making sense?
Answer: When you call the setState in the dropdown that will force an update of the page.
Then if you call this.state in your component you should have the value you need there.
You should go over the basic tutorials to grasp the react basics.
But it goes like this:
     getInitialState: function() {
        return {
          myVar: ''
          //set your variables here
          //getInitialState will get called before the component gets mounted
        };
      },
     myCustomFunction: function(newVariable) {
        this.setState({
          myVar: newVariable
        });
      },
     render: function() {
        return (

      <input
        onChange={this.myCustomFunction}
        />
      <MyCustomComponent myProp={this.state.myVar}/>
           //everytime the state changes MyCustomComponent will have that new value as a prop

        );
      }

Comment: thanks, I tried something like this, seems to have solve the problem. i don't have enough Karma to upvote you.
Answer: There is a lot of ambiguity in your question but I'll try for the simplest case.
You have a Panel component and a Dropdown component.
You want to the Panel to have access to a value that was set when the Dropdown was used.
Solution: When the Dropdown is actuated, it creates an Action that Stores the selected value.
When the modal button in the Panel is actuated, it creates an Action that requires the DropDownStore. Then it decides what to do based on that value.
The pattern I am loosely describing is known Facebook's Flux architecture which is basically just a more specific application architecture for React applications similar to pub/sub or an event bus.
Comment: yes, that is exactly how I want. You described it perfectly. to be honest I was looking for some code sample? or a pointer to where I can find it. what does "requires the dropdownstore" means?

React.JS, pass data between components


Is $d^2(x,C)$ a convex function?

Question: Let $d(.,C):\mathbb{R}^n \rightarrow \mathbb R$ defined as $d(x,C)=\inf_{c \in C}||x-c||$ where $C$ is a nonempty closed convex subset of $\mathbb{R}^n$. We know that $d(.,C)$ is a convex function.
Is the function $d^2(.,C)$ convex?
I computed that function with $C$ is $R_+$ and $[0,1]$, then $d^2(.,C)$ is convex.
I also got the relation $d^2(x,C)=||x-P(x,C)||^2$ where $P(x,C)$ is projection mapping.
I guess it is convex. Do you have any idea?
Answer: It's convex. For instance, its gradient is $x-P_Cx$, which is (firmly) nonexpansive and thus monotone. Or, you can argue that the function is an infimal convolution of the norm squared and the indicator function of $C$. This should be in any good book on Convex Analysis, starting with the trusted one by Rockafellar.
Comment: Could you tell me any relation between convexity and monotone?
Comment: A function is convex iff its gradient is monotone.
Answer: It is easy to see that $d(x,C)$ is convex. Since $x \to x^{2}$ is an increasing convex function on $[0,\infty)$ the composition $d^{2}(x,C)$ is convex.

Is $d^2(x,C)$ a convex function?


Prestashop background bulk product import

Question: I'm trying to import efficiently 15K+ products periodically in a Prestashop e-commerce parsing an ASCII file (3.5MB) and using only Prestashop's API.
All run under docker, with official images from docker hub.
If a product with the same reference field does not exists I'll have to insert a new product, if it's present to update it.
I developed a module that makes this via a click un a custom admin tab, and it works, but the whole system freezes till the process is finished or terminated: (almost 77 minutes).
I also tried to split the (not so) big file in chunks of 500, 100, 50 but the time for the processing decreases linearly, it doesn't help very much:

153 avg seconds for 500 elements
31 avg seconds for 100 elements
15 avg seconds for 50 elements

I could surely configure a cron every 90 seconds to process 50 elements, and complete the whole import in 7-8 nightly hours, but it seems a very bad compromise: 15 seconds offline every 90.
I cannot use pthreads since this will be a production web server.
I tried to tune Apache increasing memory_limit, max_input_vars, max_execution_time but without any differences: DB using from 450MB to 550MB of RAM and server almost identical.

Linux #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64
Versione software del server: Apache/2.4.10 (Debian)
Versione di PHP: 5.6.35
memory_limit=2048M
max_input_vars=1000000
max_execution_time=600000
MySQL: 5.6.40

Am I facing the problem in the wrong way, or Prestashop's API are not performant and made for bulk (and performant) product import?
public function batchImportAllProductsFromFile($productsToBeInserted){
    foreach ($productsToBeInserted as $key => $customProduct ) {

        $productIDs = $this->getProductIDsByReference($customProduct->MS_CODMAG);
        if (sizeof($productIDs) == 0) {
            $product = new Product();
        } else if (sizeof($productIDs) == 1) {
            $product = new Product($productIDs[0]);
        } else {
            continue;
        }

        $product->reference = $customProduct->MS_CODMAG;
        $product->name = trim($customProduct->MS_DESCRIZIONE);
        $product->price = $customProduct->MS_PREZZO_1;
        $product->out_of_stock = ($customProduct ->MS_ESAURITO === "S" ? true : false);

        $category = null;

        $msGruppoConverted = $this->buildSubGroupCode($customProduct->MS_GRUPPO, $customProduct->MS_SGRUPPO);

        if ($customProduct->MS_GRUPPO !== 0 && $msGruppoConverted !== 0) {
            $product->id_category = [$customProduct->MS_GRUPPO, $msGruppoConverted];
        } else if ($customProduct->MS_GRUPPO === 0 && $msGruppoConverted !== 0) {
            $product->id_category = [$msGruppoConverted];
        } else if ($customProduct ->MS_GRUPPO !== 0 && $msGruppoConverted === 0) {
            $product->id_category = [$customProduct->MS_GRUPPO];
        }
        try {
            if (sizeof($productIDs) == 0) {
                if ($product->add()) {
                    $product->updateCategories($product->category);
                    $product->addFeatureProductImport($product->id, 1, $customProduct->MS_FAM);
                    //StockAvailable::setQuantity((int)$product->id, 0, $product->quantity, Context::getContext()->shop->id);
                }
            } else if (sizeof($productIDs) == 1) {
                if ($product->update()) {
                    $product->updateCategories($product->category);
                    $alreadySavedFeatures = $product->getFeaturesStatic($productIDs[0]);
                    if (sizeof($alreadySavedFeatures) != 1 || $alreadySavedFeatures[0] != $customProduct->MS_FAM) {
                        $product->deleteProductFeatures();
                        $product->addFeatureProductImport($product->id, 1, $customProduct->MS_FAM);
                    }
                }
            }
        } catch (Exception $e) {
            var_dump("Errore: ", $e, $product);
        }
    }
}

EDIT 22/10/2018:
Upgrading to PHP7.2 and using MariaDB 10.3.10 brought me no changes: timing was still the same.
What did brought benefits was to mount the FS (EXT4) where the DB stores info with option barrers=0 in /etc/fstab: performances improved from 153 to 35 seconds for 500 elements, resulting in circa 18mins totally (was 77).
The problem that remains open is why the system became unresponsive while importing.
Comment: Have you tried switching your SQL Engine from MyISAM (I suppose) to InnoDB to avoid any table lock issues? Also switching to PHP 7+ and MariaDB might drastically improve your performances.
Comment: The DB and all tables (except 2 for CRON(s)) were already InnoDB, but thanks for the hint regarding PHP 7+ and MariaDB: I'm going to setup all with PHP7.2 and MariaDB 10.3.10 and give it a try
Answer: You should check that your products needs an update before updating it.
Here is what I do (Highly simplified) when importing entities on Prestashop:
<?php

class myProductImporter {
    protected $products;
    protected $products_checksum;

    public function __construct($products) {
        // Your products from the csv file
        $this->products = $products;
        // Here you get an associative array of products references and checksums
        // ex: array('REF01158' => '489f9ze4f4ze9f49ze8', 'REF15616' => '48949844561233132')
        $this->products_checksum = getProductsChecksum();
    }

    public function run() {
        foreach ($this->products as $product) {
            // If the product ref is present in my checksum list, then its an update
            if (isset($this->products_checksum[$product['reference']])) {
                // If the checksum is different, the product needs an update
                if ($this->products_checksum[$product['reference']] != $this->getChecksum($product)) {
                    $this->updateProduct($product);
                }
            // Else it's a new product
            } else {
                $this->addProduct($product);
            }
        }
    }

    protected function updateProduct($product) {
        $PSProduct = getProductByReferebce($product['reference']);
        // Update your product and save its new checksum
    }

    protected function addProduct($product) {
        $PSProduct = new Product();
        // Create the product and save its checksum
    }

    protected function getChecksum($product) {
        // Create a string containing all your product properties
        $checksum = $product['reference'];
        $checksum .= $product['name'];
        $checksum .= $product['description'];
        $checksum .= $product['id_category'];
        return md5($checksum);
    }
}

When there's no modification, your import will finish instantly because there's no queries being processed.
Concerning the freezes on your installation it seems to be a Docker problem and not linked to Prestashop.
As mentioned by @bruno-leveque you should consider upgrading to PHP7.
Comment: Thanks for the advice on the checksum, this will surely be useful after the first import: it was the missing piece of my alghorytm.
I'm going to setup everything with PHP7.2 and MariaDB 10.3.10, I'll update the post with results
Comment: Can you tell me to which file is this code meant to be added to?
Answer: What really solved my problem of Prestashop being stuck while importing products is to move my code from being inside a ModuleAdminController to a WebserviceSpecificManagementInterface: this way the import goes in background without saturing the system.


Red card for clear goal scoring opportunity

Question: I was watching the two big games in the Champions League this week and both of the games were spoiled by the red cards (the card in the City-Barca game was a clear red as it was a clear clear goal scoring opportunity whereas the Arsenal-Bayern game was not so clear cut).
It got me thinking;
1: when was the rule around clear goal scoring opportunities introduced and
2: was there a case that made them introduce it? I'm thinking something like when they clamped down on tackles from behind because it was this kind of tackle that forced Marco Van Basten into early retirement or Lampard's goal in the 2010 World Cup which got a debate on goal line technology going.
Comment: I cannot give a time frame on when it was introduced but the case is simple enough, if you are the last defender and you stop the attacker, or the ball, with an action that would normally be a foul (a tackle or handball), that's a red card for unsportsmanlike conduct. The case is to hinder teams from committing desperate fouls to prevent an almost certain goal. Specific cases may trigger commitment to a rule change but the arguments have certainly been out there for a while in most rule changes.
Comment: I understand why the rule is in place I was wondering was there an incident that caused Fifa to introduce the rule
Answer: This rule was first issued to the Football League referees prior to the 1982-83 season.
FIFA instructed its referees to send off for this kind of foul prior to the 1990 World Cup.
IFAB(International Football Association Board) incroporated this law in 1997.

The concept gained notoriety in association football after an infamous incident in the 1980 FA Cup Final when Willie Young of Arsenal committed a deliberate foul on Paul Allen of West Ham, when Allen had a clear run at goal. As the laws of the game stood, the referee (George Courtney) could only award West Ham a free kick, which he did. This provoked a national debate on deliberate fouls that denied opponents the chance to score a goal.

Source: WikiPedia
Video: YouTube