systemd script for starting my app server as soon as it is down or system is rebooted

Question: I have a upstart configuration file as shown below which works fine in Ubuntu 14:
#/etc/init/data_server.conf
#sudo start data_server
#sudo stop data_server
#sudo status data_server

start on runlevel [2345]
stop on runlevel [016]

chdir /opt/hold/data_server
respawn

post-start script
    echo "data server started at `date +"%F %T"` on `hostname -f`" | mailx -r "firstname.lastname@example.com" -s "data server Started" "email@example.com"
end script

post-stop script
  sleep 30
end script

limit core unlimited unlimited
limit nofile 100000 100000
setuid goldy
exec ./data_server --init_file=../config/tree.init --port=8080 --dir=/data/hold/ --max_sec=2400 --max_mb=100 --active=5

Now we are moving to Ubuntu 16 so we can't use upstart and looks like we need to use systemd here. What are the changes I need to do to write script in systemd?
I have to make sure whenever system is rebooted or app is killed it should start my systemd script automatically which in turn start my data server.
Comment: Please try writing the unit yourself, and ask for help with anything where you run into difficulty. This is not a service for doing your work for you. See the [documentation](https://www.freedesktop.org/software/systemd/man/systemd.service.html).
Answer: Your upstart script will change a lot. You shouldn't need to manually specify so many directives, so it will also be a lot shorter.
Making a process always start on boot and restart when it crashes is a simply one line:
Restart=always

I don't know how many custom changes you need to keep, but at the very minimum, it looks like you'd need to keep the chdir and exec commands.
Here is an example of a basic systemd script for your app:
[Unit]
Description=My-service

[Service]
Type=simple
WorkingDirectory=/opt/hold/data_server
ExecStart=/path/to/data_server --init_file=../config/tree.init --port=8080 --dir=/data/hold/ --max_sec=2400 --max_mb=100 --active=5
Restart=always

[Install]
WantedBy=multi-user.target

I found this to be a handy resource when writing custom unit files: https://www.freedesktop.org/software/systemd/man/systemd.unit.html
Comment: @BoomShadow: AFAIK the `ExecStart` has to start with an absolute path to an executable.
Comment: @Thomas Ah yes, you're right. I forgot about that. Thanks. I updated my answer with generic example.
Comment: got it make sense now.. I am gonna check that link.. Also by any chance do you know what I can do for these stuff `limit core unlimited unlimited`, `limit nofile 100000 100000` and `setuid goldy`.
Comment: I don't know much about upstart, so I couldn't advise all the translations. However, if you search that page that I linked, it does talk about setting the UID. It maybe also have info for those other custom directives.


Resources for Probability

Question: I'm currently studying joint distribution functions (discrete and continuous) with their expectation, variance and covariance etc. I would like to get as much practice as possible. Can you suggest books/sources for practising such problems?
Comment: Does this answer your question? [Path to mathematical statistics without analysis background: ideal textbook for self study](https://stats.stackexchange.com/questions/160261/path-to-mathematical-statistics-without-analysis-background-ideal-textbook-for)
Answer: Answers to such questions always involve opinions. So here are a couple of opinions.
(1) Wackerly, Mendenhall, Scheaffer: Math. Stat. with Appl is available in
several editions. (Probability parts have been published in a shorter book under Schaeffer's name.) Separate student guides, containing skeletal solutions and answers to some questions, are available for some editions. For your purposes, an older edition at a cheaper price would be as good as the latest edition.
I have taught from several editions. The material of interest to you requires multivariate calculus. Worked examples are well chosen. The approach is transparent, and often too wordy to be called elegant--perhaps what you need for self study. Later editions mention the use of R, but do not use computer technology meaningfully.
(2) Another possibility is Bain, Englehardt: Intro. to Probability and Math. Stat., available in paperback. The approach is a little more oriented toward practical applications than in Mendenhall et al., but sufficiently rigorous for an introduction. Terse answers to nearly half of the problems in the back of the book. No use of computer technology for solutions.

Resources for Probability


Update CRM 2011 Access Right Programmatically

Question: I would like to know if there is some way to update an Access Right in CRM 2011.
Let's say I have an Access Right named Marketing Manager which has user level privilege to create Account, I'd like to change this privilege to Organization level programmatically.
I'm waiting hopefully,
Thanks in advance
Answer: Please have a look at the code below. However it actually copy Role Privileges from existing Role to new Role. But I think it can give you an idea.
<code>Guid existingRoleId = new Guid("C85F0FFF-4C80-E211-A877-1CC1DE79B4CA");
Guid newRoleId = new Guid("B6690FFF-4C80-E211-A877-1CC1DE79B4CA");

// Step 2
RetrieveRolePrivilegesRoleRequest getPrivilagesRequest =
                 new RetrieveRolePrivilegesRoleRequest();

getPrivilagesRequest.RoleId = existingRoleId;
RetrieveRolePrivilegesRoleResponse privilagesResponse =
         (RetrieveRolePrivilegesRoleResponse)service.Execute(getPrivilagesRequest);

if (privilagesResponse != null && privilagesResponse.RolePrivileges != null)
{
    // Step 3
    AddPrivilegesRoleRequest addPrivilagesRequest = new AddPrivilegesRoleRequest();
    addPrivilagesRequest.Privileges = privilagesResponse.RolePrivileges;
    addPrivilagesRequest.RoleId = newRoleId;
    AddPrivilegesRoleResponse addPrivilagesResponse =
            (AddPrivilegesRoleResponse)service.Execute(addPrivilagesRequest);
}
</code>
Ref: Copy a security role programmatically – CRM 2011
Link 2
Comment: Firstly, thank you for the quick answer.Secondly I do appreciate your attempt to give me an idea, but I really didn't get the point, in this code he just copied the access without change anything.
Comment: Honestly I don't know if we can update privileges or not. But I think you can use workaround if there is no other option. Create a Role called "New Role" with Organization level privileges. When you want to update a role privileges. Copy the privileges from "New Role" to the role you want to update. I have also updated the answer, please check the link 2 as well. Regards
Comment: Unfortunately none of these tips helped me. Anyway, I'm guessing it's impossible, thanks


Indent lines matching a word, then print them with a replacement

Question: I am trying to replace all occurrences of open with not open.  I want to print the original text with modified lines indented and preserved in original form followed by the modified form.
This command was not enough:
sed 's/open/not open/' file.txt

Sample input:
this is the first line of text
this text is very open
and this is the next line of text

Sample output:
this is the first line of text
    this text is very open
    this text is very not open
and this is the next line of text

Comment: Please, try to be more clear. Edit the question to give a sample input and expected output.
Comment: OK, I see the input. Where's the expected output?
Comment: `sed '/open/not open/` would replace `not open` by `not not open`
Comment: And it would replace "very open" with "very not open", not "not very open"...
Answer: Given this input.txt:

this is the first line of text
this text is very open
and this is the next line of text

The following code
sed '/not open/ b; /open/ { s/^/    /; p; s/open/not open/ }' input.txt

outputs:

this is the first line of text
    this text is very open
    this text is very not open
and this is the next line of text

The first clause merely skips any line matching not open (which means a line reading The door is not open. Do open it. has no replacement).  The b command skips to the end of the script.  You can omit this part if you don't want this logic.
The second clause keys on any other line's mention of open and performs three actions on it: indent by four spaces, print immediately, then make the replacement.  Finally, sed automatically prints the line (modified or not, even if already printed) at the end of its evaluation.
Comment: First, why do you answer a *question* which obviously doesn't fit the quality rules? Obviously because it isn't a question at all. Further note that your script makes `not not open` out of `not open`
Comment: I'm trying to be helpful. Rather than criticize the poster's English skills, I've guessed at the question and tried to give a useful and educational answer.
Comment: I believe that you had good faith, but you see it doesn't help much. Better would be to request more information and wait unless it will be provided (or not).. Btw, I would not criticize poor English because my English is also mostly incorrect. But IMHO it is not the English which makes this question a quality issue.
Comment: @AdamKatz thank you sir, all i want is to replace the word and print the original text and print it after i have replace the word
Comment: @odishoeivazi: If this is what you're looking for, please accept it as an answer. Otherwise, please clarify your question.
Comment: @hek2mgl I have addressed your `not not open` issue (as thoroughly as POSIX `sed` can). I also adjusted the question now that it has been clarified.
Answer: This might work for you (GNU sed):
 sed '/\<open\>/!b;s/^/\t/p;s/\<open\>/not &/g' file

Any line that does not contain the word open print as normal. Otherwise, insert a tab at the start of the line and print the original line, then replace all open with not open and print.


Realizar um select com count e somar

Question: Estou precisando realizar um select, que tragar a quantidade total de alunos matriculados por turma e empresa, só que quando for empresa ('X' e 'Y') somar a quantidade total deles, ('D' e 'F') também somar as quantidade total deles, for empresa. 'P' trazer quantidade total de alunos somente desta empresa e as outras empresa trazer o total das somas delas.
Realizo este select, através do período e á turma, que pode ser uma ou mais turmas. Porem não sei como trazer a somar total das empresas ('X' e 'Y')  somar a quantidade total deles, ('D' e 'F').
SELECT COUNT(*) AS 'quant_total', empresa, turma FROM `alunos`
WHERE data_matricula >= '2014-01-01' AND data_matricula <= '2014-02-01'
AND substr(turma, 1,4) IN ('AAAA') AND situacao = 'AT'
GROUP BY empresa, turma

Resultado:
quant_total       |empresa       |turma

1                 | X            | AAA
4                 | X            | BBB
4                 | X            | CCC
3                 | X            | DDD
1                 | F            | AAA
16                | D            | AAA
33                | D            | HHH
28                | D            | XXX
18                | D            | DDD
12                | P            | AAA
1                 | P            | HHH
11                | P            | BBB
14                | P            | CCC
23                | P            | ZZZ
1                 | P            | KKK
3                 | Y            | AAA
5                 | Y            | BBB
4                 | Y            | CCC
5                 | Y            | DDD
1                 | O            | BBB

Comment: Quando fazer um questionamento, nunca abrevie as palavras, aqui o entendimento é primordial.
Comment: Não manjo muito de SELECT, então um caminho ao meu ver seria no WHILE do PHP realizar as validações de turma, porém ainda não consegui imaginar como se surgir algo te dou um toque.
Answer: segue uma solução usando CASE
SELECT COUNT(*) AS 'quant_total',
            (case when empresa in ('X','Y')
                  then 'GRUPO XY' else empresa end) empresa, turma
FROM `alunos`
    WHERE data_matricula >= '2014-01-01' AND data_matricula <= '2014-02-01'
    AND substr(turma, 1,4) IN ('AAAA') AND situacao = 'AT'
    GROUP BY (case when empresa in ('X','Y')
                   then 'GRUPO XY' else empresa end), turma

Uma solução melhor seria criar uma coluna GRUPO na tabela de EMPRESA , poderia ser usar este campo para agrupar empresas
Answer: Não sei se entendi bem o que você precisa. Talvez a consulta a seguir ajude. Utilizei dois comandos SELECT, um para calcular a soma de alunos por empresa e turma e outro para calcular o total para as empresas 'D' e 'F'.
SELECT empresa, turma, count(*) AS 'quant_total'
FROM alunos
GROUP BY empresa, turma
UNION
SELECT empresa, 'Total' AS turma, count(*) AS 'quant_total'
FROM alunos
WHERE empresa IN ('D', 'F')
GROUP BY empresa;

O resultado é
+---------+-------+-------------+
| empresa | turma | quant_total |
+---------+-------+-------------+
| D       | AAA   |           3 |
| D       | BBB   |           3 |
| D       | CCC   |           4 |
| F       | AAA   |           3 |
| F       | BBB   |           1 |
| F       | CCC   |           2 |
| X       | AAA   |           3 |
| X       | BBB   |           3 |
| X       | CCC   |           2 |
| X       | DDD   |           1 |
| Y       | AAA   |           2 |
| Y       | BBB   |           2 |
| Y       | CCC   |           3 |
| D       | Total |          10 |
| F       | Total |           6 |
+---------+-------+-------------+

Sendo que a tabela alunos que utilizei para testar é
+----+-----------+---------+-------+
| id | nome      | empresa | turma |
+----+-----------+---------+-------+
|  1 | Helena    | X       | AAA   |
|  2 | Davi      | X       | BBB   |
|  3 | Gabriela  | X       | CCC   |
|  4 | Laura     | X       | BBB   |
|  5 | Beatriz   | X       | AAA   |
|  6 | Enzo      | X       | CCC   |
|  7 | Valentina | X       | DDD   |
|  8 | Samuel    | X       | BBB   |
|  9 | Rafael    | X       | AAA   |
| 10 | Lucas     | Y       | CCC   |
| 11 | Gabriel   | Y       | AAA   |
| 12 | Maria     | Y       | BBB   |
| 13 | Ana       | Y       | BBB   |
| 14 | Guilherme | Y       | CCC   |
| 15 | Luiza     | Y       | AAA   |
| 16 | Heitor    | Y       | CCC   |
| 17 | Arthur    | D       | BBB   |
| 18 | Alice     | D       | BBB   |
| 19 | Bernardo  | D       | CCC   |
| 20 | Manuela   | D       | AAA   |
| 21 | Mariana   | D       | BBB   |
| 22 | Gustavo   | D       | CCC   |
| 23 | Isadora   | D       | CCC   |
| 24 | Felipe    | D       | AAA   |
| 25 | Julia     | D       | CCC   |
| 26 | Matheus   | D       | AAA   |
| 27 | Rafaela   | F       | BBB   |
| 28 | Isabella  | F       | CCC   |
| 29 | Miguel    | F       | AAA   |
| 30 | Giovanna  | F       | CCC   |
| 31 | Nicolas   | F       | AAA   |
| 32 | Pedro     | F       | AAA   |
+----+-----------+---------+-------+

Por favor, detalhe melhor a sua pergunta se as respostas aqui apresentadas não resolverem o seu problema.
Answer: Oi amigos estou refazendo minha perguntar aqui, para que compreendeste melhor.
A resposta de Motta, está quase como eu quero, porem só somou as empresas ('X','Y'), o restante do resultado que espero, mencionado abaixo não trouce, mais mesmo assim agradeço muito pela força.
A questão é, na escola existe varias turmas com alunos de diferentes empresas, dado o um determinado período e seleção de uma ou mais turmas, preciso realizar um SELECT para trazer quantidade total de alunos registrados na escola, agrupando por turma e empresa. Á logica é, se forem alunos das empresas -> ('X'  e  'Y'), empresas -> ('D' e 'F') somar a quantidade de alunos destas duas empresas, para obter um resultado único, se for alunos da empresa 'P' trazer quantidade  total de alunos somente desta empresa, e por ultimo, se forem alunos de qualquer outra empresa, somar a quantidades de alunos total destas empresas, para obter também um único resultado, tudo isto agrupado por turma e empresa.
No momento só conseguir fazer um SELECT que trás a quantidade total de alunos  agrupados por turma e empresa. Porem não sei como realizar á somar das quantidade total dos alunos das empresas ('X' e 'Y'), empresas -> ('D' e 'F'), para obter um resultado único delas.
Conseguir a Solução galera, Obrigado á todos pela ajuda.
Solução:
SELECT COUNT(*) AS 'quant_total',
(CASE WHEN empresa IN ('X','Y')
THEN 'X/Y'
WHEN empresa IN ('D','F')
THEN 'D/F'
WHEN empresa = 'P'
THEN 'EMP.P'
ELSE 'OU' END) empresa, turma
FROM alunos
WHERE data_mat >= '$dt_inicio' AND data_mat <= '$dt_fim' AND substr(turma, 1,4) IN ($turma) AND situacao = 'AT'
GROUP BY (CASE WHEN empresa IN ('X','Y')
THEN 'X/Y'
WHEN empresa IN ('D','F')
THEN 'D/F'
WHEN empresa = 'P'
THEN 'EMP.P'
ELSE 'OU' END), turma

Realizar um select com count e somar


Christoffel Symbol - what does a comma mean in the footer?

Question: I am trying to understand the expression for Scalar curvature in terms of the Christoffel symbols.
This is given on Wikipedia by
\begin{equation}
S = g^{ab}(\Gamma^c_{ab,c} - \Gamma^c_{ac,b} + \Gamma^d_{ab}\Gamma^c_{cd} - \Gamma^d_{ac}\Gamma^c_{bd})
\end{equation}
(see here for the Wikipedia post)
The thing I'm not sure about is the comma in expression $ab,c$, for example in the footer of the first Symbol. My guess would be that it means
\begin{equation}
\Gamma^c_{ab,c} = \partial_c \Gamma^c_{ab}
\end{equation}
Is that correct ?
The site on Christoffel Symbols on Wikipedia doesn't explain what the comma means so I was wondering whether somebody could help? Many thanks!
Answer: In differential geometry, a subscript with a comma often is used to denote the coordinate partial derivative relative to some fixed coordinate system.
Along the same lines, a subscript with a semi-colon is often used to denote the covariant derivative.
So, given a scalar function $f:M\to\mathbf{R}$,
$$ \nabla_a f = \partial_a f = f_{,a} = f_{;a} $$
For a tensor quantity with coordinate components $f_{abcd}$, the expression
$ f_{abcd,e}$ means the $\partial_e$ of the scalar function $f_{abcd}$. Which is different from $f_{abcd;e}$ which often means the scalar component $(\nabla f)_{eabcd}$.
Comment: that's great help, thanks also for giving this a little context!
Comment: See also http://en.wikipedia.org/wiki/Covariant_derivative#Notation


Tutorial for creating website using existing MySQL db?

Question: I'm an experienced web developer, but I'm new to Drupal.
I have an existing MySQL database and I would like to build a website around it, with one page for each record in the main table. That would mean a few thousand pages for the site right from the start.
Are there any good tutorials that explain how to do this in Drupal?
Answer: It's not too hard. Assuming Drupal 7 and a one-time import:

Create a new content type in Drupal with the fields you want
Export a CSV file of your records with a header row from your legacy database with the fields you want.
Use Feeds to import the csv file.  You will setup a mapping between the csv fields and your content types fields (from within the Drupal UI).  This is pretty straightforward and the feeds module guides you through it.
Feeds will create one node per record (line) in your csv file.

Now if you will continue to use your existing database and want to import new records as they are added, I believe Feeds has some support for this but I haven't personally used it.
Answer: You might find the Feeds Module useful. Here is the link to the tutorials.
Answer: Another approach is to wrap views around your table(s). You can have a look at the maillog module for an example. The module integrates its maillogs with views.
Once you have the integration, views gives you a very powerfull query and report generator.


Java Client Socket blocking on BufferedReader?

Question: I am using old school java.net.Socket
My client connects to server and does:
BufferedReader in =
   new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
String fromServer = in.readLine();

It hangs at in.readLine().
My server accepts a connection from the client and does:
PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
out.print("Hello client\n");

My expectation would be that as soon as out.print("Hello client\n"); is invoked, the client should stop blocking and continue on.
What are my doing wrong?
PrintWriter out = new PrintWriter(clientSocket.getOutputStream(), true);
BufferedReader in =
   new BufferedReader( new InputStreamReader(clientSocket.getInputStream()));
String fromServer = in.readLine();

Answer: I guess you are not calling out.flush(); See PrintWriter.flush
From Javadoc

autoFlush - A boolean; if true, the println, printf, or format methods will flush the output buffer

So Autoflush will not work for print you will need to call flush() mannually
Comment: Specifically, `autoFlush - A boolean; if true, the println, printf, or format methods will flush the output buffer`. This is quoted from the link that AmitD made in his post to PrintWriter.
Comment: autoflush is set PrintWriter constructor.
Answer: You are calling out.print("Hello client\n"), which isn't autoflushed, when you should be calling out.println("Hello client"), which is autoflushed.


Autosys file watcher for a particular filename on Windows

Question: I am trying to write a file watcher job in autosys that would watch out for a particular file. The file name format would be filename_ddmmyyyy.
The requirement is that the file comes at 7.15am everyday and the file watcher job starts running at 6.50am and the runs till 8am. If the file is received by then, job is successful else an alert is raised.
Now what I am trying to do is to watch out the file filename_ddmmyyyy for a particular day. e.g. if today is 22nd Feb 2013, the file name will be filename_22022013 and this is the file that I am looking for. If I use wildcards like filename_*, it would look for all possible files which I don't want.
I am not sure how to do this in Windows.
Any help would be much appreciated.
Let me know in case of questions.
Answer: You will need to use the profile job attribute to initalize variables when the job starts.  One of these variables will need to be the date pattern you are looking for (you'll need another process that outputs that dynamically).  Then once you set it to a variable in your profile script, you can refer to that variable name from within the watch_file attribute.
Answer: Create global variable as variable with date and us that variable :
example:filename_$${GV_DATE}
GV_DATE: ddmmyyyy

Answer: Pretty late to answer, but here is an answer without using global variable. You can use formatted system date variable in the file name.
File_to_watch: filename_%date:~10,4%%date:~4,2%%date:~7,2%


Angularjs how can I unit test a Service which depends on another Service with promises?

Question: How can I test a service which depends on another service. I currently get Service1Provider not found error in this implementation. How can I properly inject Service1 so I can unit test Service2? Thanks for any tips or tricks.
jsfiddle
gist
!function(ng){
'use strict';

 var module = ng.module('foo.services', []);

(function($ng, $module) {
  function Service($q) {

    return {
        bar: function(a,b,c){

            var baz = a+b+c;
            return function(d,e,f){

                var deferred = $q.defer();
                if(baz > 0){
                  deferred.resolve({result: baz + d + e + f });
                } else {
                  deferred.reject({ err: 'baz was <= 0'})
                }
                return deferred.promise;

            }
        }
    };
  }

   $module.factory("Service1", ['$q', Service]);

 })(ng, module);

  (function($ng, $module) {
    function Service(Service1) {

       function doSomething(){

        var result;
        var whatever = Service1.bar(5,6,7);

         var promise = whatever(8,9,10);
        promise.then(function(data){

            result = data.result;
            //data.result should be 45 here
        }, function(err){

        });

        return result;
    }

    return {
        bam:doSomething
    };
}

  $module.factory("Service2", ["Service1", Service]);

  })(ng, module);
}(angular);

var myApp = angular.module('myApp',['foo.services']);

Comment: Hcabnettek, I can only suggest that Service2 would need to be modified to accept a service as a parameter passed to `doSomething`, otherwise Service2's dependency on Service1 is impenetrably internal to Service2. If you are looking for an answer to what might be wrong, then `result` can't be returned directly from from `doSomething` - you need to return `promise`.
Comment: In fact, the modification needs to be slightly more extensive. The returned promise needs to take account of the chained `.then()`, whose callback should return data.result` (I think).
Answer: If you are just testing Service2 then you should try to eliminate any dependency on Service1 in the test. Your test could have something like:
module('foo.services', function($provide) {
  $provide.value('Service1', MockService1());
});

This will give the return value from the function MockService1 instead of actually using the service, whenever it is injected.
Then you have MockService1 function return a skeleton of the actual service with the same function. In your test you can then wait for the promise to be resolved by doing something like this:
bar: function(...) {
  var def = $q.defer();
  $timeout(function() {
    def.resolve('test');
  });
  return def.promise;
}

// This is in your test
bar.then( /* .... some tests */ );
// This executes the timeout and therefor the resolve
$rootScope.$digest();
$timeout.flush();

Hope this helped
Comment: This is not a bad idea, however, if the service being mocked up is huge or unknown (with a lot of unexpected errors, for example,) it's hardly going to be helpful. In my case, I'm testing with [express-cassandra](https://github.com/masumsoft/express-cassandra) and doing a valid mock up would take months of work...


ecology community sample comparisons

Question: I have 2 batches of tree species and abundance data collected for a city. The difference between them is the sampling strategy - one was random unstratified, and the other was stratified and clustered. At a glance the species mixes and relative abundances are fairly similar but i want to test this statistically. I have considered Bray curtis but this test has the assumption that the samples come from identically sized areas, which these arent - the random sample covers a much smaller area of the city.
How can i compare these populations?
thanks
Answer: Null model approaches are generally used in Ecology in these situations.  I understand the question to mean: do the two sampling strategies draw from the same underlying species-abundance distribution?
Here is what I would do:
1) Randomly rarefy the dataset with more individuals until it contains the same number of individuals as the dataset with fewer individuals.  Ultimately the entire analysis can be repeated several times, using different instances of the random rarefaction in this step.
2) Compute an appropriate dissimilarity metric between the two communities (Bray-Curtis is good, but other metrics can also be useful).
3) Compute two null distributions for the dissimilarity metric using the following procedures.  A) Randomly swapping the totals corresponding to each species between datasets.  Repeat many times, and calculate the dissimilarity metric each time.  B) Pool all of the individuals in both datasets, and then randomly re-sample them into two separate communities. Repeat many times, and calculate the dissimilarity metric each time.
4) Compare the true dissimilarity value to the null distribution, to see whether it reflects samples that are more distinct than you could expect by random sampling from a single species abundance distribution.

ecology community sample comparisons


Show that if $K$ is a normal subgroup of $H$ such that $H/K$ is abelian, then K contains all of the 3-cycles.

Question: Claim: For $n≥5$, if $H$ is a subgroup of $S_n$ which contains all of the 3-cycles, and $K$ is a normal subgroup of $H$ such that $H/K$ is abelian, then K also contains all of the 3-cycles.
Attempt: let $f:H\to H/K$. I've already shown that for a group $G$ and $G'$ the subgroup generated by $\{ghg^{-1}h^{-1} \; : \; g,h \in G \}$; if $f:G\to H$ is a homomorphism then $Im(f)$ is abelian iff $G' ⊂ ker(f)$. How do I use this to show that $H' ⊂ K$? How do I show that every 3-cycle belongs to $H'$? (i.e express an arbitrary 3-cycle $(a_1a_2a_3)$ as $ghg^{-1}h^{-1}$ for some 3-cycles $g,h$ so that $g,h \in H$ and $(a_1a_2a_3) \in $H'$.)
Answer: You have $f:H \to H/K$ defined by $f(h)=hK$ is a homomorphism (the natural projection). Since $H/K$ is abelian, the image of $f$ is abelian and thus $H' \subset ker(f)$.  But $ker(f)=K$ (because $f(h)=hK=K$ iff $h \in K$).
Now consider a 3-cycle $(a_1a_2a_3)$. Since $n \geq 5$, there are some numbers $a_4$ and $a_5$ disjoint from $a_1,a_2,a_3$. Note that if we let $x=(a_1a_2a_4),y=(a_1a_3a_5)$. Then since $H$ contains all 3-cycles $x,y \in H$. Therefore,
$$xyx^{-1}y^{-1} = (a_1a_2a_4)(a_1a_3a_5)(a_1a_4a_2)(a_1a_5a_3)=(a_1a_2a_3) \in H'$$
Now $H' \subseteq K$. So $K$ contains all 3-cycles.


Apachectl Missing from Apache HTTPD Binary

Question: I extracted the httpd...tar.gz from here, but I don't see apachectl.
[vagrant@centos48 opt]$ ls
httpd-2.2.23  httpd-2.2.23.tar.gz
[vagrant@centos48 httpd-2.2.23]$ find . -name apachectl
[vagrant@centos48 httpd-2.2.23]$

Answer: This is because you did not build the code. You downloaded the source code. So, you need to build it first to create the binaries and libraries.
Comment: So apachectl doesn't come with the binary? Why?
Comment: @Kevin: Yes, that's the source code.
Comment: @Kevin: It seems to me that you downloaded the source code not the binary. If you downloaded the binary, you should find it already there.
Comment: I downloaded httpd-2.2.23.tar.gz from the above link, and then extracted it using 'tar -zxvf {file.tar.gz}'
Comment: Doh! #Homer. I should've noticed that there wasn't a bin directory. Thanks
Answer: You have downloaded the source code. apachectl is in there, but it's only in source form. You'd have to compile that and install the resulting binary.
But, unless you have a special need, don't. Instead,
yum install httpd

That will get you the compiled package, properly configured with a basic setup, and with all of the tools, including /usr/sbin/apachectl.
The version of Apache is slightly older than the one you've downloaded, but it includes backported security patches, and we're way past the era when new minor releases of Apache included very exciting new features, so it's unlikely that you really need it.


How to match sample points to a probabilistic model?

Question: This might be a trivial question
I have a probabilistic model, say a Gaussian mixture model of known parameters, and with that I have a set of defined sample points.
I would like to know how likely it is that the sample points were generated from (i.e. that they match) that model.
What would be the theoretical solution to this problem?
The issue is that simply using the pdf of a GMM does not seem to make sense because:

It is a continuous distribution while the points are infinitesimal samples
If all samples lie exactly at the mean of the sharpest Gaussian component, their probability according to the pdf would be higher than having them spread among all the Gaussian components according to the components' weights, although the latter case would match the underlying model better.
Answer: If I understand your question correctly, you are looking for goodness-of-fit tests. Check out this link for a basic description:
https://onlinecourses.science.psu.edu/stat414/node/259


How to sketch area/volume found using an integral?

Question: I have a series of practice questions for Calculus 3 that as me to solve some definite double or triple integrals, and ask me to sketch the area or volume of the integral.  I'm not really sure how to do this?
For example, I'm given the following integral and asked to sketch the area:
Integral from 0 to x^2 dy surrounded by integral from 0 to 1 dx.
Or another problem, for example, asks me to sketch the volume:
Integral from 0 to 2 of 3 dy, surrounded by integral from 0 to 1 dx.
Answer: The integrals over $y$ do not contain $y$ and so the double integrals are simple area integrals you know from Calc I. For example, the first integral is the area under the parabola $y=x^2$ from $x=0$ to $x=1$.  Can you do the second?
Comment: Okay, well the second one would be something like: 3y evaluated from 0 to 2 = 6 for the inner integral, and: 6x evaluated from 0 to 1 = 6 for the outer integral, right?


What key could the progression Cm - G - Dm - Am be in?

Question: I'm working on a song (my first ever actually) and I love the chord sequence:
Cm - G - Dm - Am
I started off working in C-Major, but borrowed Cm from the parallel minor key C-Minor. But really, in my mind, it's in A-Minor not C-Major... however is taking a chord from the parallel minor of the associated major key even "a thing"? Cm is the only "weird" chord I want to use (I probably will use F and Em elsewhere) and while I know you can put any chord in any key if it sounds right, how would more knowledgeable theorists analyse this, and is my understanding of why Cm 'fits' in the key of A-minor reasonable?
Update: I recorded a guitar part and looped it as the "verse" to my song:

I actually played it a tone up (Dm - A - Em - Bm) as it's more natural (I'm playing Dm, Em and Bm using the Bm barre shape but with an open top E) but you can transpose your answers however you wish as long as it's clear!
Comment: Everything you can possibly do in music is "a thing" - it's just that some "things" have better-known names than others!  What note do you feel is the root note of the piece?
Comment: @Mr.Boy when you want to add chords, you can use `..` instead of the preformatted text, if you don't want these voicings to show
Comment: @Mr.Boy Your first song - and you are already thinking outside the box - I love it!  When I first started writing, I used all the same tired old tried and true progressions.  And I am still sort of stuck in that rut - so to speak.  Good luck with your song.
Comment: I guess I mean that taking a chord from a parallel minor/major key is quite common, and it fits quite nicely. But I don't know what relation Cm has to the key of A minor - why it works - and if really this progression fits some other key more sensibly.
Comment: It's hard to tell just from the chord sequence (and without knowing how many measures each chord is played for, etc) what is the root. The way you are playing/hearing it, is A the clear tonal center of the piece?
Comment: @RockinCowboy on the one hand I'm like "I wanna be creative" on the other "am I just doing this for its own sake". Maybe a question in its own right - although to me this sequence sounds really melancholic which is what I'm aiming for. Almost Radiohead-esque.
Comment: @Mr.Boy is Cm the first chord in your song? If i write a song in C major or A minor i wouldn't start with that chord. It might fit somewhere down the track, maybe the line before chorus.
Comment: @mey it is. However _all_ the other chords I'm using are from that key - F doesn't seem to work but Em does. The Cm->G sound is perhaps the "feature", I think it's quite dramatic.
Comment: @Mr.Boy perhaps it is in G major then. Cm is a minor iv chord in the key of G and works awesome with G maj chord. (iv-I progression).i love to use this especially in the ending.  Haven't tried it in the beginning though ☺
Comment: Would it be off-topic for me to link to a rough recording of a bit of the song so people can actually analyse the chords/melody?
Comment: @Mr.Boy in the overall scheme of things, it's really not so important to know what key it's in unless you are going to put it in notation and need to write a key signature.  The important thing is that it's a very effective chord sequence and I think it's parallel modulation and I plan to try that concept on some new songs that I write as a way to add something new and different and interesting (might even add as an answer to the question I posted) and I thank you for inventing this new key (Cminor/Major) lol.
Comment: @Mr.Boy Go ahead and link to the recording! Honestly I don't think this is truly answerable until we hear the melody -- context accounts for quite a bit.
Comment: Not every song has to be "in a key". Yours is ambiguous. That's fine.
Answer: Whilst on a major scale ,borrowing chords from the minor scale with the same name is common (and vice versa). Really common to be honest. Some of the most common borrowed chords are the V (dominant), IVm and bVI.
Now for your progression. Since you said you were on C major, we have:

Cm (borrowed)
G (V)
Dm (II)
Am (VI)

Or if we are on A natural minor, we have:

Cm (borrowed)
G (VII)
Dm (IV)
Am (I)

In the most common progressions, after the V or VII you would find I or VI. Not that what you have is wrong, but it is unexpected. (I personally like these).
I think that if you were on C major, it would be the most fitting to change the first chord from major to minor. C to Cm. This usually happens after a dominant chord.
So, if you had a progression like C-G-Cm, it would sound really good. It is a common progression.
So, your progression could make sense in the C major.
As far as A minor gores, substituting the III with the IIIm is something I haven't really seen (or don't remember). But, like you said, if you like it, keep it.
Comment: If there's *no C major* chord in the progression, but there *is* a c minor, isn't it rather a push to say that the piece could be in C major? You don't normally have a key that contradicts the tonality of the (I) chord, so you?
Comment: It would make perfect sense if it was after a dominant chord `This usually happens after a dominant chord.` Also, I don't see why someone can't be on the C major scale and use the chords from there, but instead of C, they use Cm.
Comment: @AndrásHummer interesting - I actually resolve it to `Amaj7` (when I play based on Dm)!
Comment: `So, if you had a progression like C-G-Cm ...`
Comment: `So, your progression could make sense in the C major.` - the OP's progression is `Cm - G - Dm - Am`. So I'm reading that you're saying that `Cm - G - Dm - Am` could make sense in C major?
Comment: After listening to it, I found it to be in C major. In every cycling I'm just waiting for a C major chord to finally resolve this thing :)
Answer: I mostly agree with Shevliaskovic on his approach where A minor and C major are the two most likely, however I want to point out a few other possibilities.
If you look at the notes you use in your progression, the only note outside of the key of A minor and C major is Eb, but E is also used in your piece in the Am chord. So if you put all the notes together you would get the C Major/A Minor blues scale. It's not really a key, but there's nothing stopping you from using this scale to create chords and it will also give you a better way to improvise  over your progression nothing is really borrowed.
Another way to look at it is calling it C minor, but deriving all the chords from the melodic minor ascending scale and just borrowing the E in the Am from the parallel major. This is slightly clunker then calling it just C major or A minor with a borrowed chord, but it makes slightly easier to think about in terms of soloing and melodies and in my opion explains the progression a little better then just thinking C major or A minor.
Comment: Interesting idea with A blues. Although quite often in using the blues scale, notes are played deliberately which are not included in the underlying chords,  rather than matching exactly.
Answer: I'll tell you what my ears tell me. I'm referring to the progression that you actually played:
Dm A Em Bm
I hear this progression not in one key but I hear two tonal centers. The first is A major: iv I. The second is B minor: iv i
There are several reasons why this works and why it keeps circling seemingly without end. In both tonal centers it is a subdominant-tonic progression (no strong dominant, hence the ambiguity). Furthermore, both tonal centers are closely related (3 sharps vs 2 sharps). Also, the final chord Bm is very much related to a D major chord (relative keys), so when the progression moves back to Dm, it sounds like going from (D) major to minor, which is yet another connection of the two parts of the progression. Last, and maybe most importantly, there is a strong underlying chromatic guide tone line through this progression: F-E-G-F#(-F-etc.). This creates this endless circle effect.
Comment: This answer makes the best sense of the musical excerpt provided.   Two tonal just means we modulate from one key to another, and this is not at all uncommon.
Comment: While I understand each thing you say, that level of theory is just a bit beyond my normal level in that I'm not sure what the conclusion is... does "two tonal centres" mean it is key-less/atonal? If what I played - the picking pattern - was being scored how would you accomplish this?
Comment: For what it's worth I was playing with some ideas for a chorus and found that things like `E - Bm - A - E` and even `E - Bm - A - B` worked well... meaning we have minor and major versions of E and possibly B. Does that support or weaken your answer?
Comment: @Mr.Boy: Two tonal centers definitely means that it's tonal (otherwise there wouldn't even be one tonal center). If I were to score it, I'd write it in A major. I guess your ideas for the chorus neither support nor weaken my feeling about the chorus. It is probably also in A (well, 'support' then ...), with the B chord being a secondary dominant leading back to E (the V). But this is just speculative because I don't know your melody etc.
Comment: @MattL. I really struggled who to mark as correct and award the bounty to between Matt and Shevliaskovic as they come from different angles and both seem totally credible. I ended up feeling this one was closer to my intent.
Comment: @Mr.Boy: Thanks, good to know that what you intended was what I heard. I strongly believe that it's always the ears coming first, then the theory (which is always descriptive and can never be prescriptive).
Answer: There are two ways to determine what key something is in. The first is to look at the set of notes (the scale) that you're using, and find a key signature that matches. The second is to find which note chord is being "tonicized." This is done by using a rising-fourth chord progression, which acts as a V-I (dominant-tonic) relationship. Both of these methods can be circumvented or subverted: the first by using chromatic notes outside of the scale, and the second by using secondary dominants. It is worth noting that, since the major and (natural) minor scales share the same key signature, the second method (tonicization) is the best way to distinguish between them.
In your case, neither method works completely:

There is no rising fourth progression at all. In fact, every progression is a falling fourth (also known as a plagal cadence), except for the rising third at the end that takes us from Am back to Cm. As a result, there is no chord that is tonicized. This means that even if we could pick a scale, there's no reliable way to distinguish which mode of the scale (such as major or minor) we are using.
There is no single scale that contains all the notes that you use. Specifically, as Dom notes, you use both an E and an E♭.

So there is no correct answer to the question of what key this progression is in. There's simply not enough information in the progression to establish a key. That's not to say that it is atonal -- there are several keys that it could belong to, but there's not enough information to say which one it definitely is. This is called being tonally ambiguous.

As exotic as it might sound, tonally ambiguous chord progressions really aren't all that uncommon. This type of music is also often called modal, not because it is necessarily in a specific mode (e.g. Mixolydian, or Phrygian, etc...) but because it avoids (or uses sparely) the conventions of tonal music. Some common features of modal style music include: a preference for plagal progressions (falling-fourths) over dominant-tonic progressions (rising-fourths), a frequent use of borrowed chords (♭III, ♭VI, ♭VII), a disreagard for the expected chord quality (using majors instead of minors, or vice-cersa), and using the ♭seven of the key to avoid a strong dominant sound. This style is especially common in modal folk music, or blues-inspired music. It could probably be debated whether it arose by intention, or by accident due to the convenience of certain chord shapes on the guitar.
At any rate, one way that this technique can be used is to weaken the tonic during a verse, so that when a dominant-tonic progression is finally heard in the chorus it will be much more striking. Once you're ready to define the key that you're really in, you need only pick a dominant chord, and following it with a tonic.
Another way that tonally ambiguous progressions can be used is to change between two different keys. In fact, this is an excellent way to modulate gracefully without being noticed.

So which keys could this progression be used in? Many of the existing answers have already given good options for this.
If you go back to looking at the scale, you'll notice that any scale with an E♭ also has a B♭ in it. Since your progression contains a G chord (which has a B natural in) the simplest assumption is that the E♭ is the chromatic note (and thus the C minor is the borrowed chord). This would place the piece in C major/A minor (again, there's not enough information to say which).
On the other hand, you might consider that E♭ and B♭ are part of the scale, in which case the G major chord would have to be borrowed (which suggests C minor, with G being its dominant). However, this would also require the A minor being borrowed (vi instead of the usual ♭VI), which I don't think is a common borrowing.
In fact, if you restrict yourself to keys that only need to contain the roots of all four of your chords (instead of all the notes in the chord), then you could use any key which contains the notes C, D, G, and A:
By my count, these are:

B♭/Gm
C/Am
F/Dm
G/Em

As I mentioned earlier, you can make any one of these keys sound like the home key by simply playing a dominant-tonic progression in that key. As an example: if you wanted to go to Dm, simply replace the A minor chord with an A major (or better: an A7) then follow up with a D minor and you're there. You could even go to D major this way, if you wanted to.
Comment: It should be noted that especially in modern music there are progressions that don't use the dominant still have a tonic that is dictates the key. For example, Led Zeppelin's Good Times, Bad Times definitely has a tonic which is E, even though the dominant chord B major is only used in passing and most of the song is just E major and D major.  Just because the harmony isn't what we expect it to be does not mean there isn't a tonic or a chord that is preceded as a tonic. I do understand the argument, but like I said before  there will be a tonic when a melody is placed on top of the progression
Answer: EDIT: Clarified that the logic outlined herein is best explained as parallel modulation between the MELODIC minor (not natural minor) and major key (see explanation below).
Well there does not seem to be much agreement on what key the song is in and I have seen no completely convincing arguments that put this progression solidly in any one particular key.
I have listened to the sound file and played it myself using the Cm - Am progression and the one played in the recording (Dm - Bm).
What I can say is that is a very effective piece of song composition and succeeds beautifully at conveying a very unique and distinct feeling. To me it defiantly feels more minor than major and seems to create an ambivalent feeling of uncertainty.  Or a feeling like the waves coming to shore and then rolling back to sea. Very melancholy as well.
I am going to say that (aside from the simple fact that the Cm chord is not often used in the key of Cmaj) since it really feels/sounds very minor, it's difficult to reconcile it being only in C major (or any major key given 75% minor chords).  So I would say that it is an arrangement that to a very good effect uses a subtle but constant back and forth parallel modulation between C melodic minor and C major.
C melodic minor contains the Cm the Gmaj and the Dm but would use an A half diminished chord (A C Eb) instead of an A minor.  If we used A half diminished instead of Amin in the progression - it would fit squarely in the key of "C melodic minor" (not C natural minor or C harmonic minor).
And since the G major chord is in both the melodic minor and major parallel keys, it's difficult to tell exactly where the modulation takes place and that to me is part of what makes this progression so powerful.  It's like watching waves come to shore and back - you can't really tell exactly when the wave stops coming in and starts to go back out (what point they actually reverse direction).
The ongoing back and forth parallel modulation creates a yin - yang, ebb and flow effect.  Perhaps unconventional, but again very effective at conveying a certain swaying movement to the music and certain emotional feeling.
So rather than try to put it in a box and say it's definitely in this key or that, I think we should just agree that it can be described as a very effective use of parallel modulation between the melodic minor and major key of whatever tonic chord we open the song with.
I see this as a great example of a composer willing to be inventive and think outside the box.  Sometimes this type of unconventional device is beyond classification.  Other than to just say it's brilliant!
Answer: If your sample is anything to go on, your progression isn't really Cm-G-Dm-Am (or its transposition up a tone), it's (Cm x 3)-G | Dm-Am-Cm-G | (repeat last phrase ad lib.). Each phrase is ending on the major chord, so what you really have is, tentatively, i-i-i-V | (v of V)-(v of ii)-i-V | etc.
I say "tentatively" advisedly, as Laurence is quite right that the passage is tonally ambiguous (which is not the same thing as atonal).
What tends to make C major or minor somewhat ambiguous is that C is nowhere asserted as tonic; rather, it is implied by repetition in the first phrase and the i-V half-cadences at the end of each phrase. Also, of course, either Dm and Am are substitutions (for D° and A♭ respectively) in C minor, or Cm itself is a modal substitution in C major.
No doubt other interpretations are possible, which reinforces what I say about tonal ambiguity.
The start of your second phrase gets its justification through the continuation of the cycle of fifths from the preceding phrase end. Within the phrase, you have a pattern of root motion of down a fourth, up a minor third, down a fourth, so the phrase itself divides into a short harmonic sequence. In effect what the Dm and Am are doing is creating a kind of dominant prolongation on G, as they continue the root motion outwards from G while remaining diatonic in C major. (It is incorrect to call it two keys - there hasn't been enough done to establish a modulation.)
However, I think that, if you were to play a C chord (major or minor - it doesn't matter) immediately after the G chord, you'd very rapidly disambiguate the phrase. In fact, you may find that a bridge of some sort starting on C minor, falling in the flat direction by fifths and rising back to G or G7 works well for this. (That's assuming you want to disambiguate it. It's not always necessary.)
But... the beginning of understanding what you're doing with this passage is to understand how you've phrased it, and that is not with Cm-G at the beginning, but at the end of each phrase.
Comment: Plus one for the interesting observation.  I can see your argument that after an intro on the Cm chord ending on G the progression is actually Dm-Am-Cm-G and not the Cm - G - Dm - Am progression indicated by the text. I heard that as well but ignored what I heard and defaulted to the text. I agree you can't put it in C major or C minor.  If the entire composition remains tonally ambiguous, is it possible to write a key signature? I think it could stay ambiguous and still work but not sure you can actually write a key signature for a composition that is not sure if it's minor or major.
Comment: @RockinCowboy, well, for this passage (and it's long enough to justify it), you could leave it in C major, i.e., no signature, as there is only E♭ to worry about. What happens after this passage depends on how ambiguous any subsequent passages are, and how long they are (not worthwhile changing key signature for four bars, for instance).
Comment: I think it's a stretch to say the passage is squarely in C major (sounds minor to me with all those minor chords).   But I suspect despite it's name, a key signature does not always define the key?
Comment: I wasn't saying that it was in C Major, I was saying that the key signature (no sharps or flats) was appropriate. A key signature can be appropriate for _many_ keys. I have written pieces in E and D where the C major signature covers the key's diatonic collection quite nicely (i.e., they're in white key modes). The only thing that keeps this from sounding like C major, though, is the E♭ in the Cm chords: everything else is diatonic in C major. That and the lack of cadential confirmation is all that keeps it ambiguous.
Comment: Yes I know you did not say it was in C Major.  That's why I was asking you to confirm that a key signature does not necessarily always denote the key - which you did.  Who woulda thunk???
Comment: Hmm, you might be hearing it that way @Patrx2 but in my head as I recorded that example I was playing Cm x 2 as the intro and then jumping into the progression as stated.
Comment: Nope. Not in this case, gents.
Comment: Either it groups in 2 bar phrases or it groups in 4 bar phrases, because it is coming out as a multiple of 4 bars. The vamp is going to _sound like_ all 3 bars of Cm with a cadential dominant in the fourth bar, because the _5th bar_ marks a noticeable change of pattern. Note that this is a very common pattern for intros, ending on a dominant.
Answer: Some observations:
1) It's interesting that base of the chords moves by fifths, I mean:

C +5 --> G
G +5 --> D
D +5 --> A

Is this a thing? I don't know...
2) Having plain chords Cm - G - Dm - Am (having no seventh, sus e.t.c) that contain both Eb, E and F as notes, makes it imposible to be just one minor or major key.
3) Transposing it by a tone (just like the OP suggested and played) gives Dm - A - Em - Bm and it's almost in natural A key. So the progression would be totally "make sense" if
Dm (IV) - Am (I) - Em (V) - Bdim (II), "only" two notes change (quite different sound, I know).
Answer: I'm going with assuming these are the chords from a C Dorian mode, meaning you are really in the key of B flat.
The two accidentals now are changing your G minor to a major and raising the fifth of your A chord to make it a regular minor. Otherwise everything adds up.
Your F chord will also fit right in. Your proposed Em will be interesting to deal with as it is not a shoe in.
Answer: Taking a chord from the parallel minor of the associated major key is a thing!
Let's break your song down to simple notes:

Cm chord - C, Eb, G
G chord - G, B, D
Dm chord - D, F, A
Am chord - A, C, E

The only accidental here is Eb, but we also have B and E, so it's safe to say we're not exactly in Bb major/G minor.
Notes C, D, G and A appear in more than just one chord, and they are mostly minor chords so the A minor sound is surely present here.
The Cm chord itself has C and G, so the only anomaly is that Eb. This note can be considered a tension, because the dissonance it creates contributes to it's purpose inside the song. You need that Eb. It's also a nice melody movement from the E in the Am chord.


Relation between squared standard normals and $\chi^2$

Question: I have a very basic question regarding introductory statistics. It is a well known result that if we have two standardized normally distributed variables $X$ and $Y$, then $$
X^{2}+Y^{2}\sim\chi^{2}(2)
$$
What does this really mean? In practical terms, would $X$ and $Y$ be random normal vectors? Would then the squared sum of the vectors be as above? Or does this type of notation indicate a realized value of the random process? Are we looking at two specific realizations of $X$ and $Y$?
Comment: $X$ and $Y$ have to be *independent* standard normal for the relation to hold.
Comment: You might be confused over random variables, random vectors, realizations of random variables etc. You could find [this](http://stats.stackexchange.com/questions/50/what-is-meant-by-a-random-variable) post describing random variables helpful.
Answer: Suppose

$X$ is standard normal random variable; denote $U:=X^2$
$Y$ is standard normal random variable; denote $V:=Y^2$
$X$ and $Y$ are independent

then

$Z:=U+V$ is a $\chi^2(2)$ random variable.

Here $U:=X^2$ is a random variable generated in the following way: whatever realization is generated by $X$, we take the square of this realization and call it a realization of $U$. The same with $V$ and $Y$. (Any realization of $X$ is univariate, and its square (a realization of $U$) is univariate, too. The same with $Y$ (and $V$).)
Comment: Here is a scenario.  Suppose we do two different statistical tests and want to combine them.  We know the first test gives a statistic $Z_1$ that is standard normal under the null hypothesis (H0: $\mu = 0$ and Ha: $\mu\not= 0$).  Similarly, the second test also has a statistic $Z_2$ that is standard normal under the null hypothesis.  The first test was performed independently of the second test.  To combine these tests, we can form a new statistic $X^2 = Z_1^2 + Z_2^2$.  Since the sum of the square of two independent standard normal random variable has the chi-square distribution with 2 df...
Comment: Thank you for your clear response (+1). I wonder if you could possibly provide with a scenario where this operation would come handy.
Comment: ... we can perform the combined test by looking for a rejection region for $X^2$ using chi-squared tables with 2 df, because this will be its distribution under the null hypothesis H0: $\mu_1=0 \ and\ \mu_2=0$.
Comment: @AlaskaRon very nice. Ty!
Comment: I have a follow up question: if indepedent squared realizations of X and Y when summed are distributed as exactly chi-squared with 2 df, then why is it the case that when I generate 2 random vectors of X and Y and square them and sum then, the mean that I get of the new variable is not exactly 2? Does it have something to do with LLN?
Comment: You cannot sum them completely, like `sum(x^2+y^2)` in R where `x` and `y` are standard normal vectors. Also, `mean(x^2+y^2)` is not supposed to be $\chi^2(2)$-distributed. But if you have a vector `x^2+y^2`, then each of its elements should be distributed as $\chi^2(2)$.


Kubernetes requires Docker 18.09 but CentOS yum upgrades it to 19.03

Question: Kubernetes seem to require docker version 18.09 even if newer versions exist such as 19.03...
When installing on CentOS, I made sure to pass the right version to yum.
Unfortunately, I did a yum update / yum upgrade after and it upgraded docker's version to the newest...
In Kubernetes documentation, it is specifically written to use 18.09, and I also get some warnings after issuing the kubeadm init command.
2 questions here...

Why is still the newest K8s version not supporting 19.03?
How can I make sure CentOS doesn't update 18.09?

Thank you very much!
Comment: Alternatively, check out [versionlock](https://serverfault.com/questions/546100/install-a-specific-package-version-in-centos/546101#546101).
Answer: As You mentioned Kubernetes documentation states that 18.09 is the newest version suggested.

On each of your machines, install Docker. Version 18.06.2 is recommended, but 1.11, 1.12, 1.13, 17.03 and 18.09 are known to work as well. Keep track of the latest verified Docker version in the Kubernetes release notes.

Although main recommendation is for Docker 18.06.2 because it is on the list of validated docker versions found in each kubernetes changelog since kubernetes version 1.12 to current newest stable version.
You can find lists of validated docker versions for each kubernetes changelog here.
The currently newest docker version 19.03 has not been fully tested to be placed on list of validated docker versions.
P.S.:
I have been working with Kubernetes, docker 19.03(current) was working fine and I had no issues on my test environments using Debian 10 and Ubuntu 18.04. Note that on other distributions it might not.
Comment: Debian 10 uses docker 18.09, did you install 19.03 from upstream yourself ? (I know this message was written a year ago, but maybe you remember)


How to get expected value from a probability mass function?

Question: A certain biased coin is flipped until it shows heads for the first time. If the probability of getting heads on a given flip is $5/11$ and $X$ is a random variable corresponding to the number of flips it will take to get heads for the first time, the expected value of $X$ is:
$$E[x] = \sum_{x=1}^\infty{x\frac{5}{11}\frac{6}{11}^{x-1}}$$
I'm not sure how to find an exact value for $E[x]$. I tried thinking about it in terms of a summation of an infinite geometric series but I don't see how that formula can be applied.
Answer: In general, if $X\sim\operatorname{Geo}(p)$, i.e. $\mathbb P(X=n)=p(1-p)^{n-1}$, $n=1,2,\ldots$, then
\begin{align}
\mathbb E[X] &= \sum_{n=1}^\infty np(1-p)^{n-1}\\
&= -p\sum_{n=1}^\infty \frac{\mathsf d}{\mathsf dp}\left[ (1-p)^n\right]\\
&= -p\frac{\mathsf d}{\mathsf dp}\left[\sum_{n=1}^\infty (1-p)^n\right]\\
&= -p\frac{\mathsf d}{\mathsf dp}\left[\frac {1-p}{p}\right]\\
&= -p\left(-\frac1{p^2} \right)\\
&= \frac1p.
\end{align}
Here we have $p=\frac 5{11}$ so $\mathbb E[X]=\frac{11}5$.
Answer: The expectation is not a geometric series (at least not when you write
it directly), but its resemblance to a geometric series is a good observation.
First let's get that factor of $\frac{5}{11}$ out of the way,
because it will become annoying at some point if we keep it inside the
summation.
$$E[x] = \sum_{x=1}^\infty{x\frac{5}{11}\frac{6}{11}^{x-1}}
   = \frac{5}{11} \sum_{x=1}^\infty{x \frac{6}{11}^{x-1}} = \frac{5}{11} S,$$
where
$$
S = \sum_{x=1}^\infty{x \frac{6}{11}^{x-1}}.
$$
Now write out $S$ and $\frac{6}{11}S$:
\begin{align}
\newcommand{x}{\left(\frac{6}{11}\right)}
S    &= 1 \cdot \x^0 + 2 \cdot \x^1 + 3 \cdot \x^2 + 4 \cdot \x^3 + \cdots\\
\frac{6}{11}S &= \phantom{1 \cdot \x^0 + }\
                 1 \cdot \x^1 + 2 \cdot \x^2 + 3 \cdot \x^3 + \cdots
\end{align}
From here you should be able to work out what $S - \frac{6}{11}S$ is as a series, taking the difference of the right-hand sides of the two equations above, and then apply what you know about geometric series.
Notice how conveniently $S - \frac{6}{11}S = \frac{5}{11}S$,
which happens to be the value we need in the end.
Answer: $X$ is a geometric random variable with parameter $p$. A way to compute its expected value is through the total expectation theorem:
\begin{align}
E[X] &= E[X\mid X=1]P(X=1) + E[X\mid X>1]P(X>1)\\
\end{align}
When you already know that $X=1$, its expected value is 1, therefore $E[X \mid X=1] = 1$.
When you know that $X>1$, i.e., $X = 2, 3, \ldots$, then you can imagine that you have a "new" random variable, $X-1$, with values $X-1 = 1,2,\dots$, and that is also geometric! This remarkable property of the geometric distribution is formally called the memorylessness property, and implies that $E[X-1 \mid X>1] = E[X]$. Finally,
\begin{align}
E[X] &= E[X\mid X=1]P(X=1) + E[X\mid X>1]P(X>1)\\
     &= P(X=1) + E[(X-1)+1 \mid X>1]P(X>1)\\
     &= p + (E[X-1 \mid X>1]+E[1 \mid X>1])(1-p)\\
     &= p + (E[X]+1)(1-p)\\
\end{align}
This shows that $E[X] = \frac{1}{p}$. In your case $p=\frac{5}{11}$.
Here a great explanation of this approach (minute 24:54) from the MIT's professor John Tsitsiklis: https://youtu.be/-qCEoqpwjf4?list=PLUl4u3cNGP60A3XMwZ5sep719_nh95qOe
Best.


How to correctly get the twig value

Question: I have the following response using kint. How can I correctly get the value of the title.
I tried stream.node_title[0]['#context'].output but getting null result.
Answer: Looks like it should be
{{ stream.node_title[0][0]['#context'].output }}

You can get the other two with
{{ stream.node_title[0][1]['#context'].output }}
{{ stream.node_title[0][2]['#context'].output }}

But obviously this would be hard coded. To make all of this dynamic, you can loop.
{% for title in stream.node_title[0] %}
    {{ title['#context'].output }}
{% endfor %}

Comment: wow perfect! I think the reason why there's an additional `[0]`, its because theres an array. Thank you very much!
Comment: thank you for your quick response @No Sssweat. Will try that now.

How to correctly get the twig value


What is the simplest way to understand $\nabla \cdot (\nabla \times \textbf{A} ) =0$?

Question: Of course one can prove it by brute force.
But how to Understand it, in the simplest way?
Another issue is $\nabla \times (\nabla f) =0 $.
Comment: If you're a physicist, you're going to have to learn a huge amount of differential geometry eventually. It's a huge part general relativity, gauge theory, and it even pops up in electromagnetism, quantum mechanics (through the Berry connection), etc.
Comment: A surprising amount of physics these days takes a fair bit of knowledge of differential geometry. I can't speak for your specific field of knowledge, of course, but it's certainly not a bad thing to know just because you're a physicist.
Comment: In the language of [differential forms](http://en.wikipedia.org/wiki/Differential_form), both of these are specific cases of the fact that $d^2 = 0$, where $d$ represents the [exterior derivative](http://en.wikipedia.org/wiki/Exterior_derivative).
Comment: If you've learned Stokes' theorem and the Divergence theorem, these together with the fundamental theorem of line integrals explain why these vanish.
Comment: No, I am a physicist. I do not want to plunge into differential geometry.
Answer: It's just a way to remember/understand. Think that $\nabla $ is a vector. Then, by definition, $\nabla \times A$ is perpendicular to $\nabla $ and so the product scalar of $\nabla $ and $\nabla \times A$ is nul, therefore
$$\nabla \cdot  (\nabla \times A)=0$$
Comment: True, but unlike real vectors which satisfy $\mathbf{A} \times \mathbf{A} = 0$, the $\nabla$ operator does not, as can be shown $\nabla \times \nabla \times \mathbf{A} \neq 0$.
Answer: Here is an explanation based on Stokes' theorem, the Divergence theorem, and the fundamental theorem of line integrals. Recall that Stokes' theorem states that the line integral along a closed curve is equivalent to a certain surface integral of the surface enclosed:
$$\int_{\partial D} \mathbf{F}\cdot d\mathbf{r}=\int_{D} (\nabla\times \mathbf{F})\cdot d\mathbf{S}$$
But the fundamental theorem of line integrals states that $\int_a^b(\nabla f)\cdot d\mathbf{r}=f(b)-f(a)$. So if $\mathbf{F}=\nabla f$ for some function $f$, then
$$\int_{D} (\nabla\times (\nabla f))\cdot d\mathbf{S}=\int_{\partial D} (\nabla f)\cdot d\mathbf{r}=f(b)-f(a)=0$$ since the endpoints coincide (it's a closed loop). So the surface integral vanishes for any surface $D$; the only way this can make sense is if $\nabla\times \nabla f=0$.
Similarly, recall that the Divergence Theorem states that we may express the flux integral over the surface of a volume $V$ as
$$\int_{\partial V} \mathbf{F}\cdot d\mathbf{r}=\int_V (\nabla\cdot \mathbf{F})dV$$
Now, if $\mathbf{F}=\nabla\times \mathbf{A}$, then Stokes' theorem implies that we may write this as a line integral of $\mathbf{A}\cdot d\mathbf{r}$ along the boundary of the surface $\partial V$---but this surface has no boundary! So the integral vanishes immediately. (If that seems too quick, cut the surface integral in two and show that the two line integrals obtained from Stokes' theorem cancel.) Therefore the volume integral over $\nabla\cdot \mathbf{F}=\nabla\cdot(\nabla\times \mathbf{A})$ vanishes for any volume $V$, and so we must have $\nabla\cdot \mathbf{(\nabla\times \mathbf{A})}=0$.


Rationalizing denominator containing a root of a polynomial - why is this possible / why does it work?

Question: The problem is to express the number
$$1\over x^5 + 2x^4 + 3x^3 + 3x^2 + 2$$
using only rational numbers in the denominator, knowing that $x^5 + 2 = -2(x + 1)(x^3 + x^2 + x)$.
(This is an example from a final exam of an algebra course from the past year (which is freely available to students).)
I know how to solve this problem, but I do not understand why this is possible. The trick comes from knowing that this number can be expressed using addition and multiplication of $x$ and rational numbers, ie. as
$$r_0 + r_1x + r_2x^2 + r_3x^3 + r_4x^4 + r_5x^5 + \dots$$
My solution follows, skip to questions if you are not interested.
From the polynomial equation it follows that $x^5$ equals $-2 - 2x - 4x^2 - 4x^3 - 2x^4$, therefore an expression containing $x$ with exponent 5 or greater can be rewritten as an expression with only $x^0$ to $x^4$.
So my number can be expressed as
$$ax^4 + bx^3 + cx^2 + dx + e$$
Putting an equals sign between these two expressions and multiplying by the denominator gives
$$1 = (x^5 + 2x^4 + 3x^3 + 3x^2 + 2)(ax^4 + bx^3 + cx^2 + dx + e)$$
I can multiply the right hand side and "normalize" it (by replacing $x^{\ge5}$) to get
$$1 = (e - f)x^4 + \dots + (2e - 2d)$$
From this formula I can create a system of linear equations, because the coefficient of $x^n$ has to be one if n equals zero, and zero otherwise. Solving this system will let me find the values of the coefficients.
Question 0: Why does this problem even have a solution of the form I used?
Question 1: Is there a solution like this for any expression $1 \over f(x)$, where f is a polynomial of $x$, knowing $x$ is a root of another polynomial? Will my method always work? Why?
Apologies if this is a duplicate, but everything I could find about rationalizing the denominator was about simple fractions like $1 \over 2 + \sqrt 6$.
I'd appreciate even just comments directing me towards what I can study because I am out of keywords.
Comment: I don't understand what we are talking about. "The problem is to express the number:" then something *which is not a number* follows. Please, add more context, by saying what $x$ represents.
Comment: I used the wording from the exam, but I guess it would be more appropriate to state "the problem is to express the expression / rewrite the expression". Do you agree?
Comment: $x$ is any number satisfying $x^5 + 2 = -2(x + 1)(x^3 + x^2 + x)$
Comment: Have you ever heard of Galois Theory ? You should check out any undergrad level book/ lecture notes about this subject.
Comment: See also [here](https://math.stackexchange.com/q/100195/242) and [Why rationalize the denominator?](https://math.stackexchange.com/q/1084891/242)  I explicitly highlight the viewpoint of *rationalizing the denominator* in some of my answers there and in the dupes.
Answer: In short one possible way to learn why this works is via Algebraic Number Theory, but it could also come from other areas.

The given equation means $x$ is an algebraic number, which are complex numbers that satisfies an irreducible (cannot be factored over $\mathbb Z[x]$) integer polynomial with finite degree. In this case it's an algebraic integer since leading coefficient of the determining polynomial is $1$ (since $1\cdot x^5$).
Usually we fix one of the roots of
$$
x^5 + 2 = -2(x + 1)(x^3 + x^2 + x),
$$
say $\alpha$, and work using it (so all $\alpha$'s instead of $x$'s), but we'll continue with $x$ here since it's largely similar for intuition purposes.
Now the key part is if you studied some basics of Algebraic Number Theory, one of the first things you'll learn is that sum and product of algebraic integers are still algebraic integers (same for algebraic numbers). This means that the new arbitrary polynomial you formed, which we denote as $\beta$:
$$
\beta:x^5 + 2x^4 + 3x^3 + 3x^2 + 2
$$
is an algebraic integer (sum and product of $x$) and hence must satisfy some irreducible integer polynomial $g(y)$ with leading coefficient $1$ (That is the definition of algebraic integers). So you have
$$
g(y):=y^k+a_{k-1}y^{k-1}+\cdots+a_0,\;\;\;\; g(\beta)=0
$$
Note that the $a_i$'s are integers and $y$ is a variable. Now in order for $g(y)$ to be irreducible, $a_0$ cannot be $0$. Otherwise we can factor out $y$. Then since $a_0\neq 0$ we can now form
$$
\begin{align*}
0 &= g(\beta) = \beta^k+a_{k-1}\beta^{k-1}+\cdots + a_1\beta + a_0\\
\frac{-(\beta^{k-1}+a_{k-1}\beta^{k-2}+\cdots + a_1)}{a_0} &= \frac{1}{\beta}
\end{align*}
$$
So your $1/\beta$ always can be turned into an expression in terms of rational coefficients. Notice that the only requirement is that $\beta$ is an algebraic integer (works similarly for algebraic numbers), which provides you the equation to work with.

On a related note, using your linear algebra arguments you can also argue in an elementary way that such a $g(y)$ exists. i.e. solve for such a $g(y)$.


Google Maps API Alternative Route Description

Question: I am working with Google Maps API V3 to calculate all the possible routes from a given Source to the specified Destination. For this I takes the Destination and Source as inputs from the user and pass these values in the request with option provideRouteAlternatives: true. I am successful in calculating different Routes and also marking i.e. displaying them on my Map.
Now I am wondering if it is possible to name the Different Routes. By naming the Routes I mean adding a small description to each Route like the Street Name etc. which that particular route takes while moving from Source to Destination. I want to do this so as to make it look more interactive. For example if there are 3 Routes possible from location A (Arizona Street, San Diego, CA, USA) to location B (Fenton Parkway, San Diego, CA 92108, USA) then instead of showing results like Route 1, Route 2, Route 3 it should show the Street names like "Texas St & Rio San Diego Dr", "Texas St & Friar Rd", "Friars Rd".
I am able to calculate all the other things like the trip distance and total trip time for each of my Route but have no clue to how to add this description to each of my Route.
Any help will be appreciated.
Thanks
Answer: That is what appears in the panel if you provide one:
Suggested routes:
Texas St and Rio San Diego Dr 2.6 mi - about 9 mins
Texas St and Friars Rd 3.3 mi - about 10 mins
Friars Rd 4.8 mi - about 11 mins

example
Comment: Exactly...But I want these results to be added into a ListBox as I am not displaying the Panel. So is there any way to access these using JavaScript?
Answer: Try using summary from the returned result. From the Google Maps API:

summary contains a short textual description for the route, suitable for naming and disambiguating the route from alternatives.


Reliable file transfer between three systems

Question: I've got three systems I need to transfer files between daily:

Vendor A (SFTP site) - Deletes files after 14 days
Vendor B (SFTP site) - Deletes files after processing them
Company Server (Windows 2008) - Stores files as backups

Each day I want to download new files from Vendor A to our company server and upload them to Vendor B. The caveat with a simple file transfer script is that the file transfer from Vendor A to the company server might succeed but fail when transferring from company server to Vendor B. Thus vendor B is missing a days worth of files. Vendor B doesn't tell me when they are missing files and Vendor A doesn't always generate a file on a given day.
Any ideas on how to automate this file transfer scenario so that in the event of a failure the system will try again at a later time?
Answer: You just have to add some logic to retry the upload to Vendor B after some period of time. In case you reach the maximum number of failures, log an error or send an email to have it verified manually by the support team.
If you expect Vendor A won't generate files every day, determine how important the missing days are the decide if you should also generate an alert/email for those or not (e.g. you have to call Vendor A to check why the file is missing at the expected time).
Comment: Add in *serious* logging, and alerting as appropriate.
Answer: I'd do:

One-way synchronization of files (Vendor A -> Company Server "backup" folder).
Clone newly downloaded files to a separate folder with files to upload ("upload" folder).
Check how old the latest file in "backup" folder is. When older than X days, alert.
Try to upload all (not only new) files from "upload" folder to Vendor B
Remove files that you succeed to upload from "upload" folder
Alert on any (download or upload) error

Repeat this at least once a day, possibly few times a day. Or do the steps 1-3 once a day, but the steps 4-6 few times a day.
Possible implementation using PowerShell:

Use the WinSCP .NET assembly method Session.SynchronizeDirectories:
$synchronizationResult = $session.SynchronizeDirectories(
    [WinSCP.SynchronizationMode]::Local, "c:\backup\", "/remote_path/", $False)

Iterate the SynchronizationResult.Downloads and Copy-Item each:
foreach ($download in $synchronizationResult.Downloads)
{
    Copy-Item $download.Destination -Destination "C:\upload"
}

Use the Get-ChildItem to find the latest file:
# Find the latest file
$latest = Get-ChildItem "C:\backup" |
    Sort-Object LastWriteTime -Descending | Select-Object -First 1

# Is the latest file older than 5 days?
if ($latest.LastWriteTime -lt (Get-Date).AddDays(-5))
{
    # alert!
}

Use the WinSCP .NET assembly method Session.PutFiles:
$transferResult = $session.PutFiles("C:\upload\*.*", "/remote_path/")

Iterate TransferResult.Transfers and Remove-Item each:
foreach ($transfer in $transferResult.Transfers)
{
    if ($transfer.Error -eq $Null)
    {
        Remove-Item $transfer.FileName
    }
}

Test both the SynchronizationResult.IsSuccess and the TransferResult.IsSuccess

(I'm the author of WinSCP)


Debug assertion does not prompt in IIS 7

Question: since moving to Windows 7 (IIS 7.5), the debug assertions do not prompt a pop up dialog anymore.
I have tested this in a separate project, and noticed that they do work when using the integrated Visual Studio Developer server (Cassini) but they do not work when using IIS Web Server.
This is a big issue for us since we are counting on debug assertions to identify potential programming errors, so any help would be appreciated.
Thanks. Eyal.
Answer: That's because failed debug assertions are displayed in the Output window now instead, under the Debug section.
To view the Output window in Visual Studio 2008 go to the 'View' menu and click 'Output'.
I also find it inconvenient. Some more info...
Answer: That is the default behavior now and it cannot be changed.
What you can is generate a file with the assertions that failed in your app. That will still help you track your issues without halting the app when running inside the IIS.
Take a look at this good article about it.
Answer: You can work around by using
<code>System.Diagnostics.Debugger.Launch();
</code>
You can make for example this function
<code>[Conditional("DEBUG")]
public static void AssertEx(bool condition, string message)
{
    if (condition) return;

    System.Diagnostics.Debugger.Launch();

    // Still write the message on output
    Debug.Fail(message);
}
</code>
and get similar results.


Creating large file ties up memory after process is done

Question: I'm using an Ubuntu 14.04LTS server (AWS EC2) to do a custom database dump but I'm getting a very odd memory behavior.
I'm using PHP CLI script to loop through the custom dump query that outputs to a numbered file then cat that file into the output file. The memory usage climbs as expected because of the mysql. The mysql dump file and the output file are all written to a separate mounted drive (EBS).
Before starting the script the memory usage is about 20%. Once the process is done the memory usage is about 90% based on graphs in htop. I've verified the usage with free -m. A mysql restart drops the memory usage down to about 50%.
I've used lsof to verify that nothing is using the file. I've even done lsof on the directory and only get my bash process and lsof.
The really weird part is as soon as I delete the file the memory usage drops back to 20%.
I've tried deleting the file without restarting mysql and the usage drops by the 30% the numbers above would expect.
Using the memory viewer script found here does not show the same memory usage so I assume it's not completely accurate. It shows no memory consumption by keeping the file.
Is there some other way I can tell what is holding the memory?
Answer: What you're actually looking at is the system's disk cache usage.  All modern system will cache disk operations to make recurrent access of the same disk area faster.
This caching process is considered to cost very little resource for the frequent benefits it may generate later.  Since the system knows that the cache is not important compared to any other memory requirement, it will instantly dump parts of it should it be needed for something else.
You are also noticing the MySQL server process' internal caching which will act similar to the operating system's cache, but the OS does not recognize this as cache, which is why it is only freed based on operations through the server process itself, such as restarting it.
Comment: Disk cache seemed to be exactly what it was. Looking for a flush I came to `echo 3 | sudo tee /proc/sys/vm/drop_caches` which immediately dropped it back down. Kind of annoying when the instance reports memory usage and it doesn't drop back down.
Comment: @DavidFairbanks Use that command very sparingly, as it will nearly always *decrease* the performance of your server.
Comment: @EEAA I understand and it does feel like a hack. I't only used during the script execution which only happens once a month. I wish there was a way to temporarily disable it for a process.
Comment: Disk caching by process is not possible because the memory that would be needed just to track which process uses which part of the cache would drastically increase the cache's memory usage and decrease CPU performance.
Comment: You worry too much on something that will not impact your server's operations.  You should probably look at a better approach to monitoring the usage than considering the cache usage as a problem.
Comment: @DavidFairbanks I agree 100% with Julie. You do not need to worry about this type of memory usage. The cache consumed by this file will be **immediately** given up to any process that tries to allocate RAM. Additionally, after the dump, as your system reads other files, those will gradually replace the dump file in cache. Life is too short to worry about this sort of thing. Let the kernel's memory management routines so what they're good at.
Comment: Thanks EEAA and Julie, that's exactly why I put the question up. I was getting crashing but I guess that was handled with the mysql changes. Looking at the memory stats got me worried that a production server would crash. Trying the full job it does in fact run now.


Customizing menu html

Question: How do you set an id for the menu's root
 tag?
I know about the module "Menu Attributes" but that doesn't let me add an ID to the root
, or at least I do not know how.
Also, I've seen a solution which uses theme_menu_tree() and a regex check, but that's just too hacky for my taste.
Answer: In D7 default theme bartik, you can see the main menu is rendered using a theme hook links__system_main_menu:

    <?php print theme('links__system_main_menu', array(
      'links' => $main_menu,
      'attributes' => array(
        'id' => 'main-menu-links',
        'class' => array('links', 'clearfix'),
      ),
      'heading' => array(
        'text' => t('Main menu'),
        'level' => 'h2',
        'class' => array('element-invisible'),
      ),
    )); ?>

links__system_main_menu is a theme hook pattern of the form [base hook]__[context].
When links are themed with theme('links__system_main_menu', $vars), theme() will search for and use theme_links__system_main_menu() if it has been defined. If not, it will use theme_links().
Drupal will use MYTHEME_links__system_main_menu(), if you define it in your theme.
Thus, you could implement the hook_links__system_main_menu as follows. Probably you will do that in the template.php file in your theme folder.:
function mytheme_links__system_main_menu($variables) {
  $html = "
\n";
  $html .= "
\n";
  foreach ($variables['links'] as $key => $link) {
    $html .= "
".l($link['title'], $link['path'], $link)."
";
  }
  $html .= "
\n";
  $html .= "
\n";

  return $html;
}

This will override the links__system_main_menu theme and customize HTML output for the main menu.
[Edit]
If you are looking for the quickest way, you just need to tweak your page.tpl.php by adding an extra wrapper to the menu ul.
For example, in the Bartik theme,

    <?php print theme('links__system_main_menu', array(
      'links' => $main_menu,
      'attributes' => array(
        'id' => 'main-menu-links',
        'class' => array('links', 'clearfix'),
      ),
      'heading' => array(
        'text' => t('Main menu'),
        'level' => 'h2',
        'class' => array('element-invisible'),
      ),
    )); ?>

Then, you can access it using #your-id ul{ } in CSS or $('#your-id ul') in JS.
But, the above code is just for example, you should not hack the core themes or modules.
I assume you have your own theme.
Comment: @Ashnur: if you want to create the html document as you like, I think you should not use Drupal. If you use Drupal, you have to follow Drupal's rules, standards and API, but you can feel the power of Drupal. Theming in Drupal is not as simple as creating a custom html file.
Comment: So I basically have to recreate the whole menu structure just so that I can put a html attribute in an already existing menu? I am not sure about this. Why is this better than just search/replace?
Comment: So, you used the hook `mytheme_links__system_main_menu` as my answer?
Comment: yes, I did try out this solution, and it does work, but it kinda freaks me out. I have to edit two files, add a lot of arbitrary code which does not have any correlation with the original issue, which is, adding a simple html id attribute to a specific ul tag.
Comment: I just updated my answer. Please see the Edit section. It would be a simplest solution.
Comment: Your update is ok, but it's not what I need. The menu already has wrappers with ids. I need to set an id attribute for the `` tag. Not for a parent of it, and I woul preferably not write a whole menu structure just for this small thing.
Comment: Another issue with this approach is that it won't display submenus.
Comment: Another way is to write js code to add the html id to a specific  tag. It would also require to create a javascript function. I think that's all Drupal can do for you. If you can find a very very simple solution by yourself, please post it and it will be appreciated.
Comment: And can you please tell me the purpose to set an id attribute for the  tag?
Comment: Dear @Sithu, the question is not why, or for what purpose I want to do something, because if we are asking such questions, we could simply start at why should I use drupal, php or even the internet. The answer is of course, that I am trying to create the html document as I like it. Clean and simple. Also, using javascript to modify an already created html node might be a good solution for some people, but I don't think of that as a Drupal solution, just another hack.
Comment: thanks for clearing that up. I've had to come to this conclusion after having issues with such small things. Anyway, thanks for the help, your answers helped me understand more about drupal theming.
Answer: You want to overwrite menu tree theming function as I mentioned in IRC. Easiest way would be to copy the attribute usage from theme_links.
Using this method does affect all menus, but it doesn't change anything for those which doesn't use attributes.
// Default theme_menu_tree
function theme_menu_tree($variables) {
  return '
' . $variables['tree'] . '
';
}

// Overwritten theme_menu_tree
function YOURTHEME_menu_tree($variables) {
  $attributes = array("class" => array("menu"));
  if(isset($variables['attributes']))
    $attributes = array_merge_recursive($attributes, $variables['attributes']);
  return '
' . $variables['tree'] . '
';
}

Comment: That will affect every call to `theme_menu_tree()` though. As the OP wants to add a css _ID_ (most likely a specific one), this isn't really an option as it will result in multiple elements with the same ID. You wouldn't even get away with using `drupal_html_id()` as I imagine the whole point of this is so that the OP can set a predictable ID to target with CSS
Comment: I found a working hack with this approach where we use a string lookup to be sure we are modifying the one node we want to modyfy.
I guess if this function could get a bit more info about what it's contents are going to be, it wouldn't be so hacky, but I am unsure if that's possible to achieve just with hooks.
Answer: I managed to get it right with the following code, based on @Sithu's response
function themename_links__system_main_menu($variables) {
    $html = '
'."\n";
    foreach ($variables['links']['system_main-menu'] as $key => $link) {
        if(is_array($link)){
            $html .= "
".l($link['#title'], $link['#href'], $link)."
";
        }
    }
    $html .= "
\n";
    return $html;
}

Comment: but I still don't think that this should be the accepted answer. It's just way to much code for such a simple thing.
Comment: Why did you post the same answer as a different post? This is not a way of this community.
Comment: If you read it carefully you will see that it's not quite the same answer. The one you posted thrown a lot of errors, this one worked for me.
Comment: Yes, I can see that, but the logic and hook usage are the same. I did not expect from you to accept my answer. And you should not also expect error-free ready-made codes from this community. You can also see some accepted answers in this site are just advice.

Customizing menu html


O que usar para armazenar status em banco de dados?

Question: Estou criando um banco de dados em MySQL e nele vai haver uma tabela para fazer controle de ponto: quando o funcionário iniciou o expediente, quando encerrou, quando parou para almoçar e etc.
Estive fazendo um protótipo de tabela:
+------------------+---------------------+--------+----------+-------------+
| employee_name    | datetime_registered | status | register | observation |
+------------------+---------------------+--------+----------+-------------+
| Fulando da Silva | 2020-10-28 14:49:00 | 0      |        1 | NULL        |
+------------------+---------------------+--------+----------+-------------+

Sendo que status é um código que vai traduzir para: início de expediente, término de expediente e etc. E o backend teria uma regra de negócio para fazer essa tradução do código para o valor. Pensei dessa forma para economia de memória, uma vez que armazenar uma string de tipo 'Início de expediente' ocupa mais memória a longo prazo que apenas '0', '1' ou '2'.
Porém não acho que essa seja a melhor prática para isso. Pesquisei sobre e não achei nada diretamente relacionado a isso.
Explicando o restante da tabela: o campo employee_name guarda o nome do funcionário, o campo datetime_registered guarda o horário que o status de expediente foi atualizado, register é o id de cada registro, sendo a chave primária da tabela, e observation é um campo de texto que armazena alguma ocasional justificativa, caso o funcionário encerre o expediente mais cedo e precise justificar de forma sucinta.
Answer: Me parece óbvio que o tipo tinyint é o mais adequado para guardar os códigos de estado e depois na aplicação faz a descrição em cima de cada código. Mas qualquer inteiro pode ser útil se o espaço não for absurdamente restrito. Já falei sobre isso em Varchar ou int para campos "tipo".
Algumas pessoas podem pensar em um enum mas não vejo vantagem. Tem link na resposta linkada que fala das desvantagens disso.
E claro que pode ter uma tabela auxiliar com a descrição dos estados, que pode trazer alguma vantagem, mas acho que traz desvantagens demais para adotar. Toda consulta teria que fazer uma busca secundária o que deixaria lento. Teria que usar uma chave estrangeira. Em alguns cenários pode não ser um problema, mas acho que complica a escrita, deixa lento para ter uma facilidade muito pequena na leitura que a desvantagem acaba anulando.
Não consigo imaginar um motivo forte pra usar string nesse caso porque é um código identificador, não é algo descritivo, mas também não é um grande problema, mesmo consumindo um espaço um pouco maior, só acho mais fácil o número.
Não sei o problema todo, mas pode ser que a modelagem esteja ruim e deveria fazer outra solução que talvez nem envolva essa coluna. Pode ser o ideal assim mesmo, só estou abrindo a porta para pensar sobre.
Answer:
Pensei dessa forma para economia de memória, uma vez que armazenar uma
string de tipo 'Início de expediente' ocupa mais memória a longo prazo
que apenas '0', '1' ou '2'.
Porém não acho que essa seja a melhor prática para isso. Pesquisei
sobre e não achei nada diretamente relacionado a isso.

Primeiro, o status pode ser descrito no banco de dados normalmente, o problema aqui é que isso está desnormalizando a tua tabela.
Se o objetivo da tabela é armazenar os dados de ponto, e o status não faz parte da chave, segundo a normalização deveria estar numa entidade diferente, por exemplo "RegisterStatus".
Essa tabela sim pode ter o id e a descrição do status, deve haver um único registro por descrição, sendo o espaço consumido irrelevante baseado no que colocou como exemplo, e na tabela que colocou no exemplo, apenas o id, que seria a chave estrangeira. O seja:

Ou seja, uma tabela/entidade para o status, que resolve o problema da descrição, e uma foreign key na outra tabela, resolvendo o problema do tipo do campo e também da correta normalização do modelo.