Differences in book definitions of $\beta$-reduction leading to confusion on when an abstraction needs to be $\alpha$-converted

Question: I've been learning Lambda Calculus in my free time recently to try and learn how to make programming languages & interpreters. I am struggling a little bit with some inconsistencies on different texts on when to use $\alpha$-conversion to avoid capture in $\beta$-reduction.
For example, in Lectures on the Curry-Howard Isomorphism, it makes no mention of any capture restrictions in its definition of $\beta$-reduction.
$$
\label{1}\tag{1}
(\lambda x.P)\, Q \rightarrow_\beta P[x := Q]
$$
However, in Write You A Haskell, it does

The fundamental issue with using locally named binders is the problem of name capture, or how to
handle the case where a substitution conflicts with the names of free variables. We need the condition in
the last case to avoid the naive substitution that would fundamentally alter the meaning of the following
expression when y is rewritten to x.
$\label{2}\tag{2}[y/x](\lambda x.xy) \rightarrow \lambda x.xx$
By convention we will always use a capture-avoiding substitution. Substitution will only proceed if the
variable is not in the set of free variables of the expression, and if it does then a fresh variable will be created in its place.
$\label{3}\tag{3}(\lambda x.e)\,a \rightarrow [x/a]e \quad\text{if}\quad x \notin fv(a)$

Which this piece makes sense to me that if you're substituting a free variable in the body of a lambda expression it can become captured, however the second part is what is throwing me off. The first part appears to be dealing specifically with substituting in lambda abstractions, where the second part dealing with "capture-avoiding substitution" is for any lambda expression.
So I'm not sure I understand why you would need to rename in the following case
$$
\label{4}\tag{4}
(\lambda x. x\ y)(\lambda y. y x) \rightarrow_\beta  [x/x](x y)
$$
According to $\ref{3}$, this wouldn't be allowed because $x \in FV(\lambda y. y x)$, but I don't see the problem with it and it seems pretty straightforward to reduce this to $y x$.
When I plug into a lambda calculus interpreter online with steps (such as this one), it agrees and reduces it down to $y x$ with no renaming.
Renaming seems to only apply if the body of the abstraction of the $\beta$-redex is itself another lambda abstraction, such as
$$
\label{5}\tag{5}
(\lambda x.\lambda y. x y)(\lambda x. x y) \rightarrow_\beta [x/(\lambda x.x y)](\lambda y. x y)
$$
which seems to me also according to $\ref{3}$ would not require an alpha conversion because $x\notin FV(\lambda x.x y)$, however when I plug that into  the same lambda calculus interpreter it does alpha convert.
One example that I worked out that makes perfect sense to me is the following (using the other notation/definition from Lectures on the Curry-Howard Isomorphism, side note: the differing notations confuse me also):
$$
\label{6}\tag{6}
\begin{align*}
(\lambda y.\lambda x.x y)(\lambda z.x z) &\rightarrow_\beta (\lambda x.x y)[y := (\lambda z. x z)] \\
&\rightarrow_\alpha (\lambda a. (xy)[x := a][y := (\lambda z. x z)]) &\text{if}\quad x\in FV(\lambda z. x z)\,\text{and}\, y\in FV(x y) \\
&\rightarrow (\lambda a.(a y)[y := (\lambda z. x z)] \\
&\rightarrow (\lambda a.a (\lambda z. x z))
\end{align*}
$$
and the condition makes sense because you need to rename $x$ if it's in the free variables of the substitution to avoid capture, and $y$ needs to be in the free variables of the body because you can only substitute free variables.
But then when I go back to the other examples I end up re-confusing myself on when I need to rename because it seems inconsistent.
In total, the question is

What am I doing wrong in the examples that are confusing me using the Write You A Haskell ($\ref{3}$) definition?
Am I correct that the renaming only applies when the $\beta$-redex abstraction body is itself another abstraction?
Answer: The rule from Write you a Haskell is incorrect. The correct rule is:
The reduction $(\lambda x . e) a \to_\beta [x/a]e$, where $x$ is a variable and $a$ and $e$ are terms, holds whenever the no free variable of $a$ is a bound variable in $e$. Here, $[x/a] e$ means a simple replacement of all occurences of $x$ in $e$ by $a$.
Whether $x$ is free in $a$ is completely irrelevant. What is relevant is that free occurences of a variable $t$ in the term $a$ do not become "captured" by $\lambda$-abstractions in $e$.
Your first reduction would thus proceed: $(\lambda x . x y) (\lambda y . y x) \to_\beta (\lambda y . y x) y \to_\beta yx$ with no further opportunities for reduction.
The reduction that confused you is that of $(\lambda x . \lambda y . xy) (\lambda x . xy)$. In this case, we see that $y$ is a free variable in $\lambda x . x y$ but $y$ is a bound variable in $\lambda y . xy$. Thus, it is not possible to do a $\beta$ reduction immediately: we must first do an $\alpha$-conversion. $(\lambda x . \lambda y . xy) (\lambda x . xy) \equiv_\alpha (\lambda x . \lambda z . xz) (\lambda x . xy) \to_\beta \lambda z . (\lambda x . xy) z \to_\beta \lambda z . zy$.
If you permitted an immediate $\beta$-reduction, you would get the following:
$(\lambda x . \lambda y . xy) (\lambda x . xy) \to_\beta \lambda y . (\lambda x . xy) y$
Notice that $y$ no longer occurs free. Continuing:
$\lambda y . (\lambda x . xy) y \to_\beta \lambda y . yy$
So we get that $\lambda z . zy \equiv \lambda y . yy$, which is clearly not what we expect intuitively. This is, of course, because we did an invalid $\beta$ reduction.
Comment: That's really unfortunate that it has the definition wrong, I was excited to build a language following it. Thank you for the clear explanation!
Answer: Εven though the answer of Mr Mark Saving is complete, I would like to add something, since I am also studying the CHL isomorphism. There is a formality that is usually admitted, about the notion of pre-terms. By defining pre-terms as you would define regular λ-terms, you can then define α-conversion as an equivalence relation, with a "minimality clause" which enables inductive proofs on the equivalence classes. Finally, λ-terms are defined as these equivalence classes. It can be proved that substitution is always defined for these λ-terms, since α-conversion is assumed to be an automatic procedure.
I know you are looking at these from a programmer's POV, since you actually want to create a language, so there is no "automatic procedure". Despite that, I can send you a pdf of my work, where I try to create a theoretical background for efficient variable renaming. That is, if you are interested.
Comment: That'd be wonderful. I'm interested in the practical uses of course, but the theory interests me also. Thanks!


Shell enviar mensagem de erro

Question: Tenho uma tarefa cron rodando que chama um shell script. Esse arquivo sh faz um backup do banco de dados e salva em uma pasta no servidor. Até aí está funcionando.
Sei que no momento de gerar o backup e/ou salvar pode acontecer um erro. Gostaria de enviar esse erro para um e-mail ou gerar algum log de erro, alguma coisa assim. Alguém sabe como fazer isso?
Segue o script:
#!/bin/bash
# bkpDBData.sh
#
# gera um dump da base de dados, somente dos dados para um zip no servidor
#

# echo 'Iniciando backup SGI!';

local_save_backup=/local/salvar/backups
ano=`date +%Y`
path_complete=$local_save_backup'/'$ano

# verifica se existe a pasta do local_save_backup+ano
# se nao existe, entao cria a pasta
if [ ! -d $path_complete ]; then
    # echo "criar pasta";
    mkdir $path_complete
    # echo "pasta criada";
fi

mysqldump -h localhost -u user -psenha -x -e -t nome_da_base | gzip > $path_complete/`date +%Y%m%d_%H%M`_mysql.gz;

# echo 'Fim do backup SGI!';

Answer: Quaisquer mensagens de erro serão enviadas para o stderr, que é o output padrão de erros do shell. O strerr é identificado pelo número 2 quando se quer fazer alguma ação com esses erros.
Assim, suponhamos que sua cron faz uma chamada do tipo:
bkpDBData.sh

Vamos alterar essa chamada para gravar os erros em um log:
bkpDBData.sh 2>/var/log/bkdbdata.log

Em seguida, você pode programar uma cron para pegar as mensagens do log de 1 em 1 minuto e enviá-las para um email:
mail -s "bkpdbdata" firstname.lastname@example.com < /var/log/bkpdata.log

Comment: Muito bom, algo que não sabia e vou estudar sobre. Já fiz o teste e gerou um erro (no caso coloquei uma senha errada para poder gerar). Eu não tinha a ideia se era no cron ou no shell que eu tinha que fazer essa verificação, mas agora já me abriu mais a mente para isso.

Shell enviar mensagem de erro


validate form field for null or empty using javascript

Question: I am using the following javascript code to check for nothing being entered in a form and it is not working. Can anyone suggest a reason?
           function validateForm()
{
    var a=document.getElementById("quiz_01").value;
    $question = a;
    if (a=="" || a==null) {
      alert("Question 201 must be filled out");
      form1.quiz_01.focus();
      return false;
      }
}

An extract of the html is as follows:
<form id="form1" name="form1" method="post" action=""  onsubmit="return validateForm()">

            201 A one followed by 100 zeros is a Googol

            <select name="quiz_01" id="quiz_01">
                <option value=" "> </option>
                <option value="100">100</option>
                <option value="90">90</option>
                <option value="80">80</option>
                <option value="70">70</option>
                <option value="60">60</option>
                <option value="50">50</option>
                <option value="40">40</option>
                <option value="30">30</option>
                <option value="20">20</option>
                <option value="10">10</option>
                <option value="0">0</option>
            </select>

    <input type="submit" name="next" value="Next >">

Answer: The first option:
<option value=" "> </option>

Has a value that is a single space, not an empty string. So either change the value or change the JS code. Note also that the .value property will never be null so you don't need to test for that.
if (a == " ") {

Demo: http://jsfiddle.net/RyN5W/
Comment: Thanks very much. I changed to " "  and "if (a == " ") {" and that worked. The comments of subhash and mohkahn were also relevant, thanks.
Answer: Trim the whitespaces.
 var a=document.getElementById("quiz_01").value.trim();

Answer: change
if (a=="" || a==null)

to
if (a==" " || a==null)

Answer: You should be checking for " " as per your HTML
if (a == " ") {

Answer: change the code to
var e = document.getElementById("quiz_01");
    var ev = e.options[e.selectedIndex].text;

if(ev===' '){
alert('question 201 is a required field');

}

you should use === not == whwn comparing strings
Answer: You could not access the value directly when working with an <select> element.
use following:
var element = document.getElementById("quiz_01");
var a = element.options[element.selecedIndex];

EDIT: (credits to nnnnnn)
In modern browsers the direct access will work, but if you're targeting an old Version (I think especially of Internet Explorer ;) ) the showed approach will gain you compatibility.
Comment: That was true in certain old browsers (except you want `element.selectedIndex` not `element.selecedValue`), but I think you'll find every modern browser lets you use `.value` directly on the select element to get the selected item.


Parse column value in SQL Server

Question: I have a table with a Varchar column with the value as mentioned below. How can i parse this string into columns
This string in a column
<item id="1" value="1"><item id="2" value="6"><item id="4" value="8">

into
Item ID  value
--------------
1        1
2        6
4        8

Comment: If you change your column to `xml` then you can trivially achieve this with xQuery.
Comment: While asking a question, you need to provide a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example):
(1) DDL and sample data population, i.e. CREATE table(s) plus INSERT T-SQL statements.
(2) What you need to do, i.e. logic and your code attempt implementation of it in T-SQL.
(3) Desired output, based on the sample data in the #1 above.
(4) Your SQL Server version (SELECT @@version;).
Answer: Your data resembles XML fragment. And it is not well-formed.
All what you need is to compose XML on the fly. After that XQuery makes it very easy (as @Larnu pointed out it already). No need to parse anything.
SQL
-- DDL and sample data population, start
DECLARE @tbl TABLE (ID INT IDENTITY PRIMARY KEY, tokens NVARCHAR(MAX));
INSERT @tbl (tokens) VALUES
(N'<item id="1" value="1"><item id="2" value="6"><item id="4" value="8">');
-- DDL and sample data population, end

SELECT c.value('@id', 'INT') AS id
    , c.value('@value', 'INT') AS [value]
FROM @tbl t
CROSS APPLY (SELECT TRY_CAST('<root>' +
    REPLACE(tokens, '>', '/>') +
    '</root>' AS XML)) AS t1(x)
CROSS APPLY x.nodes('/root/item') AS t2(c);

Output

id
value

1
1

2
6

4
8

Parse column value in SQL Server


Sum binomial distribution not binominally distributed if the p is not equal for all

Question: Suppose we have $X_1, ... X_n$ independent with $X_i \sim bin(n_i, p_i)$, prove that $Y := X_1 + ... + X_n$ is not binomially distributed.
My attempt: suppose $Y$ is binomially distributed, then it must be $bin(\sum_{i = 1}^nn_i, q)$ for some $q$ because it is possible that $Y = \sum_{i = 1}^nn_i$, but not $Y > \sum_{i = 1}^nn_i$. Then we have that $$E[Y] = \sum_{i = 1}^nn_iq$$ but also $$E[Y] = \sum_{i = 1}^nE[X_i] = \sum_{i = 1}^nn_iq_i$$ and $$Var(Y) = \sum_{i = 1}^nn_iq(1 - q)$$, but also, as the $X_i$ are iid: $$Var(Y) = \sum_{i = 1}^nn_iq_i(1 - q_i)$$
I'm trying to show that this implies that $q_i - q = 0$ for all $i$. From the expected values we know that $\sum_{i = 1}^nn_i(q - q_i) = 0$, but this doesn't imply that $q_i - q = 0$. I also tried substituting them in each other, but that didn't work out either. I'm not sure whether this will work, because it's possible that $Y$ satisfies these equations but is not binomially distributed.
I have reduced the information to the two equations $$\sum_{i = 1}^nn_i(q - q_i) = 0$$ and $$\sum_{i = 1}^nn_i(q^2 - q_i^2) = 0$$
Comment: You'll need to impose some conditions on the $p_i$ because if $p_1=p_2=\cdots=p_n$, then the sum does have a binomial distribution.
Answer: This is an instance of Jensens inequality: you can view $$\sum \frac{n_i}{\overline{n}} q_i = \mathbb{E}_{\mu}[f]$$ where $\overline{n} = \sum_i n_i$. I leave it to you to figure out what $f$ and $\mu$ are.
Then your first equation gives you $$ \mathbb{E}_{\mu}[f]=q$$ while the second reads $$ \mathbb{E}_{\mu}[f^2]=q^2.$$
Now Jensens inequality is strict unless the function you are moving under the integral sign is affine. The only way a square can be affine is that the set of points it is defined on is a singleton, i.e. $q_i=q$ for any i.
Comment: What does the $\mu$ in $\mathbb{E}_\mu[f]$ mean?
Comment: It's the measure w.r.t. which we take the mean. I put the index in order to differentiate it from the mean you consider in your question.


Appending to an object literal

Question: I pass an object literal to a function like this:
pre( {model:'MUserTry'} );

I then wish to augment it, but I end up writing over it like this.
    pre : function( o_p ) {
        o_p = {
            result: '0',
            page  : {}
        };

model is now gone as o_p is completely new
What is the best way to append object properties.
Do you have to explicitly define them now.
o_p.result = 0;
o_p.page = {};

or is there a function that will allow me to write out the object literal and combine them?
kind of like this
object1.push (object2 );

Comment: If you want to combine them http://stackoverflow.com/questions/171251/how-can-i-merge-properties-of-two-javascript-objects-dynamically
Answer: To add new object properties or methods, use the dot operator:
obj.dog = "woof";

This is also equivalent:
obj["dog"] = "woof";

Answer: You can just add properties by assigning to them:
o_p.result = '0';

This won't create a new object and thus retain your model property.
Answer: In your code the local variable o_p is redefined by the assigning:
function( o_p ) {
    // assign a new value to local variable o_p
    // object passed as an argument is untouched
    o_p = {
        result: '0',
        page  : {}
    };
}

You could merge your objects like this :
function( o_p ) {
    var mergee = {
        result: '0',
        page  : {}
    };
    for (var attrname in mergee) { o_p[attrname] = mergee[attrname]; }
}

Alternatively, if you can use JQuery, you can just use the extend method:
function( o_p ) {
    var mergee = {
        result: '0',
        page  : {}
    };
    $.extend(o_p, mergee);
}

Comment: This question is also addressed more extensively [here](http://stackoverflow.com/questions/171251/how-can-i-merge-properties-of-two-javascript-objects-dynamically?lq=1)

Appending to an object literal


How to retrieve a value from an executed string?

Question: I didn't quite know how to word my question as this isn't something one would often do.
Here's my situation:
I have a list of values.
Values= ["1","2","3","4"]

I then want to be able to separately access each of the values and combine them into a string with commas in between them. I wrote a bit of code that I want to format a string ('{}'.format())in such a way so that the outcome looks like this: '1,2,3,4'. My code that I have so far (it is also the code structure that I'd prefer to use if possible) is the following:
string = "{}".format(exec(
    "Out= ''\n"
    "for value in Values:\n"
    "    Out+= value + ','\n"
    "Out"))
print(string)

If, within the string that is given as the parameter to the 'exec' function, I write "print(Out)"
IE:
string = "{}".format(exec(
    "Out= ''\n"
    "for value in Values:\n"
    "    Out+= value + ','\n"
    "print(Out)"))

Then the program will properly run and print out "1,2,3,4", but it will not store it in 'string' (Which is to be expected)
I therefore thought of adding a function in order to make sure that it can return something. I wrote the following, but as expected it also didn't work. string is still None:
string = "{}".format(exec(
    "def do_stuff():\n"
    "    Out = ''\n"
    "    for value in Values :\n"
    "       Out += value + ','\n"
    "    return Out \n"
    "do_stuff()"))

Now, I do realise that this method of combining strings is nowhere near the preferred way to combine strings, but let me give you some context.
I am making a game in python. There is a Character class and it contains various attributes of various data types (such as dict, list, int, str, float.... as well as some custom ones: AI, Item, ActiveSkill, PassiveSkill, Effect, Weapon, Armour etc...)
Now, my plan is to be able to store the game progress in a json tree. For that I started giving each of these entities a method which will convert all of its attributes into json trees stored as strings. That way, when I want to save the game and it comes to the Character, it will sort all of its attributes, but it'll also have to make a call to objects that it's referencing (such as ActiveSkills). For each of ActiveSkills in a list, it'll have to call for their own json syntaxed strings and combine these using commas.
Here's some of my code to help you understand what I want to do:
json += \
    '"ActiveSkills":[{oActiveSkills}],' \
    '"PassiveSkills":[{oPassiveSkills}].' \
    '"Effects":[{oEffects}],'.format(
        oActiveSkills=exec(
            "Skills = ''\n"
            "for skill in self.ActiveSkills:\n"    # ActiveSkills is a list of ActiveSkill objects
            "    Skills+=skill.encode_to_json() + ','\n"
            "return Skills"),
        oPassiveSkills=exec(
            "Skills = ''\n"
            "for skill in self.PassiveSkills:\n"    # PassiveSkills is a list of PassiveSkill objects
            "    Skills+=skill.encode_to_json() + ','\n"
            "return Skills"),
        oEffects=exec(
            "Effects = ''\n"
            "for effect in self.Effects:\n"    # Effects is a list of Effect objects
            "    Effects+=effect.encode_to_json() + ','\n"
            "return Effects"))

I know that this code currently doesn't run, as you can only return from within a function, but I really don't know how to go about it in a quick and easy way.
One solution that I have saved up as a last resort is to just do everything by hand. IE: Have some kind of for loop that creates a string from returned values. Add this string to the json string, after which you will manually open and close with "},{" and make sure that there isn't a "," at the end. Then repeat this for PassiveSkills and Effects also. Finally, close the json string with "}," and add it to the 'json' (string type) variable...
I want to minimise the space taken up by this, and I want this system to be flexible, so that no matter how many skills in a list that you have, you could continue adding more and more of these json strings without changing the code.
Comment: Take a look at [`str.join()`](https://docs.python.org/3/library/stdtypes.html#str.join).
Comment: When a list is passed into JSON. One does need to worry about commas. Also when you extract the list back, you do not need to worry about the commas either. JSON worries about it for you. Unless I am getting the idea wrong here....
Comment: @El'endiaStarman
Yes, but later on I present a situation in which I have a list of objects, from which I call on a method that returns a value. Should I then create a new list? IE: `Skills=[i.encode_to_json: for i in ActiveSkills]` followed by, as you suggested `json += ",".join(Skills)`?
Comment: Your explanation still doesn't make clear why you're using `exec` at all.  Just take whatever you're doing in the exec-ed string, move it out into a real function, call it, and store the result.
Comment: Yes, don't use exec - see my answer about using json or pickle.
Answer: To join a list using ',':
Values = ["1","2","3","4"]
result = ",".join( Values )
print(result)

More info about str.join(iterable)can be found here in the online documentation.
For a better solution to store dictionaries and lists (and, using pickle, objects), you should look at the json and pickle packages in the Python Standard Library.
For example, with json:
import json

mydata = {}
mydata['hello'] = { 'value1':1, 'valuuelist':[1,2,3,4] }

print(json.dumps(mydata))

Output:
{"hello": {"valuuelist": [1, 2, 3, 4], "value1": 1}}

So, don't roll your own encoder, unless you really have to.
Comment: The problem is that the data I want to save in a json file is scattered across various objects (EG: Character has its own attributes, but so do skills of the character.) To store all of the data as a single tree I require each object to create a string in json syntax that will then be combined with any objects that are referenced by that object (Each object is referenced by only one other object. That way I have a hierarchy and I'm trying to backwards engineer the json tree.)
Comment: I can imagine (though I can't point at an implementation) that the main objects could have registered all the child objects, and could provide a way to load/restore all the data by calling the registered child objects 'pickle'. This is probably an important part of your design, difficult to retrofit later. See e.g. http://stackoverflow.com/questions/4529815/saving-an-object-data-persistence-in-python
Comment: I have used pickle before and my biggest problem was that I didn't understand _how_ it encoded the data. As a result, it would be difficult for me to then load up the data if the system has been updated, since I didn't know how to pick out the required data out of a pickled object, without loading up the previous version of the object in one swoop.
I decided to go with json as my data storage, as it is quite simple to load up the data, even if the process (method) that uses the data has changed. I have designed it to make it as easy to retrofit as I could... But I might have missed something.
Comment: You use pickle.load() to unpickle - don't do it yourself! To quote the docs: "The data format used by pickle is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as XDR (which can't represent pointer sharing); however it means that non-Python programs may not be able to reconstruct pickled Python objects. By default, the pickle data format uses a printable ASCII representation."
Comment: I could unpickle the data easily. The problem was then loading the old data into a new system that now has changed. The unpickled data is no longer compatible with the new updated system. With json, however, I could pick out the data that would be relevant to the new system without worrying about compatibility, as I could manipulate the data in order to make it compatible.
Comment: I understand what you mean about loading after updating the code - if you really need to solve this problem you will probably end up have different versions of the class always available to restore into, and understanding how to transform themselves into newer versions. However the same basic problem exists with using json - if the latest implementation of the class has new capabilities, and you need to be able to restore older versions, you will have to implement the old->new transformation yourself.
Comment: Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/96194/discussion-between-alexz-and-barny).
Answer: For joining each element of a sequence into a string you can use join method.
   L=['1','2','3']
   string=",".join(L)
   print(L)


MySQL Change varchar column to type set

Question: I have a table that has data stored in a varchar column:
field varchar :company
example data  : AB
                SD
                AB,SD
                SD,AB

The values are only supposed to be one of those combinations (the order does not matter). This column is probably better suited to be type set but it's varchar. I want to convert it to type set with the set's values being AB and SD.
I have a new column called companyFixed with possible values AB, SD.  How can I copy the values from the original column into the new proper column?
UPDATE myTable
SET companyFixed = <this row's company values>

Answer: Maybe I haven't understood the question, but as far as I can tell this should work:
UPDATE myTable SET companyFixed = company;

I tried it myself, creating a table with a varchar field, then adding a SET field, and updating from one to the other. It worked.
Comment: You didn't miss anything that worked exactly as I needed! I didn't expect it to be so simple.

MySQL Change varchar column to type set


Question about strongly convexity and affinity

Question: For a function $f$, it is said to be strongly convex if for all $x,y$
\begin{equation}
 (\nabla f(x) - \nabla f(y) )^T (x-y) \ge m \|x-y\|_2^2
\end{equation}
for a constant $m \ge 0$
Is it called strongly affine if
\begin{equation}
 (\nabla f(x) - \nabla f(y) )^T (x-y) = m \|x-y\|_2^2 \text{    ?}
\end{equation}
In addition as we know if a function is strongly convex it satisfy
\begin{equation}
    f(y) \ge f(x) + \nabla f(x)^T (y-x) + \frac{m}{2} \|y-x\|_2^2
\end{equation}
For strongly affine function can I say
\begin{equation}
    f(y) = f(x) + \nabla f(x)^T (y-x) + \frac{m}{2} \|y-x\|_2^2 \text{    ?}
\end{equation}
Comment: I've never heard the term "strongly affine" and I don't know what it would mean.  But if it were a term, it would seem like a "strongly affine" function should be, in particular, an affine function.  This would prevent the equations mentioned from being satisfied.
Answer: I never heard about "strongly affine" functions. However (as suggested by @LittleO to show that a "strongly affine" function is not necessarily strongly convex and strongly concave) consider the function
$$f(x)= \frac{1}{2}\|x\|_2^2 \quad \Longrightarrow \quad \nabla f(x) = x.$$
Then
$$(\nabla f(x)-\nabla f(y))^T(x-y) = \|x-y\|_2^2,$$
but
$$\begin{array}{rcl}f(x) + \nabla f(x)^T(y-x) + \frac{1}{2}\|y-x\|_2^2 &=& \frac{1}{2}(\|x\|_2^2-\|x\|_2^2+x^Ty+\|x\|_2^2+\|y\|_2^2-2x^Ty)\\
&=& \frac{1}{2}(\|x\|^2+\|y\|^2-x^Ty) \\ &\neq & \frac{1}{2}\|y\|_2^2.\end{array}
$$
Comment: Let $f(x) = \frac12 \|x \|^2$, so that $\nabla f(x) = x$.  Then $(\nabla f(x) - \nabla f(y))^T(x - y) = \|x - y\|^2$.  However, $-f$ is not convex.
Comment: @littleO Thanks for your comment, I edited my answer


Como fazer um INNER JOIN em uma unica tabela com 3 chaves?

Question: Tenho um tabela que possui a seguinte estrutura:
origem|papel|campo018|campo040|mercado|Master_key|

A coluna Master_key pode ter dois valores: 'NY' OU 'MS', gostaria de comparar as linhas em que Master_key é igual a 'NY' com as que são 'MS', as colunas origem, papel e mercado são chaves.
Tentei desta forma, porém não funcionou:
SELECT ny.*, ms.* FROM z_quotes ny
INNER JOIN z_quotes ms ON ny.origem = ms.origem
INNER JOIN z_quotes ms ON ny.papel = ms.papel
INNER JOIN z_quotes ms ON ny.mercado = ms.mercado
WHERE (ny.Master_key = 'NY') AND (ms.Master_key = 'MS');

Por exemplo, se eu tenho a tabela abaixo:
Eu quero que retorne desta forma:
Comment: Não entendi o que você quer mas com certeza não é com `INNER JOIN`, ele serve para fazer consultas unindo tabelas diferentes. Poderia dar um exemplo de saída esperada?
Comment: Adicionei um exemplo na pergunta
Comment: A relação entre as duas linhas é se todas as colunas forem iguais porém `Master_key` diferente?
Comment: @GuilhermeCostamilam  `Master_key` só pode ter dois resultados, por isso tem um Where no final da query
Answer: Pelo que entendi, basta um JOIN:
SELECT ny.*, ms.* FROM z_quotes ny
INNER JOIN z_quotes ms ON ny.origem = ms.origem
  AND ny.papel = ms.papel
  AND ny.mercado = ms.mercado
WHERE (ny.Master_key = 'NY') AND (ms.Master_key = 'MS');


javascript: can't find data inside JSON response

Question: I want to create a bot on twitter using the twitter and the spotify APIs. So far I have gotten a good start, but now I've come across a problem I can't solve on my own:
I need to know the name of the artist and the song of the top 1 of the top 50 Spotify songs. So I sent a Spotify API request and I get this json response:
{
  "href" : "https://api.spotify.com/v1/playlists/37i9dQZEVXbIPWwFssbupI/tracks?offset=0&limit=1",
  "items" : [ {
    "added_at" : "1970-01-01T00:00:00Z",
    "added_by" : {
      "external_urls" : {
        "spotify" : "https://open.spotify.com/user/"
      },
      "href" : "https://api.spotify.com/v1/users/",
      "id" : "",
      "type" : "user",
      "uri" : "spotify:user:"
    },
    "is_local" : false,
    "primary_color" : null,
    "track" : {
      "album" : {
        "album_type" : "single",
        "artists" : [ {
          "external_urls" : {
            "spotify" : "https://open.spotify.com/artist/7IlRNXHjoOCgEAWN5qYksg"
          },
          "href" : "https://api.spotify.com/v1/artists/7IlRNXHjoOCgEAWN5qYksg",
          "id" : "7IlRNXHjoOCgEAWN5qYksg",
          "name" : "Aya Nakamura",
          "type" : "artist",
          "uri" : "spotify:artist:7IlRNXHjoOCgEAWN5qYksg"
        } ],
        "available_markets" : [ "AD", "AE", "AL", "AR", "AT", "AU", "BA", "BE", "BG", "BH", "BO", "BR", "BY", "CA", "CH", "CL", "CO", "CR", "CY", "CZ", "DE",
"DK", "DO", "DZ", "EC", "EE", "EG", "ES", "FI", "FR", "GB", "GR", "GT", "HK", "HN", "HR", "HU", "ID", "IE", "IL", "IN", "IS", "IT", "JO", "JP", "KW", "KZ", "LB", "LI", "LT", "LU", "LV", "MA", "MC", "MD", "ME", "MK", "MT", "MX", "MY", "NI", "NL", "NO", "NZ", "OM", "PA", "PE", "PH", "PL", "PS", "PT", "PY", "QA", "RO", "RS", "RU", "SA", "SE", "SG", "SI", "SK", "SV", "TH", "TN", "TR", "TW", "UA", "US", "UY", "VN", "XK", "ZA" ],
        "external_urls" : {
          "spotify" : "https://open.spotify.com/album/43sqDUJkvHHK0ikoAN4JAS"
        },
        "href" : "https://api.spotify.com/v1/albums/43sqDUJkvHHK0ikoAN4JAS",
        "id" : "43sqDUJkvHHK0ikoAN4JAS",
        "images" : [ {
          "height" : 640,
          "url" : "https://i.scdn.co/image/ab67616d0000b273e2e7c5322770c06bdfead61f",
          "width" : 640
        }, {
          "height" : 300,
          "url" : "https://i.scdn.co/image/ab67616d00001e02e2e7c5322770c06bdfead61f",
          "width" : 300
        }, {
          "height" : 64,
          "url" : "https://i.scdn.co/image/ab67616d00004851e2e7c5322770c06bdfead61f",
          "width" : 64
        } ],
        "name" : "Jolie nana",
        "release_date" : "2020-07-17",
        "release_date_precision" : "day",
        "total_tracks" : 1,
        "type" : "album",
        "uri" : "spotify:album:43sqDUJkvHHK0ikoAN4JAS"
      },
      "artists" : [ {
        "external_urls" : {
          "spotify" : "https://open.spotify.com/artist/7IlRNXHjoOCgEAWN5qYksg"
        },
        "href" : "https://api.spotify.com/v1/artists/7IlRNXHjoOCgEAWN5qYksg",
        "id" : "7IlRNXHjoOCgEAWN5qYksg",
        "name" : "Aya Nakamura",
        "type" : "artist",
        "uri" : "spotify:artist:7IlRNXHjoOCgEAWN5qYksg"
      } ],
      "available_markets" : [ "AD", "AE", "AL", "AR", "AT", "AU", "BA", "BE", "BG", "BH", "BO", "BR", "BY", "CA", "CH", "CL", "CO", "CR", "CY", "CZ", "DE", "DK", "DO", "DZ", "EC", "EE", "EG", "ES", "FI", "FR", "GB", "GR", "GT", "HK", "HN", "HR", "HU", "ID", "IE", "IL", "IN", "IS", "IT", "JO", "JP", "KW", "KZ", "LB", "LI", "LT", "LU", "LV", "MA", "MC", "MD", "ME", "MK", "MT", "MX", "MY", "NI", "NL", "NO", "NZ", "OM", "PA", "PE", "PH", "PL", "PS", "PT", "PY", "QA", "RO",
"RS", "RU", "SA", "SE", "SG", "SI", "SK", "SV", "TH", "TN", "TR", "TW", "UA", "US", "UY", "VN", "XK", "ZA" ],
      "disc_number" : 1,
      "duration_ms" : 147076,
      "episode" : false,
      "explicit" : false,
      "external_ids" : {
        "isrc" : "FRZ042000666"
      },
      "external_urls" : {
        "spotify" : "https://open.spotify.com/track/34FbkFgAbv4ffvhL2XQMVm"
      },
      "href" : "https://api.spotify.com/v1/tracks/34FbkFgAbv4ffvhL2XQMVm",
      "id" : "34FbkFgAbv4ffvhL2XQMVm",
      "is_local" : false,
      "name" : "Jolie nana",
      "popularity" : 83,
      "preview_url" : "https://p.scdn.co/mp3-preview/f38309ea40d6b20b4a7a6b72209b503342b1b147?cid=8f22024a4f144cd7b965b9db6e124e94",
      "track" : true,
      "track_number" : 1,
      "type" : "track",
      "uri" : "spotify:track:34FbkFgAbv4ffvhL2XQMVm"
    },
    "video_thumbnail" : {
      "url" : null
    }
  } ],
  "limit" : 1,
  "next" : "https://api.spotify.com/v1/playlists/37i9dQZEVXbIPWwFssbupI/tracks?offset=1&limit=1",
  "offset" : 0,
  "previous" : null,
  "total" : 50
}

I put that response in the variable myJSON, and I want to create a variable who contains only Aya Nakamura and another that contains only Jolie Nana
I've already tried to retrieve the data using myJSON.items and myJSON[items] but both of those lines return undefined
I hope I've explained that well.
Thank you for the help, and please excuse me for my English, as I am a French speaker.
Comment: can you explain your question more ? what is the output you want, and if there more details you can provide
Comment: the output i want is "name" : "Aya Nakamura" and "name" : "Jolie nana",
Comment: can you explain the object shape you want as output
Comment: Is this a JSON string, or is this parsed json? You can check this with something like `console.log(typeof myJSON)`. If that prints `object` then it is parsed, if it prints `string` then you need to parse it first with `JSON.parse(myJSON)`. Also, it would be helpful if you made an example of your code using the [code snippet tool](https://meta.stackoverflow.com/questions/269753/feedback-requested-runnable-code-snippets-in-questions-and-answers).
Comment: what you post is not a valid json, i tried to parse it with json parser but it failed http://json.parser.online.fr/
Comment: are you sure ? i parse in my app with JSON.parse and it's work great
Comment: okay i parse it and all works now thanks for you help
Comment: yes it's normal i make a mistake if you parse with this code that will works
Answer: You are not parsing the json, so your data type is not an object as you suspect, but rather a string. Additionally, you can access properties of strings the same as you would if it was parsed, but they return undefined, not the data you want. Here's some examples:

const data = `
{"foo": "bar"}
`
console.log('data:', typeof data, data)

const parsedData = JSON.parse(data)

// Notice how similar this looks to the other in the console.
// It is not always clear what the type is if you don't check it,
// since JSON strings look similar to JS objects
console.log('parsedData:', typeof parsedData, parsedData)

// accessing an item:

console.log('data.foo:', data.foo) // since this is a string, it is `undefined`
console.log('parsedData.foo:', parsedData.foo) // since this is a parsed object, it works


Checking ifstream, doesn't work after an error?

Question: I've attached my code below.  If I remove the if statement to check if the file is opened, this sequence will work.  Why won't it work with the if statement?  Also, it works as written below if I supply the correct file name the first time.  It only hangs up if I enter an incorrect file name first.
Thanks for the help!
ifstream inputFile(fileName.c_str());

if(!inputFile)
{
    cout << "Unable to locate input file, please ensure it is in the working directory"
         << endl;
    cout << "Enter the name of your input file (ex. input.txt):  ";
    cin >> fileName;
    cout << endl;

    ifstream inputFile(fileName.c_str());
}
else
{
    cout << "Input file opened successfully!" << endl;
}

Comment: If the user mistypes the file name, I would like to allow them to reenter.
Answer: The code you show is perfectly legal, so I suppose that you use inputFile after this "bad filename" logic:
ifstream inputFile(fileName.c_str());

if(!inputFile)
{
    cout << "Unable to locate input file, please ensure it is in the working directory"
         << endl;
    cout << "Enter the name of your input file (ex. input.txt):  ";
    cin >> fileName;
    cout << endl;

    ifstream inputFile(fileName.c_str());
}
else
{
    cout << "Input file opened successfully!" << endl;
}
// USING inputFile here

The problem with that is, that you still have the original inputFile here. The inputFile inside of the if statement is a new std::ifstream. It may be easier to see if you use a different name:
ifstream inputFile(fileName.c_str());

if(!inputFile)
{
    cout << "Unable to locate input file, please ensure it is in the working directory"
         << endl;
    cout << "Enter the name of your input file (ex. input.txt):  ";
    cin >> fileName;
    cout << endl;

    ifstream differentInputFile(fileName.c_str()); //HERE
}
else
{
    cout << "Input file opened successfully!" << endl;
}

The correct way to close the bad file and reopen with the correct file name is:
inputFile.close();
inputFile.open(fileName.c_str());

The complete code then becomes
ifstream inputFile(fileName.c_str());

if(!inputFile)
{
    cout << "Unable to locate input file, please ensure it is in the working directory"
         << endl;
    cout << "Enter the name of your input file (ex. input.txt):  ";
    cin >> fileName;
    cout << endl;

    inputFile.close();
    inputFile.open(fileName.c_str());
}
else
{
    cout << "Input file opened successfully!" << endl;
}

It's also advisable to enable warnings. My recommendation is to use -Wall -Wextra -Wshadow -pedantic -Wfatal-errors (that's for gcc and clang).
Comment: Since the filestream was never opened, it makes more sense to `.clear()` it than to `.close()` it.
Comment: @MooingDuck `ifstream inputFile(fileName.c_str());`  opens the stream, doesn't it? http://en.cppreference.com/w/cpp/io/basic_ifstream/basic_ifstream
Comment: Thank you!  Sometimes it's the little things.
Comment: @stefan: It _attempts_ to open the stream.  If it fails, the `failbit` is set.  If the `failbit` is set, then `if(!inputFile)` evaluates to `true`, causing it to enter the block.


Novice trying to learn some music theory from an old book. Have I answered this correctly?

Question: Good day. I'm trying to learn how to read notes from an old music theory book I borrowed from a friend.
I need to answer some questions on this paper: https://i.stack.imgur.com/6VWmL.jpg
I can't find the correct answers nowhere in the book, so I hope you can help me.

What chord is it on 10? My answer: E minor

What chord is it on 11? My answer: A minor

What chord is it on 12? My answer: D7

What chord is it on 13? My answer: G major

I also need to find in what inversions they're in.
How do I do that? I don't quite understand how.
Thank you and sorry if you don't understand my questions. English is not my native language.
Comment: What is the old book?
Comment: It's a swedish book: https://www.adlibris.com/se/bok/elementar-musikteori-9789177481645
Answer: Well done! All correct. You've worked out basic triads, and added m7 to the D making D7.
It's a bit odd, but we name inversions of chords using the lowest note. If it's root, then the chord's root position. 1st inversion has 3 underneath, 2nd has 5 at the bottom, etc. It really doesn't make a difference what other of the chord's notes are above, or what order, it's simply the lowest note that determines.
Comment: One could argue that the E at 10 is a non-chord tone and that the chord is therefore G.


Surjective Functions

Question:
Possible Duplicate:
Surjectivity of Function Compositions

I am trying to prove this statement:
If $f: A \rightarrow B$  and $g: B \rightarrow C$ are both surjective functions, show that $g \circ f : A \rightarrow C$ is also a surjective function.
I know that some elements B have corresponding elements in A and likewise, some elements in C have corresponding elements in B. Is it enough to say that some elements in C must therefore correspond in A, which is what surjective function, $g \circ f : A \rightarrow C$, would show?
Comment: Start by looking up the definition of "surjective".
Comment: yes, it is an exact duplicate. Sorry I did not see the previous question. feel free to close this or whatever needs to be done
Comment: @dtldarek sorry I am still new to this website, how can I change that?
Comment: @DominickGerard I guess you already know ;-) Great!
Answer: $g$ being surjective means that EVERY element of $C$ is mapped to by (at least) one element of $B$ by $g$, and similarly for $f$. It should now be fairly obvious that if both $g$ and $f$ are surjective then their composition must also be surjective.
Answer: Hint:
Consider three sets of people $A$, $B$, $C$. Assume that for every person $c_1 \in C$ there exists a person $b_1 \in B$ that knows $c_1$, and that for every person $b_2 \in B$ ($b_1 = b_2$ might or might not be true) there exists $a_1 \in A$ that knows $b_2$. Can you conclude that for every $c_2 \in C$ there exists $a_2 \in A$ that (indirectly) knows $c_2$?

Surjective Functions


game coloring points rioplatense olympiad 1999

Question: This problem is from the Rioplatense MO 1999/3 L3. The question is the following:
Two players $A$ and $B$ play the following game:
$A$ chooses a point, with integer coordinates, on the plane and colors it green, then $B$ chooses $10$ points of integer coordinates, not yet colored, and colors them yellow. The game always continues with the same rules; $A$ and $B$ choose one and ten uncolored points and color them green and yellow, respectively.
a) The goal of $A$ is to achieve $111^2$ green points that are the intersections of $111$ horizontal lines and $111$ vertical lines (parallel to the coordinate axes). $B$'s goal is to stop him. Determine which of the two players has a strategy that ensures they achieve their goal.
b) The goal of $A$ is to achieve $4$ green points that are the vertices of a square with sides parallel to the coordinate axes. $B$'s goal is to stop him. Determine which of the two players has a strategy that ensures they achieve their goal.
I'm having trouble with part b).
For part a), A has the winning strategy. The winning strategy is to place at least $111 \times 11 ^{110}$ green points in a horizontal line. The strategy is basically to form row by row (horizontally), and put a sufficient number of points so that it can form another horizontal row.
$111 \times 11 ^{110}$ is the minimum because if we need to place $f_i$ points in the $ith$ row, we need to place $10f_{i} + f_{i} = 11f_{i}$ points in the $i-1th$ row. Then it's easy too see that it's the minimum.
I am aware that it is not formally explained, if necessary I can write it formally and better explained, although I think this idea is quite intuitive.
For part b) I have no idea how to continue with the problem. I spent a long time with this part and I'm not even sure who has the winning strategy.
I found this problem in a Pigeonhole principle resource, so that may be useful.
Answer: Hint 1:

 If $A$ can win, their penultimate move must setup $11$ different squares so $B$ can't block them all. What would this look like?

Hint 2:

 The first part of the question is a hint for the second part.

Solution:

 First $A$ creates the configuration from a). Imagine a box around all the points that have already been colored by either player. Choose a 45-degree line outside this box. This will be the diagonal of the square. $A$ colors $11$ of the intersections of the horizontal lines from a) with this diagonal. ($B$ can't block this by coloring only $100$ additional points before $A$'s eleventh move.) Then, consider the reflections of the $111^2$ intersections from a) across the diagonal. There are $111$ horizontal lines passing through these points. Because $B$ has only colored $110$ additional points, there must be one with no yellow point. $A$'s next move is to color the intersection of this line with the diagonal. Then, there are $11$ ways $A$ can make a square on the next move playing on the same line and $B$ can't block them all: these are the intersections of that line with the vertical lines passing through each of $A$'s first $11$ points on the diagonal. (Those vertical lines are the reflections of the horizontal lines that was used to construct those points.)


Calculating mean of continuous time series

Question: I'm calculating the arithmetic mean of values in a continuous time series, weighted by duration. Points in the time series are not guaranteed to be evenly spaced.
Example data:
Time    Value
0       1
1000    2
2000    3
3000    4
5000    5

Where Time is the duration since the start of the time series.
My original approach to calculate the mean was to calculate the mean between adjacent points, assign a weight to that value based on the duration between the adjacent points and total duration, and sum the results:
Mean    Weight    Normalized weight   Result
1.5     1000      0.2                 0.3
2.5     1000      0.2                 0.5
3.5     1000      0.2                 0.7
4.5     2000      0.4                 1.8
                                      ---
                                      3.3

Resulting in a mean of 3.3
Thankfully I stumbled across the MATLAB documentation for calculating the mean of timeseries data, which suggests my method is probably incorrect.
Their method assigns weights as such:

Assign a weight to each point's value; first point is the duration of the first time interval t(2) - t(1), last point is the duration of the last time interval t(end) - t(end-1), and points that are neither the first or last points are the duration between the midpoint of the previous time interval and the midpoint of the next time interval.
Normalize the weighting for each time by dividing each weighting by the mean of all weightings.
Multiply the values for each point by its normalized weighting.

This results in the following:
Value   Weight   Normalized weight   Result
1       1000     0.153846            0.153846
2       1000     0.153846            0.307692
3       1000     0.153846            0.461538
4       1500     0.230769            0.923076
5       2000     0.307692            1.53846
        ----     --------            --------
        6500     1                   3.384612

Resulting in a mean of 3.384612
My question is: Why are the weights in the MATLAB documentation calculated in the way they are? Specifically, why are the first and last points assigned the full time interval between the next and previous points, respectively?
If the first and last points were only assigned half (i.e. to the midpoint), the resulting calculated mean is the same as my original approach. Is my original approach wrong?
Answer: Rather than being right or wrong, here Matlab assumes that your start and end samples also extend towards to their left and right by equal amounts. So, according to Matlab, your signal starts from t=-0.5 and ends at t=6. This is 0-th order interpolation technique, in which every sample extends to its left or right by equal amounts.
Comment: So, for time series' where data samples do not extend beyond the first and last point (i.e. the time series starts and ends exactly at those points), does my original method produces the correct result?
Comment: Yes, exactly. If you want try to draw your signal, integrate it, and divide by total duration.


Organic Groups - How to view articles in the context of the group

Question: I have OG setup on a site I am developing. The configuration is set to remove the main sites side bars when a Group is viewed. If an article that is only published in that group is viewed, it is opened in the context of the main Drupal site (All side bars are visible from the main site)
Is there a way to view the article so that it appears the user feels they are still in the group?
Comment: What mechanism did you use to remove the main site's sidebars? Block configuration, theming or something else?
Answer: You can adjust Group context detection at admin/config/group/context when you have Organic groups context enabled. This will add the appropriate css classes in the body element, which will allow you to theme the rest of the page.
Comment: Thanks for posting that admin URL for Context settings. I have another context-related issue and don't see a link to the OG context settings anywhere.


How to compare proportions by having only the mean proportion and 95%CI (reported by a meta-analysis)

Question: I have a proportion of 31.6% (24/76) and I need to test whether this proportion is significantly different from a proportion reported by a meta-analysis study equal to 59% (95%CI: 53%; 65%). The meta-analysis found that proportion and the 95%CI from linear mixed models, pooling data from several other studies.
I suppose I cannot use the classical methodology for testing two proportions using normal approximation and estimating the standard deviation as p(1-p). I believe that the 95%CI reported by the meta-analysis should be used somehow.
Any suggestions on how I can perform this comparison?
Thank you very much.
Answer: An estimate of the standard error of your estimate $\hat\pi_1$ from a sample of $n$ is given by
$$
\hat\sigma_{\hat\pi_1}=\sqrt{\frac{\hat\pi_1(1-\hat\pi_1)}{n}}
$$
Assuming that in the meta-analysis the upper & lower $1-\alpha$ confidence bounds $\pi^\mathrm{u}_2$ & $\pi^\mathrm{l}_2$ are calculated in reasonably conventional fashion, then an estimate of the standard error of their estimate $\hat\pi_2$ is given by
$$\hat\sigma_{\hat\pi_2} = \frac{\pi^\mathrm{u}_2 - \pi^\mathrm{l}_2}{2\Phi^{-1}\left(1-\frac{\alpha}{2}\right)}$$
where $\Phi$ is the standard Gaussian distribution function. A Wald-type test of the null hypothesis that the difference between the two proportions is zero seems the most straightforward approach: asymptotically, under the null
$\frac{\hat\pi_2-\hat\pi_1}{\sqrt{\hat\sigma_{\hat\pi_1}^2+\hat\sigma_{\hat\pi_2}^2}}\sim\mathcal{N}\left(0,1\right)$
test <- function(x1, n1, p2, lb, ub, alpha){
  p1 <- x1/n1
  v1 <- p1*(1-p1)/n1
  v2 <- ((ub - lb)/(2*qnorm(1-alpha/2)))^2
  se <- sqrt(v1 +v2)
  z <- (p2-p1)/se
  pv <- 2*min(pnorm(z), pnorm(-z))
  return(c(test.stat=z ,p.value=pv))
}

test(24, 76, 0.59, 0.53, 0.56 , 0.05)
#   test.stat      p.value
#5.090600e+00 3.569321e-07

test(12, 76, 0.20, 0.12, 0.29 , 0.05)
#test.stat   p.value
#0.6988176 0.4846661

Answer: You could use $x = 24$ successes in $n = 76$ trials to test $H_0: p = .53$ against $H_a: p < 0.53,$ where $0.53$ is the lower end of the CI from
meta analysis. In R, the relevant exact binomial test
is as shown below. The null hypothesis is strongly
rejected with P-value nearly $0.$
binom.test(24, 75, .53, alt = "less")

        Exact binomial test

data:  24 and 75
number of successes = 24, number of trials = 75,
 p-value = 0.0001944
alternative hypothesis:
 true probability of success is less than 0.53
95 percent confidence interval:
  0.0000000 0.4195974
sample estimates:
probability of success
                  0.32

There may be more elegant solutions, but your data
with $\hat p = 0.32$ are so far from matching the results of the meta
analysis that the decision is hardly in doubt. [Turns out that there is a "more elegant solution"; see @Scortchi's Answer.]
Even a 99.9% one sided CI for $p$ indicates your
point estimate $\hat p = 0.32$ is not consistent with $p > 0.5035,$ below $0.53.$
binom.test(24,75, .53, alt="less",
           conf.lev=.999)$conf.int
[1] 0.0000000 0.5034918
attr(,"conf.level")
[1] 0.999

Comment: Testing against a confidence limit is invalid: it does not achieve the desired test size (confidence level).
Comment: Thank you very much, Bruce. Indeed, for this specific example, the difference is quite obvious. However, I have other comparisons which are not the case, such as 15.6% (12/76) compared to 20% (95%CI: 12%; 29%). It would be perfect if I could obtain a p-value from this comparison. Best.
Comment: As you say, the answer's clear enough in this case, but your approach seems unnecessarily conservative.
Comment: @Scorchi: Thanks for better general method (+1).
Comment: @DiegoPujoni. Right. My method is OK for the specific case you give in your Question. where it's not a close call, but I hope you will Accept Scortchi's "more elegant method" and use it for all your cases.
Answer: Because some of my proportions are next to 0 or 1, the Wald-type test can give biased results. To overcome this issue, I simulated two beta distributions with specified parameters and applied a bootstraped difference test to obtain the p-value.
Beta.t.test <- function(x1, n1, med2, lb, ub, M, Alpha){
  #Function to estimate Beta parameters
  estBetaParams <- function(mu, var) {
    alpha <- ((1 - mu) / var - 1 / mu) * mu ^ 2
    beta <- alpha * (1 / mu - 1)
    return(params = list(alpha = alpha, beta = beta))
  }

  #Data from group 1: absolute frequency
  med1 <- x1/n1 ; var1 <- med1*(1-med1)/n1

  #Data from group 2: meta analysis result with 95%CI limits
  var2 <- ((ub - lb)/(2*qnorm(1-Alpha/2)))^2

  #Estimating beta parameters for both groups
  Beta1 <- estBetaParams(med1,var1)
  Beta2 <- estBetaParams(med2,var2)

  #Ploting
  X <- seq(0,1,0.001)
  B1 <- dbeta(X, Beta1$alpha, Beta1$beta)
  B2 <- dbeta(X, Beta2$alpha, Beta2$beta)
  plot(X,B1,type = "l"); lines(X,B2,col = 2)

  #Bootstrap test
  set.seed(1)
  RB1 <- rbeta(M, Beta1$alpha, Beta1$beta)
  RB2 <- rbeta(M, Beta2$alpha, Beta2$beta)
  if (med1 > med2){
    DifB <- RB1 - RB2
  } else {DifB <- RB2 - RB1}
  #Percentile 95% CI for the difference
  quantile(DifB, probs = c(0.025, 0.975))

  p.value <- 2*sum(DifB <= 0)/M ; p.value
  return(p.value)
}

When applied to the data far from 0 or 1, we get a very close p.value as reported by Scortchi
> test(12, 76, 0.20, 0.12, 0.29 , 0.05)
test.stat   p.value
0.6988176 0.4846661
> Beta.t.test(12,76,0.20,0.12,0.29,99999,0.05)
[1] 0.4823248

But with proportions near 0 or 1, the results can be a little distinct
> test(2, 50, 0.10, 0.07, 0.13 , 0.05)
test.stat   p.value
1.8952016 0.0580657
> Beta.t.test(2, 50, 0.10, 0.07, 0.13, 99999, 0.05)
[1] 0.09390094

Comment: The meta-analysis reported the mean proportion and both the upper and lower bounds of the 95%CI. So, as suggested by @Scortchi, I calculated the Standard Error for the mean proportion considering the Normal distribution. I guess I misused the term "Bootstrap" maybe Monte Carlo simulation would be better.
Comment: I am curious how you accomplished a bootstrap without knowing the sample size underlying the meta-analysis.


Is there a way to pass a composed key for a dictionary as argument of a function?

Question: I have a function that should use the value of the key that we pass as argument. The key would lookup the value in a nested dictionary. Ex:
nested_dict = ... # dump from json a very nested dictionary
def get_value(nested_dict, composed_key):
    # ? nested_dict[composed_key]

Is there a way to do this :
get_value(nested_dict, ["level1"]["level2"]...["leveln"])

Or :
get_value(nested_dict, "level1.level2...leveln")

?
How can I do?
Thanks
Answer: You can pass a list of the keys as second argument like this:
def get_value(nested_dict, composed_key):
  for x in composed_key:
    print(nested_dict[x])

And call like:
get_value(nested_dict, ["level1","level2",...,"leveln"])

Comment: Unfortunately not working


Quando usar listas e quando usar tuplas?

Question: Qual é a diferença entre os tipos list e tuple no Python e quando devo usar cada um?
Answer: Ambas são tipos de sequências de dados, mas uma é mutável e a outra imutável. Tanto lista como tupla são sequências de dados e possuem muitas propriedades comuns, mas a diferença básica é que a lista é mutável e a tupla é imutável.
O que é imutável?
Conforme o glossário da documentação da linguagem Python, imutável refere-se a um objeto com valor fixo. Objetos imutáveis incluem strings, números e tuplas. Caso seja necessário alterar o valor de um objeto imutável, outro objeto deve ser criado a fim de armazená-lo.  Eles desempenham um papel importante em locais onde um valor de hash constante é necessário, por exemplo como uma chave em um dicionário.
Na prática, isso implica em que não é possível fazer atribuições à um objeto imutável. Para uma lista, como o exemplo abaixo, é possível alterá-la através de uma operação de atribuição:
>>> lista = [0, 1, 2, 3]
>>> print(lista)
[0, 1, 2, 3]
>>> lista[0] = 9
>>> print(lista)
[9, 1, 2, 3]

Porém, repetir o exemplo utilizando uma tupla:
>>> tupla = (0, 1, 2, 3)
>>> print(tupla)
(0, 1, 2, 3)
>>> tupla[0] = 9
TypeError: 'tuple' object does not support item assignment

Um erro é disparado ao se tentar fazer uma atribuição a um objeto do tipo tupla.
Operações comuns às listas e tuplas
As operações listadas abaixo funcionam para s e t sendo tanto listas como tuplas.

Fonte: Built-In Types: Common Sequence Operations
E qual a diferença?
A única operação que tipos imutáveis implementam que os tipos mutáveis não suportam é a função hash(). Isto permite a tipos imutáveis, tal como a tupla, serem utilizados como chaves de dicionários e armazenadas em set e frozenset.
Portanto, é possível fazer:
>>> d = {
...     (1, 2): "Stack Overflow em Português"
... }
>>> print(d)
{(1, 2): 'Stack Overflow em Português'}

Pois, em memória, a chave do dicionário é relacionada ao valor hash da tupla. Sendo uma lista um tipo mutável, fazer o mesmo resulta em um erro indicando que a lista não suporta hash.
>>> d = {
...     [1, 2]: "Stack Overflow em Português"
... }
TypeError: unhashable type: 'list'

Cuidados
Embora a tupla seja um tipo imutável, se possuir um valor de um tipo mutável, o mesmo permanece mutável enquanto dentro da tupla. Por exemplo, se considerarmos uma tupla de listas:
>>> tupla = ([1, 2], [3, 4])

É possível alterar os valores da lista:
>>> tupla[0][0] = 9
>>> print(tupla)
([9, 2], [3, 4])

Ou mesmo chamar os métodos nativos do tipo do objeto, tal como append:
>>> tupla[1].append(0)
>>> print(tupla)
([9, 2], [3, 4, 0)

Embora isso seja possível, o objeto, mesmo do tipo tupla, perde a propriedade de ser um tipo imutável e, portanto, deixa de ser um objeto hashable. Ao tentar fazer, considerando tupla do exemplo anterior:
>>> hash(tupla)
TypeError: unhashable type: 'list'

Um erro será disparado indicando que a tupla possui valores que são mutáveis e, portanto, impossível de calcular o hash.
Quando se utiliza listas?
As listas são sequências mutáveis, tipicamente usadas para armazenar coleções de itens homogêneos (onde o grau preciso de similaridade variará conforme a aplicação).
Isto é, se uma lista contém valores inteiros, espera-se que sejam apenas valores inteiros. Dificilmente você terá uma lista armazenando valores de tipos distintos, tal como [1, "Foo", True].
Fonte: Built-In Types: Lists
Quando se utiliza tuplas?
As tuplas são sequências imutáveis, normalmente usadas para armazenar coleções de dados heterogêneos (como a tupla produzida pela função enumerate nativa). As tuplas também são usadas para casos onde uma sequência imutável de dados homogêneos é necessária (como permitir o armazenamento em set ou dict).
As tuplas podem possuir valores de tipos diferentes, onde cada um representa algo específico. Como o próprio retorno da função enumerate de uma lista, o retorno será uma tupla de dois valores, em que o primeiro representa o índice do valor na lista e o segundo o valor propriamente dito.
>>> lista = ["a", "b", "c"]
>>> print(list(enumerate(lista)))
[(0, 'a'), (1, 'b'), (2, 'c')]

Ou seja, um valor da tupla é do tipo inteiro, enquanto o outro é do tipo string.
Fonte: Built-In Types: Tuples
Tuplas nomeadas
Caso ainda queira utilizar das propriedades da tupla, mas deseja acessar seus valores através de índices nomeados, pode utilizar a estrutura collections.namedtuple.
>>> import collections
>>> Person = collections.namedtuple('Person', 'name age gender')
>>> foo = Person(name="Foo", age=30, gender="male")
>>> print(foo)
Person(name='Foo', age=30, gender='male')
>>> print(hash(foo))
2739343292757077799

Perceba que mesmo sendo uma tupla nomeada, utilizar a função hash ainda é possível (desde que os valores sejam todos de tipos imutáveis).


How to filter groups of paired values in Views

Question: I have a Field Collection field in a node. The field collection has two fields, so I can associate a group of paired values to a node. For example, {a,1}, {b,2},{c,3},{d,4}.
I have a View in which I want to filter by the values above. If I use a filter for the first type of values and put a, it will return the first 2-tuple. If I put 2 in the second filter (and nothing in first filter), it will return the second 2-tuple. But if I look for c in first filter and 4 in the second filter simoultaneously, it will return the 2 last values, because c matches the condition and 4 matches the condition, separatedly. How can I make  the View filter so I get the desired result, which is empty result in this case?
Answer: I am not able to reproduce the behavior you describe. In configuring a Field Collections and Views in the way you describe I do get the desired results.
Here is a screenshot of my configuration:

And here are the results:

And here is the filtering:
By letter:

By number:

Mismatched letter/number yields no results:

And here is the full View export:
<code>$view = new view();
$view->name = 'stacktest_fieldcollection';
$view->description = '';
$view->tag = 'default';
$view->base_table = 'node';
$view->human_name = 'stacktest_fieldcollection';
$view->core = 7;
$view->api_version = '3.0';
$view->disabled = FALSE; /* Edit this to true to make a default view disabled initially */

/* Display: Master */
$handler = $view->new_display('default', 'Master', 'default');
$handler->display->display_options['title'] = 'stacktest_fieldcollection';
$handler->display->display_options['use_more_always'] = FALSE;
$handler->display->display_options['access']['type'] = 'perm';
$handler->display->display_options['cache']['type'] = 'none';
$handler->display->display_options['query']['type'] = 'views_query';
$handler->display->display_options['exposed_form']['type'] = 'basic';
$handler->display->display_options['pager']['type'] = 'full';
$handler->display->display_options['pager']['options']['items_per_page'] = '10';
$handler->display->display_options['style_plugin'] = 'table';
/* Relationship: Content: FC (field_fc) */
$handler->display->display_options['relationships']['field_fc_value']['id'] = 'field_fc_value';
$handler->display->display_options['relationships']['field_fc_value']['table'] = 'field_data_field_fc';
$handler->display->display_options['relationships']['field_fc_value']['field'] = 'field_fc_value';
$handler->display->display_options['relationships']['field_fc_value']['required'] = TRUE;
$handler->display->display_options['relationships']['field_fc_value']['delta'] = '-1';
/* Field: Content: Title */
$handler->display->display_options['fields']['title']['id'] = 'title';
$handler->display->display_options['fields']['title']['table'] = 'node';
$handler->display->display_options['fields']['title']['field'] = 'title';
$handler->display->display_options['fields']['title']['alter']['word_boundary'] = FALSE;
$handler->display->display_options['fields']['title']['alter']['ellipsis'] = FALSE;
/* Field: Field collection item: Letter */
$handler->display->display_options['fields']['field_letter']['id'] = 'field_letter';
$handler->display->display_options['fields']['field_letter']['table'] = 'field_data_field_letter';
$handler->display->display_options['fields']['field_letter']['field'] = 'field_letter';
$handler->display->display_options['fields']['field_letter']['relationship'] = 'field_fc_value';
/* Field: Field collection item: Number */
$handler->display->display_options['fields']['field_number']['id'] = 'field_number';
$handler->display->display_options['fields']['field_number']['table'] = 'field_data_field_number';
$handler->display->display_options['fields']['field_number']['field'] = 'field_number';
$handler->display->display_options['fields']['field_number']['relationship'] = 'field_fc_value';
/* Sort criterion: Content: Post date */
$handler->display->display_options['sorts']['created']['id'] = 'created';
$handler->display->display_options['sorts']['created']['table'] = 'node';
$handler->display->display_options['sorts']['created']['field'] = 'created';
/* Contextual filter: Field collection item: Letter (field_letter) */
$handler->display->display_options['arguments']['field_letter_value']['id'] = 'field_letter_value';
$handler->display->display_options['arguments']['field_letter_value']['table'] = 'field_data_field_letter';
$handler->display->display_options['arguments']['field_letter_value']['field'] = 'field_letter_value';
$handler->display->display_options['arguments']['field_letter_value']['relationship'] = 'field_fc_value';
$handler->display->display_options['arguments']['field_letter_value']['default_argument_type'] = 'fixed';
$handler->display->display_options['arguments']['field_letter_value']['summary']['number_of_records'] = '0';
$handler->display->display_options['arguments']['field_letter_value']['summary']['format'] = 'default_summary';
$handler->display->display_options['arguments']['field_letter_value']['summary_options']['items_per_page'] = '25';
$handler->display->display_options['arguments']['field_letter_value']['limit'] = '0';
/* Contextual filter: Field collection item: Number (field_number) */
$handler->display->display_options['arguments']['field_number_value']['id'] = 'field_number_value';
$handler->display->display_options['arguments']['field_number_value']['table'] = 'field_data_field_number';
$handler->display->display_options['arguments']['field_number_value']['field'] = 'field_number_value';
$handler->display->display_options['arguments']['field_number_value']['relationship'] = 'field_fc_value';
$handler->display->display_options['arguments']['field_number_value']['default_argument_type'] = 'fixed';
$handler->display->display_options['arguments']['field_number_value']['summary']['number_of_records'] = '0';
$handler->display->display_options['arguments']['field_number_value']['summary']['format'] = 'default_summary';
$handler->display->display_options['arguments']['field_number_value']['summary_options']['items_per_page'] = '25';
$handler->display->display_options['arguments']['field_number_value']['limit'] = '0';
/* Filter criterion: Content: Published */
$handler->display->display_options['filters']['status']['id'] = 'status';
$handler->display->display_options['filters']['status']['table'] = 'node';
$handler->display->display_options['filters']['status']['field'] = 'status';
$handler->display->display_options['filters']['status']['value'] = 1;
$handler->display->display_options['filters']['status']['group'] = 1;
$handler->display->display_options['filters']['status']['expose']['operator'] = FALSE;

/* Display: Page */
$handler = $view->new_display('page', 'Page', 'page');
$handler->display->display_options['path'] = 'stacktest-fieldcollection';
</code>
Comment: I think that the solution is to require the relationship in both relations. That may be the difference and explains why I yield results. Once the relationship is established, the Require relationship forces a INNER JOIN which makes the related content unique.


TS Gateway Server - internal clients connect fine but external clients see error "server certificate has expired or has been revoked."

Question: I have a GoDaddy SSL certificate installed on my Terminal Services Gateway server. I can successfully connect to terminal services computers through the TS Gateway when I'm on the LAN. However,if I try to connect from an outside network, I get the error "the remote desktop gateway server certificate has expired or has been revoked."
What might be the cause of this problem?
Answer: Do you use HTTPS on the LAN?
Also, what does your topology look like, is your TS gateway simply nat'ed behind a firewall or is there some sort of forefront gateway in front of it for external clients?
I would start simply by looking at the cert on your IIS server and confirm that you have the correct one registered.  My intial guess is that you're using HTTP internal which would explain why your cert show expired only on the outside (assming you have an HTTP to HTTPS redirect here).


AlphaComposite Transparency with repaint overlaps into black

Question: So I have an image on top of another panel, and that image is transparent so you can see the panel beneath it. What I'm trying to do is use repaint() to fade the image (which is drawn with the drawImage() method in java.awt.Graphics) out until it is completely transparent so you can see the panel beneath it clearly. As of now, the image is just fading into black instead of into a transparent texture.
This is a little bit of my code right now:
paintComponent method:
public void paintComponent(Graphics g)
{
    super.paintComponent(g);
    float alpha = 1f-(.01f*(float)opcounter);
    Graphics2D g2d = (Graphics2D)g;
    g2d.setComposite(AlphaComposite.getInstance(AlphaComposite.SRC_IN, alpha);
    g2d.drawImage(img, 0, 0, null);
}

actionPerformed method that is called for the timer
public void actionPerformed(ActionEvent e)
{
    opcouner++;
    panel.repaint();
}

Longer (uncondensed) version of my code: (including paintComponent and Mover class for timer)
                            public void paintComponent(Graphics g)
            {
                super.paintComponent(g);
                Dimension framesize = frame.getSize();
                this.setBounds(0,0,framesize.width, framesize.height-61);

                if (buff)
                {
                    //this.add(buffer);
                    if (opcounter <= 255)
                    {
                        buffer.setForeground(new Color(250, 250, 250, 0+opcounter));
                    }
                    else
                    {
                        opcounter = 0;
                        buff = false;
                        hand = true;
                    }
                }
                if (hand)
                {
                    this.add(handson);
                    if (opcounter <= 255)
                    {
                        handson.setForeground(new Color(250, 250, 250, 0+opcounter));
                    }
                    else
                    {
                        opcounter = 0;
                        hand = false;
                        log = true;
                    }
                }
                if (log)
                {
                    this.add(logic);
                    if (opcounter <= 255)
                    {
                        logic.setForeground(new Color(250, 250, 250, 0+opcounter));
                    }
                    else
                    {
                        opcounter = 0;
                        log = false;
                        pan = true;
                    }
                }
                if (pan)
                {
                    this.add(panic);
                    if (opcounter <= 255)
                    {
                        panic.setForeground(new Color(250, 250, 250, 0+opcounter));
                    }
                    else
                    {
                        opcounter = 0;
                        pan = false;
                        first = false;
                        second = true;
                        try
                        {
                            //Thread.sleep(2000);
                        }
                        catch(Exception e)
                        {
                            System.out.println("thread not slept");
                        }
                        System.out.println("opcounter = " + opcounter);
                        black.setVisible(false);
                        handson.setVisible(false);
                        logic.setVisible(false);
                        panic.setVisible(false);
                        tutpic = true;
                    }
                }
                if (tutpic)
                {
                    if (opcounter <= 200)
                    {
                        Graphics2D g2d = (Graphics2D)g.create();
                        float alpha = 1f-(.01f*(float)opcounter);
                                          g2d.setComposite(AlphaComposite.getInstance(AlphaComposite.SRC_IN, alpha));
                        g2d.drawImage(tut, 0, 0, null);
                        g2d.dispose();
                    }
                    else
                    {
                        opcounter = 0;
                        tutpic = false;
                    }
                }
            }

            class Mover implements ActionListener
            {
                public void actionPerformed(ActionEvent e)
                {
                    if (!tutpic)
                        opcounter+=4;
                    else
                    {
                        opcounter++;
                    }
                    tutorial.repaint();
                }
            }

Any help would be appreciated. Thanks!
Comment: Try reseting the alpha composite before the `paintComponent` exists...
Answer: You need to restore the state of the Graphics context BEFORE the paintComponent method exists.  This is very important, as the Graphics context is a shared resource, all the components that need to be updated will be given the same Graphics, so now every thing after you component is painted will share the some AlphaComposite...
A better solution would be to create temporary copy of the Graphics context, apply what ever settings you want to it and dispose of it after you have finished.  This will ensure that what ever changes you make to the Graphics context won't be carried on after the method exists...
public void paintComponent(Graphics g)
{
    super.paintComponent(g);
    float alpha = 1f-(.01f*(float)opcounter);
    // Create your own copy...
    Graphics2D g2d = (Graphics2D)g.create();
    g2d.setComposite(AlphaComposite.getInstance(AlphaComposite.SRC_IN, alpha);
    g2d.drawImage(img, 0, 0, null);
    // Don't forget to dispose of it
    g2d.dispose();
}

Remember, you create it, you dispose it!
Update
Try using AlphaComposite.SRC_OVER instead...

import java.awt.AlphaComposite;
import java.awt.BorderLayout;
import java.awt.Dimension;
import java.awt.EventQueue;
import java.awt.Graphics;
import java.awt.Graphics2D;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import javax.imageio.ImageIO;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.Timer;
import javax.swing.UIManager;
import javax.swing.UnsupportedLookAndFeelException;

public class TestFadeOut {

    public static void main(String[] args) {
        new TestFadeOut();
    }

    public TestFadeOut() {
        EventQueue.invokeLater(
                        new Runnable() {
            @Override
            public void run() {
                try {
                    UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
                } catch (ClassNotFoundException | InstantiationException | IllegalAccessException | UnsupportedLookAndFeelException ex) {
                }

                JFrame frame = new JFrame("Testing");
                frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
                frame.setLayout(new BorderLayout());
                frame.add(new TestPane());
                frame.pack();
                frame.setLocationRelativeTo(null);
                frame.setVisible(true);
            }
        });
    }

    public class TestPane extends JPanel {

        private float alpha = 1f;
        private float diff = -0.02f;
        private BufferedImage img;

        public TestPane() {
            try {
                img = ImageIO.read(new File("C:\\Users\\swhitehead\\Documents\\My Dropbox\\Ponies\\url.png"));
                Timer timer = new Timer(40, new ActionListener() {
                    @Override
                    public void actionPerformed(ActionEvent e) {
                        alpha += diff;
                        if (alpha < 0) {
                            diff *= -1;
                            alpha = diff;
                        } else if (alpha > 1f) {
                            diff *= -1;
                            alpha = 1f + diff;
                        }
                        repaint();
                    }
                });
                timer.setRepeats(true);
                timer.setCoalesce(true);
                timer.start();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
        }

        @Override
        public Dimension getPreferredSize() {
            return img == null ? super.getPreferredSize() : new Dimension(img.getWidth(), img.getHeight());
        }

        @Override
        protected void paintComponent(Graphics g) {
            super.paintComponent(g);
            if (img != null) {
                Graphics2D g2d = (Graphics2D) g.create();
                g2d.setComposite(AlphaComposite.SrcOver.derive(alpha));
                int x = getWidth() - img.getWidth();
                int y = getHeight() - img.getHeight();
                g2d.drawImage(img, x, y, this);
                g2d.dispose();
            }
        }
    }
}

Have a look at Composting Graphics for more details...
Comment: Hmmm...the image still fades to black instead of being transparent. I'm still not sure what the problem is. The code I provided was actually a condensed version of my real program (which is a long game). If you coud check my edited question for a longer snippet of my code, that would be great.


Is it possible to redirect stdout twice with dup2?

Question: I have a problem when I used dup2 direct stdout first to a temporal file, and after to a common file. This is a portion my code:
FILE* fdArch=fopen("file.txt","w");

    char nameBuff[100];

    memset(nameBuff,0,sizeof(nameBuff));

    strncpy(nameBuff,"/tmp/myTmpFile-XXXXXX",sizeof("/tmp/myTmpFile-XXXXXX"));

    int fdNewFile=mkstemp(nameBuff);

    if(fdNewFile==-1){
        perror("error creating temporal file");
    }

    dup2(fdNewFile,STDOUT_FILENO);

    read(fd[0], readbuffer, sizeof(readbuffer));

    char *argv[]={ "/home/user/workspace/CProgram/Debug/program", readbuffer, NULL};

    execv(argv[0],argv);

    close(fdNewFile);

    dup2(fileno(fdArch),STDOUT_FILENO);

    char *argv1[]={"/usr/bin/sort",nameBuff, NULL};

    execv(argv1[0],argv1);

    fclose(fdArch);

    unlink(nameBuff);

I run an external program called program and the out is redirected with dup2 to a temporal file, well this works fine, but after when I redirect with dup2 the stdout to fdArch and run sort over the temporal file, the out is not redirected to fdArch.
Is there an error of concept or something is missing?
Comment: You do know that `execv()` doesn't return unless there's an error, right? Are you expecting the code you put after `execv()` to execute?
Comment: Yeah. You're missing a `fork` before the `execv`s.
Comment: yes, so it does not run?, before using execv I used system but I was told that it was slower
Comment: `system()` may be slower because it must setup signal handlers (POSIX requires that `system()` blocks `SIGCHLD` and ignores `SIGINT` and `SIGQUIT`), and it starts a new bash session to run the commands, whereas if you `exec()` them directly, it's faster - you don't have to wait the extra time for the new shell to parse the commands, fork and execute them. But I'd say just stick to `system()`, you probably won't see a difference.
Comment: But the code following the execv will be run? and also, the dup2 out will be redirected to fdArch?
Comment: @user4910023, no the code following the `execv()` will *not* be run, unless, of course, the `execv()` fails.  That's why you `fork()` if you want the program to continue after the `execv()`: that creates a second process, which `execv()` can then replace without disturbing the original.  In fact, it's basically what `system()` does, with a bit less frill.
Comment: I've done this with system, but now I'm bound not using it, how would you do that?
Comment: @user4910023, I described a way for you to do this with `execv()` instead of `system()` in my answer.  Feel free to add comments there if you need clarification.
Answer:
I run an external program called program and the out is redirected with dup2 to a temporal file, well this works fine, but after when I redirect with dup2 the stdout to fdArch and run sort over the temporal file,the out is not redirected to fdArch. Is there an error of concept or something is missing?

Your usage of dup2() is fine, but you do have a major error: the execv() function does not return, except when it fails.  It, and all the other functions in its family, replace the program in which it is called with the one specified.  Thus, in fact you never do redirect stdout a second time or run /bin/sort.  If you want to do those things, then you should fork(), perform the execv() in the child process, and have the parent wait() for it to complete before continuing.
Comment: Thansk, thats what I'm gonna do.


Preferred way to start a download in a browser

Question: We are currently using a meta refresh to initiate a download on a page, I read on Wikipedia that using this is not UX (user experience) friendly.  So what is the way to have a download start after a few seconds when landing on a download page.
Comment: In my opinion, landing pages and user friendly don't go together. Remove the landing page altogether or at least remove the automatic download and have the user click a link.
Comment: @casablanca HA, that is what we had, but the customer wants it to start automatically.
Answer: User interface-friendly way would:

Not spring an unannounced surprise to the user. So postpone the download by X seconds and clearly announce that.
This is done by setting up JavaScript logic to display a changing countdown to download. See below for details on implementation.
Allow user to control this by allowing immediate download
This is done by having the download timer announcement provide a widget (button or a link) to download immediately.
Keep the current page, by opening the download in a new window/tab.
The "kick off the download" logic should preferably be - instead of the obvious setting of window.location.href - something which opens a new window for the download. This way you allow the user to keep the main download landing page intact.
If possible, present a nice download widget
Instead of just pushing the URL of the download target, consider using custom download wisget like jQuery's jDownload plugin

To implement the changing countdown, do something like this:

Set up a variable for how long till download starts:
var DownloadIn = 10;
Set up a timer in JavaScript using setTimeout()`:
setTimeout("shouldWeDownload()", DownloadIn * 1000);
Subroutine shouldWeDownload() called from a timer will:

check if time period (stored in DownloadIn variable) is greater than zero.
If the time is NOT up (greater than zero), it will:
a. Check if a special "AlreadyDownloading" variable is set to true - this variable will be explained later. If true, simply exit.
b. print to a special DIV on the page - something very obvious and visible to the user - a message "XXX seconds left till the start of download. Click on this link to start the download".
c. Decrement DownloadIn variable
d. Set the timer again using the same setTimeout
If the time's up, kick off the download.

In addition, the "this link" link in the message would also immediately kick off the download. To make things clean, the "immediate dowload" onClick JS handler should set a special "AlreadyDownloading" variable which is checked in the logic above should be set to true, so we don't start a second download due to minor race conditions.
Comment: This doesn't help at all. If people are paying attention, they would just click the link instead of waiting. If they're not, then it would still be a surprise popup after X seconds.
Comment: @casablanca - if you design your GUI so that the DIV with the message is obvious, it will help.
Comment: Like I said, if the message is in fact obvious, they would simply click the link instead of waiting 10 seconds.
Comment: @casablanca - it's a question of control. If a notional user has a download pop-up spring on me un-expectedly, he can freak out.
Answer: I don't know of any research on what users expect, but I'd suggest mentioning that it's a download in your initial download link, e.g.
Download my awesome track

(Maybe style that link like a big button, maybe even with an arrow pointing downwards in it: e.g. something like this.)
And then set your server to return that file with the Content-disposition header set to 'attachment' and the name of the file, so that the browser immediately lets the user know they're downloading something:
Content-disposition: attachment; filename=kiss_from_a_rose__dubstep_remix.mp3

That way you don't open a new page just to make the file download. The file downloads, the user's still on the page where they were when it's downloaded, everyone's happy.

Fewer steps =  fewer things for users to be confused by.
Using the browser's UI = more chance that the user will have seen it before, and thus know what's going on
Comment: would this work on browsers for mobile phones (java phones, android, etc)
Comment: @arinte — Not sure, I've not tried it. At least with the iPhone (which I own), I don't think the web browser supports downloading files at all. So it wouldn't work, but neither would any other solution, including meta refresh. You'd want to avoid offering a download in the first place.
Answer: unix user freindly? Depends entirely on the browser, and last time I checked, most/all linux browsers and safari worked fine with meta refresh.
Which is why most websites offer meta refresh with a download link.
Comment: I think it means "user experience friendly"
Comment: The user experience will be exactly the same whether you use a meta redirect or anything else. Unless the browser you are looking at has some wierd quirks or the item to download opens in browsers there's no need for any special code.
Comment: @J V - wrong. User experience consists of many different things. See my asnwer.


data.table passing in a `by` expression in non interactive way

Question: I want to use several expressions in by of data.table (not just column names) in a dynamic way
This is what I want to do:
DT = data.table(x=rep(c("b","a","c"),each=3), y=c(1,3,6), v=1:9)
DT[, mean(y), by = .(over4 = v > 4)]

I thought I might be able to pass in a character vector with the expressions, but this doesn't work:
by_expression = c("over4" = "v > 4")
DT[, mean(y), by = by_expression]

Is there a way to pass in expressions in a dynamic way?
I have this work around, but this is a bit of work if there are several by conditions, and then wants tidying up after doing the by operation later
DT[, condition := v > 4, ]
DT[, mean(y), by = condition]

Comment: You can always use bquote and eval. Obviously, you should pass an actual expression (created with quote) instead of a character vector.
Comment: This seems to work without `eval`. Doesn't look messy, does it? `by_expression = quote(list(over4 = v > 4)) ; DT[, mean(y), by = by_expression]`
Comment: Or `quote(.(over4 = v > 4))` (same)
Comment: Yes, am aware of the eval(quote approach but it's kind of messy. I'm wondering if there is a simpler way in data.table syntax that I've overlooked ...
Comment: that's great - should put that as an answer and ill upvote
Comment: I will and if @Roland posts theirs I'll withdraw mine
Answer: Passing a quoted expression:
by_expression = quote(.(over4 = v > 4))
DT[, mean(y), by = by_expression]

#    over4   V1
# 1: FALSE 2.75
# 2:  TRUE 3.80

Answer: by_expression <- c("over4" = "v > 4",
                   "over2" = "v > 2")

lapply(by_expression, function(ex) DT[, mean(y), by = .(eval(parse(text = ex)))])

Result:
$over4
   parse   V1
1: FALSE 2.75
2:  TRUE 3.80

$over2
   parse       V1
1: FALSE 2.000000
2:  TRUE 3.714286


R Displaying groups in a dot chart

Question: I've made some researches on how to display graph using R, but I really don't find any answer to my problem.
I have this matrix representing cars and a vector representing the relation between groups and the different points (individuals) of this matrix. Let's say:
cars # 2d dimension matrix
categories # vector of correspondance between cars and group

The value at index [i] of the "categories" vector represent the category number (of the i em car in the matrix).
What i do want to do, is display a graph (dotchart) which display all these cars sorted by colors corresponding to their actual category.
Let's say we have 5 categories and these 5 colors :
colors <- c("white","gray","yellow","orange","red")

Is there any one who can help me out ?
Thanks!
Comment: If you post some sample data that would be helpful in answering.  You can use dput to give a taste.
Comment: Did you mean to write scatterplot when you wrote "dot chart"?
Answer: I assume that your cars variable has two columns (say, an X and a Y), and that you want to plot their relationship.
In that case, this is simple if you use the plot function:
plot(cars, col=colors[categories])

For example, you could generate and plot some random data like this:
cars = matrix(runif(2000), ncol=2) # 2d dimension matrix
categories = sample(1:5, 1000, replace=TRUE) # vector of correspondance between cars and group
colors = c("white","gray","yellow","orange","red")

plot(cars, col=colors[categories])

Answer: do you mean something like this?
 require(ggplot2)
 p <- ggplot(mtcars, aes(wt, mpg))
 p + geom_point(aes(colour = cyl)) + scale_colour_gradient(low = "blue")

this is from ggplot help pages:
Comment: Yep this is also what i ment! Thanks :D
Comment: @hahahaha If you want to ensure that each value of cyl is assigned its own color, you'll want to use `aes(colour = factor(cyl))`.

R Displaying groups in a dot chart


Trying to understand clipping logic in PDF

Question: I have the following instructions in a deflate stream inside a PDF:
1 g
/GS1 gs
0 792 m
0 792 l
f
q
1 i
0 792 612 -792 re
0 792 m
W n
0 792.06 612 -792 re
W n

Set color to Gray.
Load some external dictionary information
Start subpath in 0 792
Append point in 0 792
Fill the path.... Why would you like to fill 1 gray pixel?
Save graphic stack
set smoothness to 1
Start a 0 792 612 -792 rectangle subpath
Start a 0 792 subpath.... Why would you override the previous rectangle path?
Make the 0 792 subpath the clipping path and remove the current path.
Start a 0 792.06 612 -792 rectangle.
Clip the previous rectangle with the previous clipping path 0 792?

The problem I have is that I didn't find good examples for how the clipping works. Which path do you clip against? What do you clip against if you didn't have a clipping path before?
Thanks!
Answer: First of all you are right to wonder, much in the excerpt you show makes no sense.
For example your items 3..5 concerning
0 792 m
0 792 l
f

The PDF specification states on filling paths like that:

If a subpath is degenerate (consists entirely of one or more points at the same coordinates), the subpath shall be considered to enclose the single device pixel lying under that point; the result is device-dependent and not generally useful.

(ISO 32000-2, section 188.8.131.52 — Filling)

Your interpretation of
0 792 612 -792 re
0 792 m
W n

in your items 8..10 is incorrect, though. In particular your assume the last line to

Make the 0 792 subpath the clipping path and remove the current path.

Actually the 0 792 m subpath is disregarded and only the 0 792 612 -792 re rectangle is intersected with the current clip path!
According to the specification clip path intersection works similar to filling

Subclause 220.127.116.11, "Filling" defines what is inside a path as well as stating rules for closing paths and for degenerate paths. For a given path definition, the same area that would be filled by the f operator is the area that would be used for a clip.

(ISO 32000-2, section 8.5.4 — Clipping path operators)
and filling would drop that dangling 0 792 m.

Any subpaths that are open shall be implicitly closed before being filled, except that if the last subpath in the path is a single-point open subpath (specified by a trailing m operator), it shall be disregarded and not considered to be part of the path.

(ISO 32000-2, section 188.8.131.52 — Filling)

Thus your interpretation of
0 792.06 612 -792 re
W n

in your items 11 & 12

Clip the previous rectangle with the previous clipping path 0 792?

is slightly off: Actually this rectangle clips the previous clipping path last clipped by the other, nearly identical rectangle. But this hardly makes more sense to do than your original assumption would have.

Concerning your explicit question

Which path do you clip against? What do you clip against if you didn't have a clipping path before?

Here the specification says:

The initial clipping path shall include the entire page.

(ISO 32000-2, section 8.5.4 — Clipping path operators)
Comment: *"So if I understood correctly: 0 792 612 -792 re 0 792 m are 2 subpaths in a Path object. Because the m one is degenerate I use the previous valid subpath, the rec."* - Because the m one is merely a single trailing m, you use *only* the previous valid subpath. In general the *whole* path consisting of any number of subpaths would be used for clipping.
Comment: *"The clipping path is the whole page at the beginning so: 0 0 612 792 re gets clipped against 0 792 612 -792 re ,and then clipped against 0 792.06 612 -792 re again?"* - If the whole page 612x792 with the origin in its lower left, yes.
Comment: Hi mkl, thanks for this in-depth answer. So if I understood correctly: 0 792 612 -792 re
0 792 m are 2 subpaths in a Path object. Because the `m` one is degenerate I use the previous valid subpath, the rec. The clipping path is the whole page at the beginning so: 0 0 612 792 re gets clipped against 0 792 612 -792 re ,and then clipped against 0 792.06 612 -792 re again?