Find duplicate items within a list of list of tuples Python

Question: I want to find the matching item from the below given list.My List may be super large.
The very first item in the tuple "N1_10" is duplicated and matched with another item in another array
tuple in 1st array in the ListA ('N1_10', 'N2_28')
tuple in 2nd array in the ListA ('N1_10', 'N3_98')
ListA  = [[('N1_10', 'N2_28'), ('N1_35', 'N2_44')],
          [('N1_22', 'N3_72'), ('N1_10', 'N3_98')],
          [('N2_33', 'N3_28'), ('N2_55', 'N3_62'), ('N2_61', 'N3_37')]]

what I want for the output is
output --> [('N1_10','N2_28','N3_98') , .... and the rest whatever match one of the
key will get into same tuple]
If you guys think , changing the data structure of the ListA is better option , pls feel free to advise!
Thanks for helping out!
SIMPLIFIED VERSION
List A  = [[(a,x),(b,k),(c,l),(d,m)],[(e,d),(a,p),(g,s)],[...],[...]....]
wantedOutput --> [(a,x,p),(b,k),(c,l),(d,m,e),(g,s).....]
Comment: Wait, how did you get the output?
Comment: i havent got the output yet. that's why i m asking here :)
Comment: I meant, what factors the output? I'm not seeing a pattern, if you get what I mean
Comment: Oh wait, I see. Is it only the first item in the tuple which is a duplicate of something else?
Comment: That simplified version helped, thanks :)
Comment: i have edited the question. The item may be any in any position but each tuple got only 2 items.. Thanks for helping out @Haidro :)
Comment: I'm afraid I may not know the answer to this, but I'll see...
Comment: it's ok :) i know there are genius out there! so should be fine ^^
Comment: Can each string not occur more than twice? In your simplified version, what would happen if a is also in sub-list 3?
Comment: related: [Replace list of list with "condensed" list of list while maintaining order](http://stackoverflow.com/q/13714755/4279)
Answer: Update: After rereading your question, it appears that you're trying to create equivalence classes, rather than collecting values for keys. If
[[(1, 2), (3, 4), (2, 3)]]

should become
[(1, 2, 3, 4)]

, then you're going to need to interpret your input as a graph and apply a connected components algorithm. You could turn your data structure into an adjacency list representation and traverse it with a breadth-first or depth-first search, or iterate over your list and build disjoint sets. In either case, your code is going to suddenly involve a lot of graph-related complexity, and it'll be hard to provide any output ordering guarantees based on the order of the input. Here's an algorithm based on a breadth-first search:
import collections

# build an adjacency list representation of your input
graph = collections.defaultdict(set)
for l in ListA:
    for first, second in l:
        graph[first].add(second)
        graph[second].add(first)

# breadth-first search the graph to produce the output
output = []
marked = set() # a set of all nodes whose connected component is known
for node in graph:
    if node not in marked:
        # this node is not in any previously seen connected component
        # run a breadth-first search to determine its connected component
        frontier = set([node])
        connected_component = []
        while frontier:
            marked |= frontier
            connected_component.extend(frontier)

            # find all unmarked nodes directly connected to frontier nodes
            # they will form the new frontier
            new_frontier = set()
            for node in frontier:
                new_frontier |= graph[node] - marked
            frontier = new_frontier
        output.append(tuple(connected_component))

Don't just copy this without understanding it, though; understand what it's doing, or write your own implementation. You'll probably need to be able to maintain this. (I would've used pseudocode, but Python is practically as simple as pseudocode already.)
In case my original interpretation of your question was correct, and your input is a collection of key-value pairs that you want to aggregate, here's my original answer:
Original answer
import collections

clusterer = collections.defaultdict(list)

for l in ListA:
    for k, v in l:
        clusterer[k].append(v)

output = clusterer.values()

defaultdict(list) is a dict that automatically creates a list as the value for any key that wasn't already present. The loop goes over all the tuples, collecting all values that match up to the same key, then creates a list of (key, value_list) pairs from the defaultdict.
(The output of this code is not quite in the form you specified, but I believe this form is more useful. If you want to change the form, that should be a simple matter.)
Comment: First, the output from this code is a list of a list of characters. You should use ```append``` not ```+=```. Second, (if you used append), the code loses relation information. I think I know what you were trying to do, but it's not right.
Comment: @korylprince: Crap, you're right about `append`. My bug. For the other thing, I originally read the question as trying to aggregate a bunch of key-value pairs, but upon a second reading, it appears to be a connected-components analysis of a graph. I'll update my response with a connected-components algorithm.
Comment: @user2357112 , yah , it is kind of adjacency matrix list representation. if you have time could you pls tell how  how to change it ?
Comment: @Peter: I've added an example implementation of a breadth-first search-based algorithm.
Answer: tupleList = [(1, 2), (3, 4), (1, 4), (3, 2), (1, 2), (7, 9), (9, 8), (5, 6)]

newSetSet = set ([frozenset (aTuple) for aTuple in tupleList])
setSet = set ()

while newSetSet != setSet:
    print '*'
    setSet = newSetSet
    newSetSet = set ()
    for set0 in setSet:
        merged = False
        for set1 in setSet:
            if set0 & set1 and set0 != set1:
                newSetSet.add (set0 | set1)
                merged = True
        if not merged:
            newSetSet.add (set0)

        print [tuple (element) for element in setSet]
        print [tuple (element) for element in newSetSet]
        print

print [tuple (element) for element in newSetSet]

# Result:  [(1, 2, 3, 4), (5, 6), (8, 9, 7)]

Comment: Tested, take out the print statements for speed.
Comment: Technically, it works (I think), but that's going to have atrocious performance. In the worst case, it'll run in exponential time, because it may end up going through every possible combination of 2 or more input elements.
Comment: Each time the while loop is done, at least one merge has happened, which makes the cardinality of newSetSet one less. So I would expect O (n^3) rather than O (exp (n) for the three nested loops. But I may very well have overlooked something. If so, please explain.
Comment: A set can merge with many other sets each iteration, not just one. These merges don't produce one big merged set; they produce a bunch of partially merged sets. newSetSet can actually grow when this happens. Try it with `tupleList = [(x, y) for x in range(n) for y in range(x)]` for various values of `n`.
Answer: Does output order matter? This is the simplest way I could think of:
ListA  = [[('N1_10', 'N2_28'), ('N1_35', 'N2_44')],[('N1_22', 'N3_72'), ('N1_10', 'N3_98')],
            [('N2_33', 'N3_28'), ('N2_55', 'N3_62'), ('N2_61', 'N3_37')]]

idx = dict()

for sublist in ListA:
    for pair in sublist:
        for item in pair:
            mapping = idx.get(item,set())
            mapping.update(pair)
            idx[item] = mapping
            for subitem in mapping:
                submapping = idx.get(subitem,set())
                submapping.update(mapping)
                idx[subitem] = submapping

for x in set([frozenset(x) for x in idx.values()]):
    print list(x)

Output:
['N3_72', 'N1_22']
['N2_28', 'N3_98', 'N1_10']
['N2_61', 'N3_37']
['N2_33', 'N3_28']
['N2_55', 'N3_62']
['N2_44', 'N1_35']

Comment: I'm pretty sure that's bugged. Whenever an input pair is processed, the set corresponding to the second item of the pair gets replaced with the set corresponding to the first item. This loses data if the second item already had stored associations.
Comment: Actually i am pretty new to python. can u explain more about forzenset? i juz now confirm with my data set , there is no duplicated data in each array of tuples however , the items may duplicated for different array across the big listA.
Comment: @user2357112 good catch. Updated the code to fix this and readability. @Peter - a ```frozenset``` is just a ```set``` that can't be modified. I use that line of code because you can't have a ```set``` of ```set```s in python (because a ```set``` isn't hashable) but you can have a ```set``` of ```frozenset```s.


How to know node/entity ID from a submit handler?

Question: How to know node/entity ID from a submit handler of a form created with hook_form()?
I need to make db_update() query on a single row from submit handler called by admin/structure/video/manage/% form created with Forms API.
Answer: I would recommend not to use extra submit handlers on node forms. Forms are not the only way that you can create a node. Modules like Feeds, Migrate, etc can create nodes programatically.
If it's a node, and you don't want to alter data that the node itself, implement hook_node_insert and you have access to the whole $node object!
$node->nid would be the node ID if you name your function's first argument as $node.
If you prefer an extra submit handlers, you can access node or entity that is being created/updated in $form (I can't remember the exact key name though).
Install Devel module. Then, you can see what's in the $form array in your submit handler like this:
function mymodule_extra_submit_handler($form, &$form_state) {
  dpm($form); // pretty-prints the entire form.
}

Usually $form['#node'] is the node object, so nid is $form['#node']->nid
Comment: It is an entity not a node. There are no '#node' element in the array $form. You don't remember the exact key and I am searching of this key long time :-(
Answer: $form['build_info']['args'][0]['id']
$form_state['build_info']['args'][0]->id;
Answer: might be you are using
$form['#submit'][] = 'custom_submit_function';
use this code:
$form['actions']['submit']['#submit'][] = 'custom_form_submit';

Definitely you will get nid on print_r($form_state);
e.g
function custom_form_alter(&$form, &$form_state, $form_id) {

if ($form_id == 'portfolio_node_form') {
   // $value = $form['nid']['#value'];
   $form['actions']['submit']['#submit'][] = 'ita_custom_form_submit';
}

return $form;
}

function ita_custom_form_submit($form, $form_state) {
      $val = array();
    $val = $form_state['values'];
    print"
";
    print_r($val);
    //exit(); // do uncomment to see form_state object
//coding
}

Answer: I used
$form['#submit'][] = 'custom_submit_handler';

to add my submit handler to run after the default submit handler.
Then in my custom handler function, I found the entity ID in the $form_state variable:
$form_state['values']['entity']->id

I hope this helps someone!


Fitting polynomial in regression: undesirable nonlocal effects

Question: So we use polynomial regression for estimating conditional mean functions of unknown form, and especially higher order polynomials are flexible. I've learned in the past that especially higher-order polynomials have undesirable effects when being fitted: values of the response variable for some special segments of the predictor will affect badly estimated values for other segments of the predictor. I would be grateful for any further explanation on this matter!
Comment: "I would be grateful for any further explanation on this matter!" is too vague for a really good answer. Please try to ask specific questions.
Comment: Yes, sorry! Old professor told me this once, and I could not remember exactly what he said.
Answer: Polynomials are notorious for undesirable non-local effects and aside from very low degree polynomials (2 or 3 in particular situations) are usually avoided for that reason. Peter's discussion gives some sense of why it happens, but in any case it's easy to observe in practice -- a simple case is to fit a high order polynomial to a few data points (e.g. a degree 6 polynomial to 9 data points say) and then notice the way the behavior of the whole fit alters as you change one of the points.
Splines are often used when there's no specific functional relationship in mind. For example, a similar situation when fitting natural cubic regression splines will have local effects - the fit for nearby points will change - but as you move more than a few knots away from the changing value, the fit doesn't alter. You can also parameterize your spline fit (via B-splines say) so the coefficients of the components are only affected locally.
Similarly, local regression methods (such as local linear regression via a bounded kernel supplying weights) is only impacted within the bandwidth of the kernel.
Comment: This clears things up greatly, this was what I was trying to remember!
Answer: Think about how polynomials behave.  Graph any polynomial.  Look at what happens when you extrapolate beyond the data.  The function will go to negative or positive infinity.  That is almost always undesirable.  But if you only need to estimate in the range where you already have data, then it should be OK, usually.
Comment: How about spurious oscillations ... even for interpolation?  If you answer were true, regression splines and smoothing splines would be pretty much out of business, or never gotten into business.
Comment: Aha, I get it. Of course!


How to group data in SQL table to show every combination?

Question: Using SQL Server 2008, and this completely imaginary scenario and data, how can I transpose this (incomplete) database table?

The final table should contain every Town in its City and every combination of thee (different) animals associated with that town and city.

I have been considering a pivot but can't find a solution that will solve this particular problem.
I am looking to understand the concept, so I can apply the technique to my real data, which is a bit more complex.
Comment: I think you have misunderstood the meaning of PIVOT. PIVOT is for transposing rows to columns...
Comment: Yes I was getting confused. Any suggestions about how the above could be implemented?
Comment: You may check my solution. You needed CROSS JOINs.
Comment: Dan, do you recall whether the below answer assisted you with your use case? Although it is not mandatory on Stack Overflow, we encourage question authors to accept good answers if they are able to.
Answer: The solution you need is CROSS JOIN the table with itself three times and then filtering out doubles and triples.
You may check this fiddle
The code is :
<code>SELECT  T1.Country,
        T1.Town,
        T1.Pet AS Pet1,
        T2.Pet AS Pet2,
        T3.Pet AS Pet3
FROM    Pets T1
        CROSS JOIN Pets T2
        CROSS JOIN Pets T3
WHERE   T1.Pet <> T2.Pet
        AND T2.Pet <> T3.Pet
        AND T1.Pet <> T3.Pet
        AND T1.Country = T2.Country
        AND T1.Country = T3.Country
        AND T1.Town = T2.Town
        AND T1.Town = T3.Town
</code>
Comment: These are not cross joins. They are inner joins (non-equi joins to be precise). You merely moved the join criteria to the `WHERE` clause, which is considered bad style.
Comment: As to the result: You are returning too many rows, because you have all combinations multifold. Change `<>` to `<` in order to avoid this.


Is there any connection between these two distributions?

Question: I was playing with standard uniform distributions where I noticed a "weird" relation between two combinations and was wondering if there was an underlying reason for it (or if it is just coincidence).
Consider two variables $X \sim U(0,1)$ and $Y | X \sim U(0,x)$. The product distribution $Z = XY$ can be calculated and I find the result to be $f_Z(z) = \frac{1}{\sqrt z} -1$ when $z \in [0,1]$.
Consider now two standard uniform variables $U,V$ and a new variable $W = (U - V)^2$. This distribution can be computed analytically as well and I get $f_W(w) = \frac{1}{\sqrt w} - 1$ when $w \in [0,1]$.
Aside from this showing that two (different?) random variables ($W$ and $Z$) can share the same distribution, is there any connection between them that I am missing?
Comment: When a distribution has multiple geneses, the interpretation of the differing generating mechanisms adds to the discussion of the distribution.  For instance, the negative binomial distribution arises from a Poisson distribution with rate parameter following a gamma distribution and also as the sum of $N$ independent random variables each having a logarithmic distribution, where $N$ follows a Poisson law.  Of course, there are dozens of geneses for the negative binomial distribution.
Comment: Can you clarify what you mean by "a logarithmic distribution"?
Comment: The distribution of $Z$ seems to be also equivalent to $X_1^2X_2$ with iid $X_i\sim U(0,1)$. That is, we can relate $X = X_1 \sim U(0,1)$ and $Y\vert X_1= X_1X_2 \sim U(0,X_1)$.
Answer: There is a different representation. For $X \sim U(0,1)$ and $Y \sim U(0,1)$ we have that
$$Z = XY^2 \sim (X-Y)^2$$
with $f_Z(z) = \frac{1}{\sqrt{z}} - 1$
Below is a plot of isolines for the two cases $Z = XY^2$ (broken black lines) and $Z = (X-Y)^2$ (solid gray lines). The equations to draw these lines are $Y = \sqrt{Z/X}$ for black and $Y = X - \sqrt{Z}$ and $Y = X + - \sqrt{Z}$ for gray.

Let's focus on the complementary cumulative distribution $\bar{F}_Z(z) = 1 - {F}_Z(z)  = P(Z>z)$.
This probability is equal to the area bounded by those isolines.

For the gray isolines, which are straight lines, we can compute it easily. It is the area bounded by a triangle of size $1-\sqrt(z)$ (and the isolines are on both sides so we have to multiply by two). $${F}_Z(z) = (1-\sqrt{z})^2$$

For the black isolines, which have a bend, we can use an integral

$$\begin{array}{}
\bar{F}_Z(z) &=& \int_z^1 1- \frac{\sqrt{z}}{\sqrt{x}} dx \\
&=&  x - 2\sqrt{x}\sqrt{z} \,\,  \Biggr|_{z}^{1} \\
&=& (1 - 2\sqrt{1}\sqrt{z}) - (z - 2\sqrt{z}\sqrt{z}) \\
 &=& (1-\sqrt{z})^2
\end{array} $$
We see that the pattern is quite different but both of these have the same distribution.
I doubt that there might be an intuitive connection. One try that can be done is to see if some transformation from the points on the line $Z = XY^2$ to the line $Z = (X-Y)^2$ might be possible.


Override a view definition programmatically

Question: I want to override a view definition programmatically (using the code that I get from "view export" in the views admin pages).
I've found examples for creating views using views_new_view(), but nothing on modifying views definition programmatically. Are there any examples / tutorials on this ?
Can I do this with views_get_view($name) / views_save_view() ?
Comment: Look at [Views hooks](https://api.drupal.org/api/views/views.api.php/group/views_hooks/7) and $view->add_item
Answer: Ok, I got to a working solution :
<code>$my_view_name = 'my_test_view';

$view = views_get_view($my_view_name);
if ( $view ) {
  views_delete_view($view);
}

// ... code from view export UI ...

$view->save();
</code>
If the view exists already then I delete it before recreating it.
Then I throw in the code taken from the views export UI, and add right after a <code>$view->save();</code>
And that does it.


Match sub-rule backwards in ANTLR4 parser

Question: I have a portion of an ANTLR4 rule that I'd like to parse backwards.  I suspect that's not the real solution, so there's likely something I'm missing.
The crux of my problem is that there's a part in the middle of my expression that I'd like to extract.  However, this part has some (defined) suffixes that I would really like to extract separately, if possible.  These suffixes can be separated by a comma or not; the grammar works fine with the comma, but if the comma is missing, it takes the entire part as unknown, even if the suffixes are present.
I've pared down my grammar into a small example, visible at the bottom of this post.
Given the string why hello, x y z foo bar baz blah blah blah, goodbye!, my grammar will parse x y z foo bar baz as a phrase.  I would like to match x y z as unknown and foo bar baz as suffixes.  If there is a comma (x y z, foo bar baz), it works:

However, if there is no comma, it takes the entire x y z foo bar baz (as well as some of the text after) as unknown:

I tried changing unknown to be nongreedy (+?), but that is undesirable as well, consuming only one token for phrase:

Is there a way to force the phrase rule to try matching suffixes from the right first before falling back to unknown?
Another way to put it: is there a way to have unknown match anything except when it ends with one or more suffixes?  (The suffixes can appear in the text as long as they're not at the end)
Example grammar:
grammar Example;

// parse tree root
exampleExpression : ignored HELLO separator phrase separator? unknown separator? GOODBYE ignored;

// what I want to match
phrase : unknown (COMMA? suffix+)*;

// convenience rule for swaths of tokens to be ignored (e.g. at the beginning and end)
ignored : (unknown | separator)*;

// roll up unknown tokens under one rule
unknown : (~(PERIOD | COMMA | PIPE | BULLET | SP_SEP_DASH))+;
separator : PERIOD | COMMA | PIPE | BULLET | SP_SEP_DASH;

// the pre-defined suffixes
suffix : FOO | BAR | BAZ;

/* TOKENS */

HELLO : 'hello';
GOODBYE : 'goodbye';
FOO : 'foo';
BAR : 'bar';
BAZ : 'baz';

/* FRAGMENTS */

fragment DIGIT : [0-9];
fragment DASH : '-';

/* REMAINING TOKENS */

LPAREN : '(' ;
RPAREN : ')' ;
COMMA : ',';
PERIOD : '.';
PIPE : '|';
BULLET : '\u00B7' | '\u2219' | '\u22c5';
SP_SEP_DASH : SP DASH SP;

SP : [ \u000B\t\r\n] -> channel(HIDDEN);

NUMBER : ([0] | [1-9] DIGIT*) ('.' DIGIT+)?;
WORD : [A-Za-z] [A-Za-z-]*;

// catch-all
OTHER : .;

Comment: Wild guess: `phrase : unknown COMMA? (suffix+)*;` instead of `phrase : unknown (COMMA? suffix+)*;`.
Comment: @500-InternalServerError: Surely `(suffix+)*` is the same as `suffix*`, but I suppose what is really wanted is `suffix+`, possibly with interspersed COMMAs. IOW: `COMMA? (suffix+ COMMA)* suffix+`. But I don't think that is the fundamental problem here.
Comment: My point was that `phrase` only appears to accept `suffix` if `COMMA` is also present.
Comment: Right, `phrase` has `(COMMA? suffix+)*` to support phrases like `x y z, foo, bar, baz`.  Suffixes are not always present, though, hence `*` instead of `+`.  FWIW, if I were to ignore the multiple-comma case, `unknown COMMA? suffix*` exhibits the same problems.
Answer: The question says:

Another way to put it: is there a way to have unknown match anything except when it ends with one or more suffixes? (The suffixes can appear in the text as long as they're not at the end)

But previously, a parse of unknown with internal suffixes was rejected:

However, if there is no comma, it takes the entire x y z foo bar baz (as well as some of the text after) as unknown

That seems inconsistent.
From the example, it seems like you are trying to do natural language parsing; ANTLR, whatever its virtues, is probably not a good tool for that. But that might just be a chimera based on your simplification.
In any event, the answer to your original question -- "is it possible to define a non-terminal as any sequence of tokens which don't end with one or more tokens from a suffix class" is "yes, that can be written as a context-free-grammar". Without getting into ANTLR specifics, here's a simple CFG:
wordlist: /* empty */ | wordlist non_suffix | wordlist suffix_list non_suffix ;
suffix_list: suffix | suffix_list suffix ;

Comment: When I said "However, if there is no comma, it takes the entire x y z foo bar baz (as well as some of the text after) as unknown," I meant that that's what's happening, though undesired.  In that case, I would have expected `unknown` `suffixes` `unknown`.  I can see how that might be difficult to figure out, though, unless it was parsed backwards somehow (or that was my thinking).

Thanks for your suggested CFG.  I will mess around with that as a starting point.
Comment: @NickAldwin: I understood that. But in your requirements, you say that unknown can contain suffixes if they are not at the end (highlighted in my quote). Then you complain that your grammar is accepting unknown with suffixes which are not at the end. That's the inconsistency. If you want suffixes to be parsed as suffixes regardless of where they occur, the grammar is trivial. If you have other expectations, you need to be more precise about them.
Comment: Understood, there's no way for the parser to understand the difference between `unknown` `suffixes` `unknown` and `unknown(with suffixes in the middle)`.
Comment: @NickAldwin: It's not just the parser which doesn't understand that difference. *I* don't understand the difference :)


Inserir dados no banco de dados MYSQL pelo PHP com mais de 1 tela

Question: Eu tenho o meu index.php e consigo inserir dados no banco de dados utilizando formulário html, porém eu preciso de várias telas para esse formulário ou seja são 3 telas, index.php, tela2.php e tela3.php como eu faço para fazer só 1 inserção utilizando um botão submit no final de tudo na tela3.php mas pegando todos os campos das 3 telas e fazendo só 1 inserção como disse? estou utilizando o código abaixo porém ele só faz a inserção nos campos de formulário na mesma tela.
if(isset($_POST['acao'])){
    $nome = $_POST['nome'];
    $sobrenome = $_POST['sobrenome'];
    $momento_registro = date('Y-m-d H:i:s');

    $sql = $pdo->prepare("INSERT INTO `clientes` VALUES(null,?,?,?)");

    $sql->execute(array($nome,$sobrenome,$momento_registro));

    //echo 'Cliente inserido com sucesso!';
}

Answer: Uma das soluções, se você vai usar somente PHP por falta de conhecimento em outra tecnologia como Javascript para fazer isso sem dar refresh na página, seria criar um formulário para cada arquivo.
Na submissão do formulário, através de um input do tipo submit, você grava os dados em uma sessão e recupera todos estes dados na última tela.
E faz o insert enviado $_SESSION['dados'] para o seu script de banco.
Faz um curso de MVC, não invista em código procedural porque já caiu em desuso. Aprendi a usar um Laravel da vida :)
Answer: Dá uma olhadela aqui. https://codepen.io/piyushpd/pen/PooPByb e aqui https://bootsnipp.com/search?q=wizard .
Isto é um exemplo de um 'Wizard Form'. O codigo fica todo na sua tela1.php mas parece que tem varias paginas. Mesmo se não quiser usar um 'wizard form' penso que a unica maneira será guardar cada form em uma variavel de sessão e então depois na submissão do ultimo form acessar as variaveis que foi guardando enquanto ia submetendo os forms ''antigos''. Pesquise por ''wizard form'' e vai encontrar mais informação se precisar.
Answer: Há duas formas de se fazer isso utilizando html e php nativo, a primeira delas é: Você pode passar os valores de um formulario para o outro e esconder os valores com o hidden, exemplo:
e só fazer a inserção na tela3.
Ou pode fazer a inserção de valores no primeiro form, preencher os campos restantes com null, e na tela 2 e 3 fazer um update neste regristro


Is Cantor-Bendixson theorem right for a general second countable space?

Question: The Cantor-Bendixson theorem states that

If $X$ is Polish then any closed subset of $X$ can be written as the disjoint union of a perfect subset and an at most countable subset.

It seems that we can weaken the condition as follows:

If $X$ is second countable (even not $T_1$), then any closed subset $F$ of $X$ can be written as the disjoint union of a perfect subset and an at most countable subset

The canonical proof applies as follows:
Suppose $\mathcal B$ is a countable base. Let $P$ be the set of condensation points, since $F$ is closed, $P\subseteq F$. For each $x\in F\setminus P$, we assign $B_x\in\mathcal B$ such that $x\in B_x$, and $B_x\cap F$ is at most countable, then there's a map $\phi\colon F\setminus P\to\mathcal B,x\mapsto B_x$. Note that the preimage of each $B_x$ is at most countable, therefore $F\setminus P$ is countable.
It remains to show that $P$ is perfect. Every point $p\in P$ is a condensation point of $F$, hence $P$, since $F\setminus P$ is countable, hence a limit point of $P$. Conversely, if $x\not\in P$, there's an open set $\mathcal O$ contains $x$ such that $\mathcal O\cap F$ is at most countable, hence $\mathcal O\cap P=\emptyset$.
Is it right?
Comment: Yes. There is no problem with this argument.
Answer: It looks fine to me. If I had to hazard a guess about why it's not general stated in that generality it's that if your $X$ is Hausdorff then this is implied by the standard version by embedding $X$ in its completion, and that there aren't many non-Hausdorff spaces of interest in descriptive set theory.
Comment: Even if it's Hausdorff, it's not necessarily metrizable, right?


Why are these two strings with linebreaks different?

Question: "[a='b\\\nc']"

And
@"[a='b\
c']"

Does the second add a \r or something? Is there an easy way to escape it so I can "see" the line break characters?
Comment: These two strings are exactly identical.
Comment: @Michael: No... they aren't. It *might* depend on your OS, but the second one contains a `\r`.
Comment: You're right - on Windows, the first contains a single linefeed, and the second a carriage-return and linefeed pair.
Comment: Was causing problems in my code and confusing the heck out of me. Now I know!
Answer: Nevermind. Just tried putting it through Regex.Escape which escapes a bit more than it should, but it does show the \r.


What is the practical use/aim of `git fetch` command

Question: I understand the purpose of git fetch but I don't get why someone want to continuously update his local repo without merging. I mean when you are ready you can just do a git pull. Why do git fetch followed by git merge in increments ?
Answer: git fetch gives you a full copy of the remote's history, allowing you do to pretty much instant diffing and things like that. Suppose you're working on a complex change on a topic branch while the main branch keeps getting updated. You don't want to merge all the time because merges are a lot of work, but you do want to be aware of what's going on in the main branch so you can anticipate (and perhaps even prevent) extra work. If you have the history available locally, it shows up in all your visualization tools (e.g. gitk --all); personally I don't think looking at Gitweb or Github interfaces all the time really compares.
And, of course, the big benefit to doing fetch/merge separately is so you can see whether the changes on the other branch are actually worth merging.


How do I manually set tnid when migrating nodes?

Question: I need to manually set tnid during migration.
My code:
  public function complete($entity, $row) {

    // load the newly created node (just in case entity isn't fully populated).
    $node = node_load($entity->nid);
    $node->tnid = '66';
    $node->title = 'CHANGE THIS !';

    // create the field collection item entity
    $item = entity_create('field_collection_item', $values);

    // attach it to the node loaded above
    $item->setHostEntity('node', $node);

    // save the entity
    $item->save();
  }

The node 66 exists and is active, but when I check on translations, the new node isn't associated. Instead the  title is changed with "changed this", then the new node is updated.
With Devel I see that the node has tnid 66.
Comment: Hi, monkeyUser. What do you  mean "check on translations"?
Answer: You want to give the node a language, and set the source node in the translation_source key:
$node->language = 'es';
$node->translation_source = node_load(66);

Reference: https://api.drupal.org/api/drupal/modules!translation!translation.module/function/translation_node_prepare/7


Unknown Music Notation for Vocals

Question: I would to ask what the notation circled in red meant. Thank you
Answer: I guess this is the intended range for the exercises. I.e. the first exercise top note is c, so they suggest to transpose it up in half-tone steps by up to a perfect fifth, when that note becomes g, as noted in the circle.
Please note this is a suggestion from the author and you should adjust it to your own voice range.
Comment: For what it's worth, my wife is a vocal teacher and understood the notation in the same way you did. Something like this : https://www.youtube.com/watch?v=fCxmDUl8__U
Comment: That seems the most likely intended meaning.  Though I don't think it's at all clear.  (I think range/ambitus is usually put at the beginning of a piece.)  Some text would have helped.
Comment: Each exercise shows two repetitions, with the second raised by a half step.  I wonder if simply, "etc. ad lib, adjusting for vocal range" would have been better?
Answer: Could it be a vocal slide? Some vocal coaches use it to smoothen out the break in the mix. Would make sense to try to gently glide across a wide range without breaking as an exercise after doing some other exercises
Comment: While slides are indeed good exercises, I don't think that was intended in this case, as the layout's wrong: the notes are small, too close together, and in incomplete bars.

Unknown Music Notation for Vocals


GraphLab Create Installation Error - Python 3.5.2

Question: I have been trying to install GraphLab Create on my system (64bit, Windows 10).
I used the Installation guide as given on the site https://turi.com/download/install-graphlab-create-command-line.html
I used the command line since I already had Anaconda3 installed in my system.
I tried the Method - 1 but it didn't work because I'm not using Python2.7,  but Python3.5.2, so, I tried Method - 2 by using Virtual Environment.
I am getting a Syntax error when I try the command:
virtualenv gl-env

I have already installed Virtual Environment through pip so this should not happen, right ?
Please help me out on what could possibly be wrong.
Comment: Could we see the full traceback?
Answer: From the installation instructions page:

GraphLab Create installation requires a Python 2.7.x environment and pip version >= 7

You have to install GraphLab in a Python 2.7 environment, and not in a Python 3 one.
Thus, install python2 first and then follow the rest of the instructions:
conda create -n gl-env python=2.7 anaconda=4.0.0


What is the purpose of the commented import statement after the package statement?

Question: Some packages have a commented import statement just after the package statement. An example is visible here.
<code>. . .
package truetype // import "github.com/golang/freetype/truetype"
. . .

</code>
It seam that this appears on a package inside a module. Not all files of the package have this comment.
What is the purpose of this comment ? Why is it there ?
I didn't find anything about this in the go specification.
Comment: This defines under which import path the package must be imported.
Comment: @blackgreen: But the fact that it can be confused with documentation is a great reason that I dislike Go's various comment-based tricks.
Comment: @chmike: it is redundant now with `go.mod`, and no longer needed
Comment: Related: https://stackoverflow.com/q/37330303/13860
Comment: @Volker It seam that this has become obsolete since we use go.mod. Can I thus ignore it ? Is it replaced by the module name in the go.mod file ?
Comment: It might be redundant if the given import path actually matches the one from go.mod but if not I do not know if your build will succeed. Did you try it out?
Answer: This is for custom import paths.
See the import path checking documentation, and also this SO answer.

A package statement is said to have an "import comment" if it is
immediately followed (before the next newline) by a comment of one of
these two forms:
<code>package math // import "path"
package math /* import "path" */
</code>
The go command will refuse to install a package with an import comment
unless it is being referred to by that import path. In this way,
import comments let package authors make sure the custom import path
is used and not a direct path to the underlying code hosting site.
Comment: It might be useful to add that it has become obsolete with the usage of go modules.


Proving left inverse for $A$

Question: Taken from Artin's book, need to prove that an $m \times n$ matrix $A$ (where $m < n$) has no left inverse. A hint given is to compare matrix $A$ to an $n \times n$ matrix obtained by taking $A$ and adding ($n-m$) zero rows to the bottom.
Answer: The point is that if you had such a left inverse, that is, an $n \times m$ matrix $B$ with $BA = I_n$, you could add $n-m$ zero rows to the bottom of $A$ and $n-m$ zero columns to the right of $B$ to get two $n \times n$ matrices $A'$ and $B'$ such that $B'A' = BA = I_n$. But for instance $A'$ cannot be invertible because its rows are linearly dependent.
More conceptually, the matrix $A$ represents a linear map $\mathbb{R}^n \to \mathbb{R}^m$ from a higher-dimensional space to a lower-dimensional space, and such a map cannot be injective (have a left inverse) because an injective linear map preserves linear independence.

Proving left inverse for $A$


App icon is not working for iPad retina and iPad retina 64bit

Question: I have uploaded three app icons which can support both iPhone and iPad but when i am running my app in iPad retina or iPad retina 64bit its showing default app icon instead of the uploaded image.
Comment: those icons are for iPhone... every label says _iPhone_... where are the _iPad_ icons?
Answer: For iPad you need to set the App icon of Size 76X76 pixels for iOS 7.0 and above also its 2x must be present for retina


DNS errors with guest wireless

Question: Our company has a guest wireless network for auditors, contractors, etc. to use when onsite. This wireless is meant to be an external only network, so it only touches the wireless controller, and a DNS server. The DNS server is only meant to provide the IP of the capture page on the WLC. After that, all DNS requests get forwarded to OpenDNS
Traffic flows like such:
Client->DHCP->Wireless Controller(WLC)->Capture Page-> DNS-> WLC ->MPLS.
Our WLC is a Cisco 5510, which cannot host its own DHCP. The capture page works by redirecting any traffic to the DNS hostname specified on the controller, and then requiring credentials.
The issue we are running into is DNS requests failing. All active domain machines seem to work fine (keep in mind that this DNS server is not apart of our domain). Our auditors came in today to try and connect, but the DNS request was failing. When attempting to get an IP from nslookup, I received a "non-existent domain" error.
To try and replicate, I got a fresh load of windows, no domain affiliation, and attempted to connect to this network. I received no errors, and the DNS request worked fine. To further troubleshoot, i added a domain suffix to the NIC and checked the box to ensure it appended that suffix to DNS requests. Unexpectedly, that was still working.
Based on what troubleshooting I did on the auditors machine, I believe the DNS request from the wireless controller uses no suffix. this is because:
Requests to "WLC" fail with the non-existent domain error.
Requests to "WLC.domain.com" succeed.
I do have option 15 in DHCP enabled with "domain.com" (our domain name) but I have not tried removing that from DHCP. Other guests are active on the wireless and I do not want to break DHCP.
Another thing:
In DNS, the forward lookup zone is "domain.com" and the A record FQDN is "WLC.domain.com".
The reverse lookup zone is 1.1.1.adr as the capture page is on 184.108.40.206
I tried adding a new forward lookup zone with a wildcard, so it became "wlc.*" but that didnt fix it
The DNS server is hosted on a Server2012R2 box. The WLC is a Cisco 5510 as noted above
Comment: Can you add clarification?  Are you saying DHCP works as expected when you connect to the wireless network? What's your IP scheme for wireless network? Is it on the same subnet as the DNS server? Are you able to ping the controller? Are you able to ping the DNS server? Are you able to ping a public DNS (22.214.171.124 or 220.127.116.11)? What happens if you manually set the DNS server on client to a public DNS (are you able to reach the internet)?
Comment: The issue is with DNS, not DHCP, but all of this is assuming that i am already on the wireless, just not authenticated with the WLC. The capture page on the WLC does not allow any traffic through, even while connected, until authenticated on the WLC. Wireless is 172.16.67.0/24. The DNS server is the .3 address and the .1 is the router address. Manually setting the DNS servers doesnt work per above requirements
Comment: I'm not sure I'm reading the question correctly: why would you expect `nslookup WLC` to work when the DNS record you're looking for is `WLC.domain.com` ?  It seems to me that unless `domain.com` is in the client's DNS suffix search list, which is unlikely for a client belonging to another organization, that lookup *should* fail.
Comment: ... if what you're actually trying to do is to get your DNS server to serve out a host record for `wlc.` (i.e., no suffix) then I imagine someone will know how to do that, but you may need to rewrite a bit to make it clear that this is in fact what you're wanting.
Comment: Thanks Harry, that is pretty much exactly what I am looking for. I dont know why it works on some computers, because like i mentioned above, a fresh load of windows (not domain) will return a value for WLC.
Answer:
Another thing: In DNS, the forward lookup zone is "domain.com" and the A record FQDN is "WLC.domain.com". The reverse lookup zone is
  1.1.1.adr as the capture page is on 188.8.131.52

Did you realize that 220.127.116.11 is a valid public IP and is actually a public DNS service by Cloudflare and APNIC ?
Comment: I did not until now. The capture page was on that IP well before I started and it worked until now. The only difference being that the old guest wireless could touch internal services whereas the new wireless is completely segregated
Comment: The service from Cloudflare is quite new, I don't know how did APNIC used the addresss before. Anyway, it might be that before you did all the resolve internally without forwarding to 126.96.36.199, so you never saw the problem. Removing the local resolution and isolating the WiFi might make your queries go to the public 22.214.171.124 searching  for the internal domain for the hotspot, and thus, failing. Try and change the IP address for the hotspot for a private one and see what happens.
Comment: I will have to schedule some downtime for that fix but I will attempt it and let you know

DNS errors with guest wireless


How can I modify the login form using hook_form_alter()?

Question: I'm trying to alter the login form with the following code:
function git_form_alter(&$form, $form_state, $form_id) {
  $forms = array(
    'user_login_block' => 'user-login-form',
    'user_login' => 'user-login'
  );
  if (array_key_exists($form_id, $forms)) {
       //my altering code
    }
}

I got this from the openidselector module that introduces some JavaScript widgets and more or less is what I'm trying to do.
When I go to mysite/user/login it doesn't call the block where I have to alter the form.
It simply doesn't call hook_form_alter().
A different thing is with the register form, or any other form, that are calling the hook.
Can anyone tell me what can I do? How is it possible that openidselector uses the same method, but in that module the code works, while it doesn't work in my code?
Answer:
When I go to mysite/user/login it doesn't call the block where I have to alter the form. It simply doesn't call hook_form_alter().

example.com/user/login doesn't show the user login block; it shows the login form. Your code should work, even if I would rather make the code simpler (if you are not using "user-login-form" in any way).
function git_form_alter(&$form, $form_state, $form_id) {
  if ($form_id == 'user_login_block' || $form_id == 'user_login') {
    // …
  }
}

As for the reason why the code doesn't work, I can think of the following reasons:

The hook name doesn't match the short name used for the module; for example, the short name of the module is "git_form" and you are using git_form_alter() for the implementation of hook_form_alter() when it should be git_form_form_alter().
You are editing a module that is enabled. In this case, when you add a new hook to the code, you also need to flush the cache used by Drupal as Drupal caches the list of hooks implemented by third-party modules.
The module is not enabled.
Comment: in the info I've defined name = GIT and the sort name is git.
the problem was this: I was adding a new hook with the module enabled :) Clearing the cache didn't solve it, I had to disable it and enable it again . thanks so much
Comment: Thanks for pointing out that I had to Clear cache I was searching why it wasn't working and then I found your answer +1!
Answer: Your code is fine, and it works ok. At least this part that you've shown.
I assume that you've put it in 'git' module, and the module is enabled? Do you have any code (there or in any other place) that could break the flow and stop your hook from executing?
Comment: yes my module is enabled and is called git. I have other functions in it that work fine.


How to debug SpringBoot using Gradle commandline parameter, without suspending server?

Question: I'd like to start my SpringBoot application, on debug mode, using Gradle (to use on my machine the same command that is used on production)
To start my server I use
$ gradle bootRun

And I know that I can add a parameter to start on debug mode
$ gradle bootRun --debug-jvm

The problem is that the above command makes the server suspends (the server is not started until I connect the debugger)
I can have the desired behavior by adding the following code to the build.gradle, however, I'd like to use a command-line parameter, to avoid to commit the build.gradle by mistake.
bootRun {
   jvmArgs(['-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005'])
}

Note: If I add the following parameters I'll be debuging the gradle daemon (or another gradle thing), not my app, so it's not a valid solution also:
gradle bootRun -Dorg.gradle.jvmargs="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"

Note 2: the following code doesn't work - the server starts on non-debug mode
gradle bootRun -Dagentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

Related: how to debug spring application with gradle
Answer: You are basically asking how to pass variables to your project.
Gradle has many ways of doing this. I prefer the project properties approach.
Create an environmental variable like:
ORG_GRADLE_PROJECT_debug_jvm=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

gradle.build
bootRun {
   jvmArgs project.findProperty('debug_jvm') ?: ''
}

See also: Pass env variables to gradle.properties
Comment: Thanks! Solved, with: `bootRun {  if (project.hasProperty('debug')) { jvmArgs(['-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005']) } }`. Then I just need to add -Pdebug on commandline


covariant type T occurs in invariant position

Question: I'm moving my first steps in Scala and I would like to make the following code works:
trait Gene[+T] {
    val gene: Array[T]
}

The error that the compiler gives is: covariant type T occurs in invariant position in type => Array[T] of value gene
I know I could do something like:
trait Gene[+T] {
    def gene[U >: T]: Array[U]
}

but this doesn't solve the problem because I need a value: pratically what I'm trying to say is "I don't care of the inside type, I know that genes will have a gene field that return its content". (the +T here is because I wanna do something like type Genome = Array[Gene[Any]] and then use it as a wrapper against the single gene classes so I can have a heterogeneous array type)
Is it possible to do it in Scala or I'm simply taking a wrong approach? Would it be better to use a different structure, like a Scala native covariant class?
Thanks in advance!
P.S.: I've also tried with class and abstract class instead than trait but always same results!
EDIT: with kind suggestion by Didier Dupont I came to this code:
package object ga {

  class Gene[+T](val gene: Vector[T]){

    def apply(idx: Int) = gene(idx)

    override def toString() = gene.toString

  }

  implicit def toGene[T](a: Vector[T]) = new Gene(a)

  type Genome = Array[Gene[Any]]

}

package test

import ga._

object Test {
    def main(args: Array[String]) {
        val g = Vector(1, 3, 4)

        val g2 = Vector("a", "b")

        val genome1: Genome = Array(g, g2)

        println("Genome")

        for(gene <- genome1) println(gene.gene)
    }
}

So I now think I can put and retrieve data in different types and use them with all type checking goodies!
Answer: Array is invariant because you can write in it.
Suppose you do
val typed = new Gene[String]
val untyped : Gene[Any] = typed // covariance would allow that
untyped.gene(0) = new Date(...)

this would crash (the array in your instance is an Array[String] and will not accept a Date). Which is why the compiler prevents that.
From there, it depends very much on what you intend to do with Gene. You could use a covariant type instead of Array (you may consider Vector), but that will prevent user to mutate the content, if this was what you intended. You may also have an Array inside the class, provided it is decladed private [this] (which will make it quite hard to mutate the content too). If you want the client to be allowed to mutate the content of a Gene, it will probably not be possible to make Gene covariant.
Comment: Nope, I don't need to mutate it and I indeed though about Vector. My main requirement is having the client code managing an array of different Gene[T], but still with type constraints on operations. I know it's difficult and by being still at my first steps maybe I simply thinking too functional, or too dynamic, but I intend to develop something bigger and this could be a requirement: automatic boxing and unboxing of values. If you want I can rephrase the question!
Comment: Please do. What you do will not work, but not knowing what you need, it is difficult to help more. Is performance the reason why you want an Array? What is the client expected to do with an heterogenous collection of Gene[T]?
Comment: Edited my answer. Please review it because I think that thanks to your help I've found a solution. Obviously I'm going to accept your answer ;)
Comment: In your code snippet there is a typo I think. You declared `typed` but didn't use it. It should be on the second line instead of `gene` right?
Answer: The type of gene needs to be covariant in its type parameter. For that to be possible, you have to choose an immutable data structure, for example list. But you can use any data structure from the scala.collection.immutable package.


Dynamic dependent field with entity reference and organic groups

Question: I created a node type "friendly match". In this node type I got a field "club" that is an entity reference/ og reference to the "club" organic groups of the current user.
The second field named "team" should display all "team" organic groups of the club selected in the first field.
Another user gave me a hint for this module https://www.drupal.org/project/ddf
This module should allow to pass an argument from the club field to the team field. The team has use mode "views: filter by an entity reference view with dynamic arguments".
It seems that I also have to define some contextual filter in the view but I just can't figure out what filter I need to apply.
A user posted a example of the way to use the module https://www.drupal.org/node/2095179#comment-8425253
Problem is I cant adapt it to my needs. The second field always show all teams the user belongs to not just the teams of the selected club.
It would be a great help if someone could give me a hint to proceed.
Thank you all.
Answer: If I understand it correctly, you want to implement something like this
<code>Field "Club"    Field "Team"
Club A          Team A1
                Team A2
Club B          Team B1
                Team B2
</code>
So i think you have to create a club entity reference field in team OG group content type. Then create the entity reference view collecting the content type: "team OG" type, contextual filter club field, and relationship referenced entity.
i'll update my answer as soon i document the steps well :-)


How to properly call drush command from another drush command?

Question: I have a drush command that does some stuff and at the end of its execution I want to invoke twig-compile --uri=foobar.com. When I call drush_invoke('twig-compile') I get error about unknown arguments(those were passed to my drush command). I tried things like this before invoking the command drush_set_context('options', ['uri', 'foobar.com']); but no success yet.
How should I reset the options so twigc won't complain about them or how do I invoke the twigc command directly with the uri argument?
Answer: If you use drush_invoke, you should first use drush_set_option('strict', FALSE); to turn off strict option checking for the subcommand.  drush_invoke runs a subcommand in the same process, which can be more efficient; however, there are a number of ways that this can go wrong, so it is often better to use drush_invoke_process instead, to run the subcommand in its own process.  If you do this, there is no need to disable strict option handling.
drush_invoke_process('@self', 'twig-compile', $args, $options);.


How to enable VBScript for custom form?

Question: I created custom Outlook form based on Message form (class IPM.Note). I added one custom label and dropdown (now class IPM.Note.MyCustomMessage). I added simple script just to test if it is running.
Function Item_Open()
    MsgBox "Opening"
End Function

But nothing was hapenning. I tried with Item_Send too.
Then I found that it's possibly due to security settings so I modified registry to enable scripting like described here: https://support.office.com/en-us/article/custom-form-script-is-now-disabled-by-default-bd8ea308-733f-4728-bfcc-d7cce0120e94.
I also went to Trust center to Allow script in shared folders and Allow script in Public folders but still no success. Script does not run.
What else can I try?
Comment: You say 'VBScript' - is it really VBScript, or VBA?
Comment: I click ViewCode on form. There is additional button to open VisualBasic editor. Maybe I should use this?
Comment: If you're viewing the code, I would believe that you're actually using VBA (Visual Basic for Applications). I would retag your question with VBA - you're more likely to get more results that way.
Comment: VBA is something different. There are two editors.
Comment: Ah! Okay. Apologies.
Comment: Morning Piotr, I did find [this YouTube](https://www.youtube.com/watch?v=QV54mj43mzA) video, but I don't think it's quite what you're after. It looks like (after watching other form editing/creation videos) that there is a disassociation between coding the higher level forms and the VBA form demonstrated (though I can't be sure).
Answer: It turned out I didn't have macros enabled. I didn't now VBScript counts as macros. I thought macros are VBA only.

How to enable VBScript for custom form?


javax.jcr.AccessDeniedException with Apache Oak with MongoDB on JBoss Wildfly

Question: Whenever I tried to access the root node within a jcr session I encounter the exception

javax.jcr.AccessDeniedException: Root node is not accessible

We have the following setup: JBoss wildfly as application server, apache oak 1.3.2 and we manage the jcr session using JcrTemplate from Spring framework using the code:
    jcrTemplate.execute(new JcrCallback() {
                        @Override
                        public Object doInJcr(Session session) throws IOException, RepositoryException {
                            Node rootNode = session.getRootNode();
}
    }

Despite the fact that in an unit test i have no problem accessing the root node in an application server environment i'm facing an access denied.
I mention that creating a JCR session factory requires a user name and password which are delegated to the JAAS component which in an application server specific. I have already added the user name using the add-user script from JBoss. Another mention is that apache oak is using a MongoDB behind that I did no configuration regarding it in JBoss. All the connection are managed using the Java monogodb driver, that is in the application classpath.
Do you have any suggestions regarding this problem?
If you need more informations feel free to ask
Thanks
Comment: A way to grant access on the root node is to configure inside jboss, in it's configuration file a security domain that will handle the JAAS call which Apache Oak will make to get the user. I _jboss 8.2.0_ I had configured one, but seems not to take it. A work around is to configure a custom security provider for apache oak. Something like this: `Jcr jcr = new Jcr(new Oak(nodeStore));

            // configure security
            jcr.with(configureSecurityProvider());`
Answer: A way to grant access on the root node is to configure inside jboss, in it's configuration file a security domain that will handle the JAAS call which Apache Oak will make to get the user. I jboss 8.2.0 I had configured one, but seems not to take it. A work around is to configure a custom security provider for apache oak. Something like this:
Jcr jcr = new Jcr(new Oak(nodeStore));
// configure security
jcr.with(configureSecurityProvider());
Then you have to grant admin permission to your jcr user like this, in your configureSecurityProvider())  method, like this(note the PermissionConstants class with different permissions):
Set<String> adminNames = new HashSet<String>();
        adminNames.add(environment.getProperty("jcr.user"));
        ConfigurationParameters configurationParameters = ConfigurationParameters.of(PermissionConstants.PARAM_ADMINISTRATIVE_PRINCIPALS, adminNames);
        SecurityProvider securityProvider = new SecurityProviderImpl();
        AuthorizationConfigurationImpl authorizationConfiguration = AuthorizationConfigurationImpl.class.cast(       securityProvider.getConfiguration(AuthorizationConfiguration.class));
        authorizationConfiguration.setParameters(configurationParameters);


After UPS battery replacement (HP R/T 3000)

Question: has any one had experience with this UPS?
i have replaced the batteries but have't ran self test yet, if i run self test is there a risk of powering down the machines the UPS powers?
just wondering if i should run the self test out of hours or during office hours?
another issue i have is the battery capacity is showing 0% i imagine after the self test this fix's its self but just wanted to be sure.

any input would be helpful
Cheers
Answer: After a battery replacement you will need to run a calibration and also set the battery replacement date.
This will simulate the UPS going onto battery power, at around 5% the UPS should go back onto mains to re-charge. Once that happens you should get the correct readings in your GUI.
There shouldn't be any risk to the system but with anything like this make sure you assess the risk.
Comment: +1 Just do it off-hours. Particularly if you didn't get HP batteries (which come with directions, so I'm thinking he got off-name batteries - this will also validate that the batteries will even work, I've seen plenty of cheap replacement batteries that plain don't work in UPSs).
Comment: turns out when i installed the batters one of the small connectors completing the circuit had came off, i removed the batteries and reseated the contact, This resolved the issue, once the battery self test ran it worked a treat, they went official HP battery's but they were from a reputable vendor, they done the trick we now have UPS uptime.

thanks for the help


Howto: Dynamic variable name resolution in Azure DevOps YAML

Question: The consistency of Variable support & the syntax vary wildly in Azure DevOps YAML.
Case in point:
trigger:
- master

# Variable Group has $(testCategory1) with value
# 'TestCategory=bvttestonly | TestCategory=logintest'
variables:
  - group: DYNAMIC_VG

jobs:
  - job:
    pool: 'MyPool' #Has about 10+ self hosted agents

    strategy:
      parallel: $[ variables['noOfVMsDynamic']]

    variables:
      indyx: '$(testCategories$(System.JobPositionInPhase))'
      indyx2: $[ variables['indyx'] ]
      testCategories: $[ variables[ 'indyx2' ] ]

    steps:
    - script: |
        echo "indyx2 - $(indyx2)"
        echo "testCategories $(testCategories)"
      displayName: 'Display Test Categories'

The step prints:
"indyx2 - $(testCategories1)"
"testCategories $(testCategories1)"

I need to print the value of $(testCategories1) defined in the Variable Group:
'TestCategory=bvttestonly | TestCategory=logintest'
Comment: have you tried `indyx: $[ variables[format('{0}{1}', 'testCategories', variables['System.JobPositionInPhase'])] ]` ? can't really test this right now, so posting as a comment, you may need to change that `.` to something else
Comment: @Luizgrs:  I'll try & let you know.
Comment: Workaround:  Use dynamic Matrix strategy with JSON created at run time.  Refer to https://stackoverflow.com/questions/59076036/generate-job-matrix-from-all-possible-combinations-of-input-parameters/62649918#62649918
Answer: This may work to you:
variables
   indyx: $[ variables[format('{0}{1}', 'testCategories', variables['System.JobPositionInPhase'])] ]

It worked for me, in a slightly different situation which also required some dynamic variable names.
Answer:
Howto: Dynamically resolve a nested variable in Azure DevOps YAML

That because the value of nested variables (like $(testCategories$(System.JobPositionInPhase))) are not yet supported in the build pipelines at this moment.
That the reason why you always get the value $(testCategories1) rather than the real value of variable testCategories1.
I encountered this issue many times in my past posts and we do not have a perfect solution before Azure Devops supports this feature.
For the convenience of testing, I simplified your yaml like following:
jobs:
  - job: ExecCRJob
    timeoutInMinutes: 800

    pool:
      name: MyPrivateAgent

    displayName: 'Execute CR'

    variables:
      testCategories1: 123456
      testCategoriesSubscripted: $(testCategories$(System.JobPositionInPhase))

    strategy:
      parallel: $[variables['noOfVMs']]
    steps:
    - template: execute-cr.yml
      parameters:
        testCategories: $(testCategoriesSubscripted)

The execute-cr.yml:
steps:
    - script: echo ${{ parameters.testCategories }}

We always get the $(testCategories1)NOT the value of it.
If I change the $(testCategories$(System.JobPositionInPhase)) to $(testCategories1), everything work fine.
Since nested variables are not yet supported, As workaround, we need to expand the nested variables for each value of testCategories, like:
- job: B
  condition: and(succeeded(), eq(dependencies.A.outputs['printvar.skipsubsequent'], 'Value1'))
  dependsOn: A
  steps:
  - script: echo hello from B

Check the Expressions Dependencies for some more details.
Hope this helps.
Comment: Thanks, Leo for confirming the nested variables do not work.  I understand how dependencies object work.  Now, how to the hard part:  How do I simulate this with an alternative approach?  Dynamically provide values to parallel jobs that use parallel or matrix strategies?  I will only know how many agents I need at runtime through the 'noOfVMs' pipeline variable.
Comment: @VyasBharghava, hm, yes, you are right. Although we can split your tasks into separate jobs with dependencies to resolve the nested variables, we could not get the value of `$(System.JobPositionInPhase)` in one of job. This is a big challenge, I will try to check if there are other ways to solve this problem.
Comment: Thank you!!!  I did a lot of work in the last couple of days to see what works.  So, it doesn't matter where the variables are (cross job, local or even in VGs), the variable won't resolve.  I think this issue is ONLY in parallel.  Without this, I'm not able to use parallel for anything useful.  I want to distribute user supplied test categories and test concurrently OR revert all active agents in a different pool to a previous snapshot (which will upon restart will install & test a windows thick client) etc.
Comment: @VyasBharghava, Indeed, It seems that this problem is stuck in an endless loop. If we want to resolved the nested variables,  we need split the tasks into separate jobs. But, in this case, `$(System.JobPositionInPhase)` variable will be invalid.
Comment: I workaround using a matrix strategy where I hardcode all variable references (Say, biggest I expect is 20 parallel jobs), with each job in matrix directly referencing the variable: $(testCategory1)...to $(testCategory20).  I plan to test this value in a shell script task and make it skip if the value is blank.  The succeeding tasks get skipped and job ends.  However, this would mean upto 20 jobs will scheduled regardless of whether only 2 jobs are needed (meaning testCategory3 onwards are blank).  Could the product team provide an official workaround for this, please?
Comment: Also, I understand this can be unreasonable to expect more than one pass at variable dereferencing to arrive at a value:  '$(var$(index))' .   However, using the variable name as a look up key string in variables['testCategory${System.JobPositionInPhase)'] must work at least
Answer: I manage to get dynamic name resolution by using get-item to read the corresponding environment variable, allowing construction of the name of the variable and then getting the value.
In our case we save the name of an autogenerated branch into a variable group and each repository will have its own variable.
$branchVarName = "$(Build.Repository.Name).BranchName".replace(".","_")
$branchName = (get-item -Path Env:$branchVarName).value
write-host "/$(System.TeamProject)/_apis/build/builds?&repositoryId=$(Build.Repository.ID)&repositoryType=TFSGit&branchName=refs/heads/$branchName&api-version=6.0"

Notice in the second line that I reference the variable content using .value because get-item returns a name/value key pair.
This extraction has to be done in script but could be exposed as an output variable if needed in another context.
Answer: If I'm understanding your issue correctly, the problem is that the pipeline evaluates all variables at the runtime of the job. The solution in this scenario is to split your tasks into separate jobs with dependencies.
Have a look at my answer in this post and let me know if it's what you're after : YAML pipeline - Set variable and use in expression for template
Comment: I only need runtime evaluation.  The parallel strategy automatically slices the steps mentioned below into separate jobs.  I do have dependency already setup between jobs.  I'm able to access the output variables of ExecPrerequisiteJob in ExecCRJob just fine.  Only issue is the subscripted access.


Issue with CLlocationManager in iOS 8

Question: When I configure the CLlocationManager with this code:
- (void)viewDidLoad {
[super viewDidLoad];
// Do any additional setup after loading the view, typically from a nib.

self.locationManager = [[CLLocationManager alloc] init];
self.locationManager.delegate = self;
self.locationManager.desiredAccuracy = kCLLocationAccuracyHundredMeters;
if([self.locationManager respondsToSelector:@selector(requestAlwaysAuthorization)]) {
    [self.locationManager requestAlwaysAuthorization];
    // Or [self.locationManager requestWhenInUseAuthorization];
}    [self.locationManager startUpdatingLocation];

}
and the set the location delegate
#pragma mark - CLLocationManagerDelegate

- (void)locationManager:(CLLocationManager *)manager didFailWithError:(NSError *)error
{
    NSLog(@"didFailWithError: %@", error);
    UIAlertView *errorAlert = [[UIAlertView alloc]
                               initWithTitle:@"Error" message:@"Failed to Get Your Location" delegate:nil cancelButtonTitle:@"OK" otherButtonTitles:nil];
    [errorAlert show];
}

- (void)locationManager:(CLLocationManager *)manager didUpdateToLocation:(CLLocation *)newLocation fromLocation:(CLLocation *)oldLocation
{
    NSLog(@"didUpdateToLocation: %@", newLocation);
    CLLocation *currentLocation = newLocation;

    if (currentLocation != nil) {
        self.textLongitude.text = [NSString stringWithFormat:@"%.8f", currentLocation.coordinate.longitude];
        self.textLatitude.text = [NSString stringWithFormat:@"%.8f", currentLocation.coordinate.latitude];
    }
}

Don´t send the alert for authorized, but the Info.plist is complete

when go to settings and change in settings -> privacy -> localize and change the status for "always", and restart the app this status is clean.
Answer: The NSLocationAlwaysUsageDescription is misspelled in the plist.

Issue with CLlocationManager in iOS 8


concatenating local variable to parameter

Question: I am sending 2 parameters to SQL stored procedure. It works, but it doesn't when I try to concatenate a local variable with it.
This works:
exec DeansSMSDB.dbo.ADD_SMS_InQueue @CellNo, 'Dear customer, Your complaint has been registered and your complaint no. is'

but this doesn't:
exec DeansSMSDB.dbo.ADD_SMS_InQueue @CellNo, 'Dear customer, Your complaint has been registered and your complaint no. is' + @ComplaintID

I need to concatenate @ComplaintID, but how?
Comment: If you look at the syntax for [`EXECUTE`](http://msdn.microsoft.com/en-gb/library/ms188332.aspx) you'll see that you can supply a *value* or a *variable*. What's not there (and would be if what you're attempting was allowed) is an *expression*.
Answer: Create another variable:
declare @arg varchar(8000) = 'Dear customer, Your complaint has been registered and your complaint no. is' + cast(@ComplaintID as varchar(255);

exec DeansSMSDB.dbo.ADD_SMS_InQueue @CellNo, @arg;

Note that if @ComplaintId is numeric, you need to cast it to a string to use the + operator as a string concatenation operator.
Answer: You could try a two step approach in which you dump the results of the execution into a small @T temp table. And then do a quick select from that @T with concatenation:

create table tableX (input_field_name varchar(20) NULL,
                     output_field_name varchar(20) NULL)
insert tableX values ('address','addr')
create procedure testP as
select input_field_name from tableX where output_field_name = 'addr'
declare @T table (F1 varchar(20))
declare @v varchar(10) = 'my'
insert @T
exec testP
select @v + ' ' + F1 from @T


Which equal correlations of three random variables are possible?

Question: 3 random variables have equal pairwise correlation, what are the possible values of correlation?
I think 1 and 0 are the obvious answers, but are there any more, and how could I find them?
Comment: A more general version of this question is addressed at http://stats.stackexchange.com/questions/5747.
Comment: Do you know the formula to calculate correlation?
Answer: Let $X_1$, $X_2$, $X_3$ be three random variables with common pairwise correlation coefficient $\rho$, that is $\mbox{corr}(X_i, X_j)= \rho$ for $i \neq j$ with $|\rho|\leq 1$. So, the correlation matrix of $X = (X_1, X_2, X_3)$ is
$$
\left( \begin{array}{ccc}
1    & \rho & \rho \\
\rho & 1    & \rho \\
\rho & \rho & 1 \end{array} \right) .
$$
Correlation matrices need to be positive-semidefinite, which implies that their leading principal minors are all positive. So $\rho$ must satisfy the following two conditions
$$
\begin{cases}
1 - \rho^2 &\geq 0,\\
1 - 3\rho^2 + 2\rho^3 &\geq 0 .
\end{cases}
$$
The first condition is always satisfied, and the second condition implies that $\rho \geq -0.5$.
Comment: In case it's not immediately obvious to everyone $1-3\rho^2+2\rho^3 = (1-\rho)^2(1+2\rho)$
Answer: Just a side note for the otherwise correct answers already given (+1 both). The correlation/covariance matrix described is compound symmetric. This has some rather important theoretical implication on how one interpreters a model; in particular one would assume that the co-variance of the variables examined can be perfectly partitioned in a  "shared" component and the "unshared" component between your variables.
A (somewhat) common setting for such structures to be used is when the assumption for equal correlation of residuals is plausible; for example when one deals with repeated trials under the same condition in an experiment.
The CV link: "What is compound symmetry in plain english?" gives a more in-depth presentation of compound symmetry.


Where to find .dSYM file for app built with Xamarin iOS?

Question: I've built an iOS application with Xamarin and need the .dSYM to get a complete crash log. The official documentation says the file is suppose to be at ~/Library/Caches/Xamarin/mtbs/builds/. However, I do not have a folder named mtbs in ~/Library/Caches/Xamarin/. Anyone seen this before and know where to find the .dSYM-file?
https://developer.xamarin.com/guides/ios/troubleshooting/questions/symbolicate-ios-crash/
Answer: Select Release configuration and Device as deployment target and build your application.

Then you´ll find it here: AppDir/bin/iPhone/Release/{YourAppName}.app.dSYM
Answer: When building an iOS app with Visual Studio for Mac or Visual Studio 2017, the .dSYM file that's needed to symbolicate crash reports will be placed in the same directory hierarchy as your app's project file (.csproj). The exact location depends on your project's build settings:
If you have enabled device-specific builds, the the .dSYM can be found in the following directory:
/bin///device-builds/-/
For example:
TestApp/bin/iPhone/Release/device-builds/iphone8.4-11.3.1/
If you have not enabled device-specific builds, the .dSYM can be found in the following directory:
/bin///
For example:
TestApp/bin/iPhone/Release/
You can refer to this link


When sampling distribution and posterior distribution under uniform prior belong to the same family?

Question: This is similar to conjugate prior, except that here I consider if it is possible that a sampling distribution and the posterior distribution  under uniform prior belong to the same family of distributions?
In other words, when viewing $p(x|\theta)$ as the likelihood function of $\theta$ given $x$, and then normalize it to be a distribution for $\theta$, are there examples where $p(x|\theta)$ and the distribution for $\theta$ belong to the same family?
More generally, are there examples for priors other than uniform and sampling distributions, where the posteriors fall into the same family of distributions as the sampling distributions?
Thanks!
Answer: Here are two examples.

$x$ is normally distributed around mean $\theta$.  The likelihood for $\theta$ is a normal distribution with mean $x$.
$x$ is Gamma distributed with rate $\theta$.  The likelihood for $\theta$ is a Gamma distribution with rate $x$.
Comment: Thanks! What is your idea of finding the examples? Is there a name for the concept?
Comment: I don't know of any name for the concept nor any general way to find examples.


How to have upper case methods in ruby

Question: I got a chatbot in ruby that if a character is repeated five times or more in the chat, in the first and second attempt, it warns the user, at the third attempt it kicks the user and the fourth one bans the user for two hours, and it's currently working
require_relative '../plugin'

class Flood
  include Chatbot::Plugin
  match /(.*)/, :method => :check_swear, :use_prefix => false

  def initialize(bot)
    super(bot)
    @data = {}
  end

  def check_swear(user, message)
    message = message.downcase
    array = ["aaaaa", "ñññññ", "bbbbb", "ccccc", "ddddd", "eeeee", "fffff", "ggggg", "hhhhh", "iiiii", "jjjjj", "mmmmm", ".....", "*****", "?????", "!!!!!", "zzzzz", "kkkkkk", "ooooo", "nnnnn", "ppppp", "qqqqq", "rrrrr", "-----", "_____", "¨¨¨¨¨¨¨¨", "{{{{{", "}}}}}", "#####"]
    array.each do |e|
      if message.include? e
        if(@data[user.name] and @data[user.name] == 3)
          @client.send_msg "%s: [[Wiki_Freddy_Fazbear's_Pizza:Reglas_y_Lineamientos|Has sido advertido. Tendrás un ban de 2 horas.]] 4/3" % user.name
          @client.ban user.name, "7200", "Ban automático por exceso de carácteres - Si crees que esto fué un error, contacta con un [[Wiki_Freddy_Fazbear%27s_Pizza:Administradores|moderador u Admin en su muro de mensajes]]."
          @client.send_msg "!mods por si acaso consideran necesario más tiempo de ban."
          @client.kick user.name
          @data[user.name] = 0
        elsif(@data[user.name] and @data[user.name] == 2)
          @data[user.name] ||= 0
          @data[user.name] += 1
          @client.send_msg "%s: [[Wiki_Freddy_Fazbear's_Pizza:Reglas_y_Lineamientos|Por favor, no repitas carácteres, última advertencia antes de un ban.]] 3/3" % user.name
          @client.kick user.name
        elsif(@data[user.name] and @data[user.name] == 1)
          @data[user.name] ||= 0
          @data[user.name] += 1
          @client.send_msg "%s: [[Wiki_Freddy_Fazbear's_Pizza:Reglas_y_Lineamientos|Por favor, no repitas carácteres, última advertencia antes de un kick]], 2/3" % user.name
        else
          @data[user.name] ||= 0
          @data[user.name] += 1
          @client.send_msg "%s: [[Wiki_Freddy_Fazbear's_Pizza:Reglas_y_Lineamientos|Por favor, no repitas carácteres]], 1/3" % user.name
        end
      end
    end
  end
end

Now, I need to change it, being all the same except for that it does not trigger with 5 characters or more, but if 5 or more words in a message are in capital letters, can someone help me?
Edit: By the way, if someone can also help me to make it trigger not just with the characters in the array list but any character, it would be awesome
Comment: @HolgerJust I tried this https://gist.github.com/anonymous/44b7737b48a256f904add3f30cde08a3 bit it didn't worked and I already runned out of ideas :(  (About the title, how it should be named, anyway? )
Comment: So I will be warned / banned for saying _"There are other dollars beside USD, e.g. AUD, CAD, HKD and NZD"_?
Answer: Something like the following will return true if a message has 5 or more uppercase words.
def is_message_shouting?(message)
  shouted_words = 0

  message.split(' ').each do |word|
    shouted_words += 1 if word.upcase == word
  end

  shouted_words >= 5
end

puts is_message_shouting? 'THIS IS A VERY SHOUTY MESSAGE'
puts is_message_shouting? 'this is not a shouty message'
puts is_message_shouting? 'THIS IS ALSO not a shouty message'

Outputs:
true
false
false

Comment: woops, I didnt end the comment, https://gist.github.com/anonymous/f4ca4a734e55bde6a5df9585e1c29aed is like this, but when someone in the chat wrote "WunnyLove: La ultima vez que ví a KaiserGreymon4 fue hace 5 dias, 9 horas, 4 minutos and 47 segundos" it triggered, why?
Comment: Oh, wait, if you put 4 separate numbers it triggers, how can I fix it? x0
Comment: You would need to check if the word is just a number or not. You can add a method to the String class as per this suggestion and change your code to shouted_words += 1 if word.upcase == word && !word.integer?
  http://stackoverflow.com/questions/1235863/test-if-a-string-is-basically-an-integer-in-quotes-using-ruby
Answer: def shouting?(message)
  message.split.count { |word| word == word.upcase } >= 5
end

Nothing original to add, just a cleaner implementation of the same.
Answer: Something like
class String
    def is_upper?
        self == self.upcase
    end
end

def shouting?(msg)
    count = 0
    0.upto(msg.size) { |i| count += 1 if msg[i].is_upper? }
    count >= 5
end

Comment: This will find messages with 5 or more uppercase characters, but the question asks for 5 or more uppercase words.

How to have upper case methods in ruby


Counting missing files or records in a table

Question: I have a table in Big Query used for monitoring purpose. This table shows at least the following information (fields):

+--------------------------------+-----------+-----------+--------+----------------+
|           timestamp            | data_type | file_name | status | loaded records |
+--------------------------------+-----------+-----------+--------+----------------+
| 2020-04-02 19:51:11.822674 UTC | aa        | e.gz      | OK     |           2293 |
| 2020-04-02 20:00:06.261044 UTC | bb        | g.gz      | OK     |             25 |
| 2020-04-02 20:02:07.304114 UTC | cc        | cd.csv    | OK     |            298 |
| 2020-04-02 20:09:26.708910 UTC | dd        | ab.csv    | OK     |          49448 |
| 2020-04-02 20:31:00.662977 UTC | ee        | ad.csv    | OK     |           4805 |
| 2020-04-02 20:37:41.459875 UTC | ff        | f.bz      | OK     |            245 |
+--------------------------------+-----------+-----------+--------+----------------+

Timestamp is timestamp when file is received by the system
Data_type signifies what data type and source of information the system collects from data source. Data type also indicates number of files expected from each data source, and they vary from source to source. Some sources are expected to generate hourly, some daily, some every 15 minutes and some every 5 minutes.
Let's assume in the table above data_type aa and bb are produced every 5 minutes; hence the expected occurrence is 12 files per hour. Data type cc and dd are produced every hour and data_type ee and ff are produced daily.
Filename is name of the file associated with datatype. So the frequency of this file appear in the report hourly or daily depends of data type.  Just to be precise, if it is associated with data type aa and bb then there is 12 files will show per hour for that data type or 288 per day. For hourly data type such as cc and dd it is expected once per hour or 12 occurrences per day, and so on so forth.
For various reasons there are missing files observed from time to time, which will affect data quality.
I would like to count missing files of each data_type for a desired period of time; possibly hourly (applies to hourly data type), daily (for all data types)) and monthly (for all data types). This will  be visualized using visualization tools such as Google DataStudio or tableau.
Expected result (table) would be
Date(or dateTime) Data_type count_missing_files
How do I do this? new table in bq and what would be the query for this?
Can I simply do this directly with visualization with added logic instead of creating a new table? If yes, how to do this?
Any help would be greatly appreciated.
Answer: I think you could do this in Tableau. Create a calculated field showing the expected number of files. How best to do this depends on how you want to report it. If reporting hourly then this would be at the hourly level, which will sum up to daily. If reporting daily then this would be the expected number per day.
Example of hourly calculated field
CASE [Data Type]
WHEN 'aa' THEN 12
WHEN 'bb' THEN 12
...
END

If the "target" formula is correct it is then simple to calculate the difference between target record count and actual number of records with a straightforward sum. For example:
SUM([Target) - SUM([Number of Records])

Comment: This should be a very simple logic but when I applied this

CASE [Data Type]
WHEN 'aa' THEN 12
WHEN 'bb' THEN 12
...
END

in tableau, the hourly  files simply shows number of  files (records) multiplied by 12 and  if I changed this to 1 then it is just the same as number of files value for that data type instead of expected number of files for that period.  For expected daily records, it shows 3456 instead of 288.

Also, when this SUM([Target) - SUM([Number of Records]) applied of course the result is incorrect.

Any hint or suggestion?
Comment: Yes, of course, you'll have more than one record per data type. I should have thought of that...
Comment: How do you intend on showing this data? The solution may depend on how you want to show the data. You have the option of reference lines, using the target as a dimension rather than measure (i.e. a constant), blending the targets from a different data source and also getting more complex with the target calculation. Best option depends on desired output.
Comment: Actually I wanted to show the missing files for each data type in a some forms; histogram chart over time that can be aggregated as desired (i.e. hourly, daily, weekly, monthly) and  in table format (e.g. date time, data type and missing record (file)).

Also, wanted to show percentage of received records against target for each data type over desired period (i.e., daily, monthly)
Comment: I think target calculation gives me more flexibility;hence preferred option.
Comment: I owe you a response to this, apologies, I haven't had the time to look at SO much recently, will try and get to it tomorrow. It's not a 5 min solution so can't do it today unfortunately.
Answer: As you stay in your question, you can employ BigQuery to generate a table to be plotted using Data-Studio.
You can use the below query to retrieve a table with the number of expected files within a certain time-range interval denoted by the difference between <END TIMESTAMP> and <START TIMESTAMP>. I have made use of a WITH clause where the data you show in the description is recorded together with the extra field frecuency_file_generation_minutes. In this column I have added the time that takes to produce the different files (in minutes): 5 for "aa" and "bb", 60 for "cc" and "dd" and 24*60 for "ee" and "ff". Computing time_window_minutes/frecuency_file_generation_minutes provides the amount of expected files within the specified range interval.
WITH
  data AS (
  SELECT
    TIMESTAMP "2020-04-02 19:51:11.822674" AS timestamp,
    "aa" AS data_type,
    "e.gz" AS file_name,
    "OK" AS status,
    2293 AS loaded_records,
    5 AS frecuency_file_generation_minutes
  UNION ALL
  SELECT
    TIMESTAMP "2020-04-02 20:00:06.261044" AS timestamp,
    "bb" AS data_type,
    "g.gz" AS file_name,
    "OK" AS status,
    25 AS loaded_records,
    5 AS frecuency_file_generation_minutes
  UNION ALL
  SELECT
    TIMESTAMP "2020-04-02 20:02:07.304114" AS timestamp,
    "cc" AS data_type,
    "cd.gz" AS file_name,
    "OK" AS status,
    298 AS loaded_records,
    60 AS frecuency_file_generation_minutes
  UNION ALL
  SELECT
    TIMESTAMP "2020-04-02 20:09:26.708910" AS timestamp,
    "dd" AS data_type,
    "ab.gz" AS file_name,
    "OK" AS status,
    49448 AS loaded_records,
    60 AS frecuency_file_generation_minutes
  UNION ALL
  SELECT
    TIMESTAMP "2020-04-02 20:31:00.662977" AS timestamp,
    "ee" AS data_type,
    "ad.gz" AS file_name,
    "OK" AS status,
    4805 AS loaded_records,
    24*60 AS frecuency_file_generation_minutes
  UNION ALL
  SELECT
    TIMESTAMP "2020-04-02 20:37:41.459875" AS timestamp,
    "ff" AS data_type,
    "f.gz" AS file_name,
    "OK" AS status,
    245 AS loaded_records,
    24*60 AS frecuency_file_generation_minutes )
SELECT
  *,
  FLOOR(time_window_minutes/frecuency_file_generation_minutes) AS expected_files
FROM (
  SELECT
  *,
  TIMESTAMP_DIFF(<END TIMESTAMP>, <START TIMESTAMP>, MINUTE) time_window_minutes
  FROM data
  WHERE
    timestamp BETWEEN <START TIMESTAMP>
    AND <END TIMESTAMP>
  )

You can compare the expected files expected_files with the actually generated ones within a given time interval. In the output table I have included all the fields but you can adapt this query to better suit your use case.
Comment: Thanks for this tip, but it seems this query requires pre-defined time window?

What I would like to crate is simply timestamp, number of files received, number of files expected and the ratio between received and expected, but time span depends on the chart I choose; hence the criteria will adjust accordingly.

For 5 minutes file type, daily expected file is 288, but if I want to show this per month then it will adjust to number of days in that month, such as 30*288 = 8640 or 31*288=8928 files per month. Similarly for data_type that is expected hourly.
Any hint?


How do I create and synchronize product fields with entity fields on another site

Question: Case description:
I have one centralized 'Asset management' Drupal website "A" (something like a simple ERP site) which manages the administration of purchased Assets, being ECK entities. The assets are unique, no stock management required.
I also have a Drupal webshop "B" using the Drupal Commerce modules, which has Commerce Product entities.
Goal:
The assets and products need to be two-way synchronized between the two sites, in such a way that:

New Assets in site A will create new Products in site B.
Some fields (title, price, status) will need to be two-way syncronized (so that when a product is sold, the Asset status field will be changed)

Requirements
Preferably without custom coding, but using existing Drupal modules in combination with Rules to allow flexibility in configuration.
Question:

What would be a good approach in this case? Preferably without custom coding.

My current investigations:
There are multiple ways to construct a non-coding contributed module combination to get the result needed. My preference is a lightweight but configurable solution using Rules. I checked out these modules:

Services module: Without other modules this requires custom coding to import and sync entities. Most services-based modules are dependent on it.
Web Service Data module, This module allows you to bypass the default data storage location for fields and entities and instead load the data from a web service. Does not import or sync. Probably not suitable because the entities are not identical (assets and products)
Remote Entity Api module but this might not be the
most efficient way because that module creates local copies of the
remote entities. This means that in site B there would exist also the
same amount of asset entities as in site A, which are synchronized
with the Product entities.
Feeds module allows one-way import and sync, could be two-way but probably complex.
The Migrate module: large framework, is a developer's tool, must have
php skills.
Services Client module the UUID needs to be inserted per entity? Not for many entities.
Service Rules module creates endpoint for Rules Components.
Requires a services client module. Might be of use (optional, see below)

[UPDATE MAY 30 2016]
According to my research the following modules combined, will likely be the most lightweight and configurable solution. I take into account that this case is not so much about migration of data, but about creating and updating using rules triggers and actions (as per the requirements).

Web Service Client module is made to consume services without custom coding. It integrates with Rules to perform the required actions.
Web Service Client for Feeds integrates the Web Servcice Client module with Feeds, which allows for a cron-based import of created or changed entities.
Sevices Views module: in order to create a view of the assets
fields that need to be created or updated.

TO DO:
The suggested approach mentioned above needs to be tested in order to prove that it works. I still don't quite understand how to configure the Web Service Client module configuration of the 'REST service descriptions'. Probably because I am a newbie to services. In order to figure that out, I posted a question about it. I will add screenshots to that where needed.
For those answering this BONUS question: The answers have been rather broad up to now. Using the abovementioned update (may 30) please provide specific support for the solution suggested by me using the Web Service Client module configuration. This allows me to choose who gets the bonus. You may answer in the connected question in order to keep the answers on topic per question.
I have also created two test sites with the modules installed and can provide admin login details if you want to test it out, or help me out with the configuration.
Answer: I'll start with saying, you should throw your "no custom coding" requirement out. That's not to say it's not possible, but it's going to require you stitching a patchwork of solutions that might drive you to conflicting or cumbersome processes. Every data migration problem, save for the most basic (e.g. migrating content title & body nodes), will have key details and rarely has pre-made default path in Drupal.
You don't want a patchwork solution to a detailed problem like this; it will be difficult to manage if you find  your future requirements or key module assumptions change in the future. Rather, you should rely on a few key tools to handle some big needs, and fill in the rest with custom code that leverages either Drupal's or contrib module's APIs.
You have 2 broad requirements. I'll list an outline of one approach.

An ETL solution between a new custom asset entities on Site A and commerce product on Site B.

You should, from the start, utilize a shared UUID on your asset entity & Commerce product; the alternative here is having some sort of shared PK mapping between sites that could have all sorts of problems.
The go-to-solution for Drupal data ETL is the Migration framework that allows you specify how to map fields, transform them if needed, and place them into your migrated destination. The duplicate insert problem is solved with UUIDs. Commerce Migrate provides extensions specific to the Commerce project.
You'll need to determine when/how such a migration sync will occur. A solution will need to be driven by your own workflow. As such, there's probably no one-size-fits-all solution here (e.g. if assets are periodically added on scheduled date a simple cron job to migrate products in might suffice a real-time requirement might require some RPC solution).

Data Field syncing.

You should rely on Serivces (as you have mentioned) and Services Entity API to provide CRUD transport. This will allow reliable data push/pull when you need it. Additionally, it could be the framework for providing a RPC service endpoint in the event you want to trigger an asset-to-commerce migration on site B from site A.
Just like with asset-to-product ETL, your sites' workflow is probably going to determine the best course of action. It could work with Rules (with a module provide custom conditions & actions to determine when to sync up via REST) or perhaps a simple custom module running a cron job.

In short, don't rely on a patchwork of module projects to somehow solve the problem for you through trial-and-error. Pick key tools that solve one single problem well (e.g. ETL, HTTP data transport, unique identification) and use your custom code to supply the logic that's unique to your use case.
Comment: I understand your motivation to use some custom coding. However, I believe that this use case is not so unique that it can't fit into the principle of Drupal being modular and extensible, instead of comparing it with 'patchwork'. Centralized asset management, even with multiple connected webshops is very common. I believe this case can be done without coding, which makes it more manageable and transparent for administrators using the production sites. In the meantime I will now test out how far I get with your suggestions and get back later. Thanks
Comment: Maybe UUID is not necessary because the entities have unique titles in both sites.
Comment: Migration framework seems an overkill to me, see update.
Answer: Cross-site sync in Drupal 7 is non-trivial (there's a large project happening to make it easy in Drupal 8...some day), and since everything can have customizations general purpose tools have to be so flexible as to require massive effort to develop. True, there is lots of desire but rarely do two people have a shared set of requirements.
I've built a couple syncing solutions and usually at their heart they become CSV or JSON file exchanges. So while I've always used either custom code or a human to solve the hard problems to avoid duplicates and update loops, it might be possible to wire together feeds (using a JSON parser) and views (with a JSON display) or services in a way that would push information back and forth as long as it stays as simple as you suggested in your original post.
Comment: I think you actually describe what I've been trying to accomplish the last few hours, using the 'Web Services Client for Feeds' module and the 'Services Views' module, with all their dependencies. I still have trouble understanding how to configure the Web Serivces Client 'service description'.
Comment: In order to understand how the Web Services Client module works I posted a separate question in Drupal Answers two days ago: http://drupal.stackexchange.com/questions/202362/how-to-create-entities-on-another-site-using-the-web-services-client-module


Prove that $\frac{1}{1*3}+\frac{1}{3*5}+\frac{1}{5*7}+...+\frac{1}{(2n-1)(2n+1)}=\frac{n}{2n+1}$

Question: Trying to prove that above stated question for $n \geq 1$. A hint given is that you should use $\frac{1}{(2k-1)(2k+1)}=\frac{1}{2}(\frac{1}{2k-1}-\frac{1}{2k+1})$. Using this, I think I reduced it to $\frac{1}{2}(\frac{1}{n^2}-(\frac{1}{3}+\frac{1}{5}+\frac{1}{7}+\cdots+\frac{1}{2n+1}))$. Just not sure if it's correct, and what to do with the second half.
Comment: Hint:$$\frac1{(2n-1)(2n+1)}=\frac12\left(\frac1{2n-1}-\frac1{2n+1}\right)$$
Comment: This is called http://en.wikipedia.org/wiki/Telescoping_series
Answer: Hint: Multiply and divide by $2$.
What you have done is correct. All you need to observe is terms cancel out in pairs starting from $2nd$ term.
$$\frac{1}{2}(1-\frac{1}{3}+\frac{1}{3}-\frac{1}{5}+\frac{1}{5}-\frac{1}{7}+...)$$
So we are left with $1st$ and the last terms.
$$\frac{1}{2}(1-\frac{1}{2n+1})$$
$$\frac{n}{2n+1}$$
Comment: Silly little things, it's late, brain isn't on full function mode. Many thanks!
Comment: Silly brain it is ! :)
Answer: $$\frac{1}{(2k-1)(2k+1)}=\frac{1}{2}(\frac{1}{2k-1}-\frac{1}{2k+1})
$$This implies that the sum is
$$\frac{1}{2}\big\{\frac{1}{1}-\frac{1}{3}+\frac{1}{3}-\frac{1}{5}+\frac{1}{5}-...-\frac{1}{2n-1}+\frac{1}{2n-1}-\frac{1}{2n+1}\big\}$$ cancel terms and complete
Answer: For what it's worth, good ol' induction works, too -- $ \ n = 1 \ $ gives $ \ \frac{1}{3} \ $ and the sum for the $ \ (n+1) \ $ case gives
$$ \frac{n}{2n+1} \ + \ \frac{1}{(2n+1) \ (2n+3)}  \ = \ \frac{n \ (2n+3) \ + \ 1}{(2n+1) \ (2n+3)} \ = \ \frac{  2n^2 + 3n +  1}{(2n+1) \ (2n+3)}  $$
$$= \ \frac{  (2n + 1) \ (n + 1)}{(2n+1) \ (2n+3)} \ = \ \frac{ n + 1}{ 2n+3} \ = \ \frac{(n + 1)}{2(n+1) + 1} \ \ .$$