Fastest way to anonymize and sanitize field in db table?

Question: For research purposes (e.g. outsourcing to 3rd party data scientist community group), I need to export production db, masking certain sensitive fields (e.g. customer name, phone number, address, etc.).
Now, since the production db on this order_requests table has around 500 million rows, I'd like to mask field nickname to nickname_transformed. How do I do it fast?
order_requests table structure:
┌──────────────────┬─────────────────────────────┬────────────────────┐
│      Column      │            Type             │Modifiers           │
├──────────────────┼─────────────────────────────┼────────────────────┤
│ id               │ integer                     │ not null default   │
│                  │        nextval('order_requests_id_seq'::regclass)│
│ vehicle_cd       │ integer                     │                    │
│ nickname         │ character varying(255)      │                    │
│ phone_number     │ character varying(255)      │                    │
│ pickup_time      │ timestamp without time zone │                    │

... 20+ fields more ...

└──────────────────┴─────────────────────────────┴────────────────────┘
Indexes:
    "order_requests_pkey" PRIMARY KEY, btree (id)
    ... 15+ indexes more ...
Foreign-key constraints:
    ... 7 foreign keys ...
Referenced by:
    ... 25+ references more ...

My current implementation with dblink (6 hours to complete, whereas CPU on DB only < 10%; standalone db used only myself):
CREATE EXTENSION dblink;
ALTER TABLE
    order_requests ADD nickname_transformed VARCHAR;
ALTER TABLE order_requests DISABLE TRIGGER USER;
CREATE OR REPLACE FUNCTION f_update_in_steps()
  RETURNS void AS
$func$
DECLARE
   _step int;   -- size of step
   _cur  int;   -- current ID (starting with minimum)
   _max  int;   -- maximum ID
BEGIN
   SELECT INTO _cur, _max  min(id), max(id) FROM order_requests;
                                         -- 100 slices (steps) hard coded
   _step := ((_max - _cur) / 1000) + 1;  -- rounded, possibly a bit too small
                                         -- +1 to avoid endless loop for 0
   PERFORM dblink_connect('postgres://username:password@localhost:5432/dbname');  -- your foreign server as instructed above

   FOR i IN 0..2000 LOOP                 -- 2000 >> 1000 to make sure we exceed _max
      PERFORM dblink_exec(
       $$UPDATE order_requests
         SET    nickname_transformed = md5(nickname)
         WHERE  id >= $$ || _cur || $$
         AND    id <  $$ || _cur + _step || $$
         AND    true$$);  -- avoid empty update
      _cur := _cur + _step;

      EXIT WHEN _cur > _max;            -- stop when done (never loop till 200)
   END LOOP;

   PERFORM dblink_disconnect();
END
$func$  LANGUAGE plpgsql;

Some questions in my mind also:

Will it faster if I create an another table with only clone id, nickname fields + empty nickname_transformed field, do my operation there, then copy back nickname_transformed?
This article mentioned more complex way to achieve the result. Is there any solid code example for this? Something like how to manage dblink connections and link to their respective operations, notify, etc.

Have read this:

What is the fastest way to apply 150M updates on PostgreSQL table
How do I do large non-blocking updates in PostgreSQL?

Pure function / query is preferred since have to redo this job every 1-2 months.
Answer: When I need to update all rows in such a large table I use the following procedure:

drop all indexes on the table, leaving only these required by foreign keys to the table;
perform the update - as it does not need to update the indexes it's much faster;
vacuum full the table - to get back the space used up by old row versions;
recreate all the indexes - this can be done in parallel - I use xargs -P for this.

md5(nickname) does not provide sufficient anonymity - it's trivially easy to restore back the original nicknames with some brute force or rainbow tables search - they don't have enough entropy.
You should rather do substring(md5(nickname||'some-long-secret-string') for 8):

a secret would not be known by your contractor, so it would remove the ability to brute force all the nicknames;
a truncated hash would be too short for reliable reverting of nicknames even if your secret would leak out.
Comment: this one covers some essentials but not complete (e.g. having low cpu, enough memory, and single dedicated host to utilise).
Answer: Why not:
update order_requests set nickname = id;

and forget about nickname_transformed?
If you must have nickname_transformed, that maintains the same obscured value for a given nickname, then:
create table nickname_hash (
    nickname varchar(255),
    hashed char(32)
);
insert into nickname_hash
select distinct nickname from order_requests;
update nickname_hash set hashed = md5(nickname);
create index idx1 on nickname_hash(nickname, hashed);
update o set
o.nickname_transformed = h.hashed
from order_requests o, nickname_hash h
where o.nickname = h.nickname;

Comment: Have tried this one. Thanks for query but still slow.
Answer: If you want to work with realistic data, you can replace the real data with fake data with a script like https://github.com/joke2k/faker/
This one is in Python but similar scripts exists in many languages


How to use .dispatch in react redux?

Question: I have the following
CatsPage.js:
import React from 'react';
import PropTypes from 'prop-types';
import {connect} from 'react-redux';
//import * as catActions from '../../actions/catActions';
import CatList from './CatList';
import {loadCats} from '../../actions/catActions';

class CatsPage extends React.Component {
  componentDidMount () {
    this.props.dispatch(loadCats())
  }
  render() {
    return (

Cats

          <CatList cats={this.props.cats} />

    );
  }
}

CatsPage.propTypes = {
  cats: PropTypes.array.isRequired
};

function mapStateToProps(state, ownProps) {
  return {
    cats: state.cats
  };
}

export default connect(mapStateToProps)(CatsPage);

catActions.js
import * as types from './actionTypes';
import catApi from '../api/CatsApi';

export function loadCats() {
  return function(dispatch) {
    return catApi.getAllCats().then(cats => {
      dispatch(loadCatsSuccess(cats));
    }).catch(error => {
      throw(error);
    });
  };
}

export function loadCatsSuccess(cats) {
  return {type: types.LOAD_CATS_SUCCESS, cats};
}

I'm getting the following error:
Uncaught Error: Actions must be plain objects. Use custom middleware for async actions. Uncaught Error: Actions must be plain objects. Use custom middleware for async actions. at dispatch (createStore.js:166)
I'm a beginner trying to learn how to use React + Redux. What am I doing wrong that I need to fix to make the dispatch work and loadCats()?
Comment: Can you include the code of your loadCats action in your question?
Comment: @PatrickHund just added loadCats in the question. Thanks!
Answer: There's a chance you didn't properly install/configure your store.  It should look something like this:
import { createStore, applyMiddleware } from 'redux';
import thunk from 'redux-thunk';
import rootReducer from './reducers/index';

// Note: this API requires redux@>=3.1.0
const store = createStore(
  rootReducer,
  applyMiddleware(thunk)
);

How to use .dispatch in react redux?


Grab HTML table using XML

Question: I am trying to read an html table using the package XML, but even though it looks easy, I haven't managed to do it. I tried everything, but the names of the columns are always fixed by R as V1, V2, V3,…
This is the code:
require(XML)

tbl <- readHTMLTable("http://facedata.ornl.gov/ornl/npp_98-08.html",
header = c("year","ring","CO2", "stem","root","leaf","fine root", "NPP"),
skip.rows=c(1,2),colClasses=c(rep("factor",3),rep("numeric",5)))

Many thanks for your help
Answer: The first row of the table is causing trouble. It maybe easiest to remove it:
library(XML)
appURL <- "http://facedata.ornl.gov/ornl/npp_98-08.html"
doc <- htmlParse(appURL)
removeNodes(doc["//table/tr[1]"]) # remove the first row with the troublesome header
myTable <- readHTMLTable(doc, which = 1)

> head(myTable)
  Year Plot  CO2 Stem Coarse Root Leaf Fine Root Total NPP
1 1998    1 elev 1540         127  362       168      2197
2 1998    2 elev 1487         139  418       175      2219
3 1998    3  amb 1085         112  333       231      1762
4 1998    4  amb 1204         113  368       185      1870
5 1998    5  amb 1136         109  382        56      1683
6 1999    1 elev 1218          98  475       295      2086

Comment: Perfect! Many thanks!I wasn't familiar with that more developed workaround

Grab HTML table using XML


How can I decode Wordnet entites?

Question: in my experiment, I am using WordNet18 data, which contains triplets of the form (subject, predicate, object), some examples are as follows:
03964744    _hyponym    04371774
00260881    _hypernym   00260622
02199712    _member_holonym 02188065
I would like to know what the entity IDs, like 03964744, stand for. Is there anybody knows how to decode the entities?
Thank you in advance.
Answer: The 8-digit numbers you see are probably the byte offset of the entry in the data files. See http://wordnet.princeton.edu/wordnet/man/wnintro.5WN.html
After quite a bit of hunting around, I think you are looking at the numbers of WordNet 3.0 (the byte offsets for a given synset differ between versions; 3.1 is the latest version).
Your first entry seems to be saying that swing is a type of toy:
http://compling.hss.ntu.edu.sg/omw/cgi-bin/wn-gridx.cgi?usrname=&gridmode=grid&synset=04371774-n&lang=eng&lang2=eng
http://compling.hss.ntu.edu.sg/omw/cgi-bin/wn-gridx.cgi?synset=03964744-n

How can I decode Wordnet entites?


debian server fails to initialize on login load and other related problems

Question: Very good day/afternoon/night, I have finally decided to expose my problem after trying several options, but unfortunately none of them is related to my issue. The case is that we have a debian server in use, which one day stopped booting. Here I attach the screenshot image that I made at that time. Just in that one it stays for hours and hours, having to force the shutdown.
The exact moment during loading when the OS freezes
I have tried with different linux live, with rescatux, with hiren's boot cd, and in all of them, as soon as I run the mount of the main partition, the next second it crashes and freezes. And I don't know what to do to revive it, that's why I turn to this website.
P.S.: I've seen that there is a question somewhat close to this place, but as I can't get beyond mounting the partition, I can't check this file they suggest
Answer: If mounting the partition in a rescue environment causes a crash, then the partition is likely to be corrupted, and the underlying hardware might be failing.
The professional solution for that would be to restore from a good backup, after replacing hardware if necessary.
But if you don't have a backup, or you want to attempt to recover some important data that is `newer than your newest backup, here are some things you could do in a rescue environment without mounting the partition:

Use smartctl -x on the disk in question to check it's health status.

Check dmesg output to see if accessing the disk causes input/output errors

Use ddrescue to image the problem partition, then try running a filesystem check on a copy of the image (as certain kinds of corruption on some filesystem types may cause worse damage). If the filesystem check fails or makes a complete mess, use other data recovery tools on the image.

If the data on the failing partition has significant value to you, consider using professional data recovery services.

Oh, and also review your backup and restore procedures to try and improve their effectiveness.
Comment: Thank you very much for your help. The previous analysis with smartctl did not show any major problems. Either that or I am a noob with it. In the case of dmesg I can't find any reference to the disk, so I took the risk of mounting it. And in the 5 seconds I could get the above command:
EXT4-FS (sda1): recovery complete
EXT4-FS (sda1): mounted filesystem with ordered data mode. Quota mode: none
Comment: Those would be completely normal informative messages on mounting a ext4 file system.
Comment: ah ok, the thing is that it is the last one displayed, just before the whole operating system freezes. And when I say everything, I mean EVERYTHING, not even the keyboard responds, and when using a live, neither does the mouse.


Difference between full house and two pair in consideration of numbers chosen?

Question: So I am a bit confused about why the number of ways to get a full house is:
$ 13 \choose 1 $$ 12 \choose 1$$ 4 \choose 3$$ 4 \choose 2$
as opposed to:
$ 13 \choose 2 $$ 4 \choose 3$$ 4 \choose 2$.
Alternatively, with 2 pairs, why it is:
$ 13 \choose 2 $$ 4 \choose 2$$ 4 \choose 2$$4 \choose 1$$11 \choose 1$
over:
$ 13 \choose 1 $$ 12 \choose 1 $$ 4 \choose 2$$ 4 \choose 2$$4 \choose 1$$11 \choose 1$
Comment: Pretty much any website on Poker hand probabilities seems to agree. E.g. http://en.wikipedia.org/wiki/Poker_probability
Comment: No doubt; it's still nice to know what you're getting them from - readers of your question may like to see where they came from. That link should be sufficient, thanks.
Comment: Are you getting those values from somewhere?
Answer: In two pair hands, we need to avoid double counting hands by switching the places of the first two pairs. Say, $(\textrm{Ah Ad Kh Kd Qh})$ and $(\textrm{Kh Kd Ah Ad Qh})$ are the same hand. With full houses, there is no such symmetry, i.e., $\textrm{AAAKK}$ and $\textrm{KKKAA}$ are not the same hand (with any suits of the cards).
The equation for the number of full houses you give can be derived as follows:

The rank of the trips can be selected in $13 \choose 1$ ways.
Given (1), the rank of the pair can be selected in $12 \choose 1$ ways.
Given (1,2), the suits of the trips can be selected in $4 \choose 3$ ways.
Given (1,2,3), the suits of the trips can be selected in $4 \choose 2$ ways.

Your alternative equation with the factor $13 \choose 2$ would in place of 1,2 just count the number of ways to select the rank of the trips and the rank of the pair without caring about which is which. This would "half-count" the number of hands (by considering e.g. selection of $\textrm{AAAKK}$ and $\textrm{KKKAA}$ to produce the same hand in the first step).
For two pair hands, the number of hands can be obtained as follows:

The ranks of the pairs can be selected as $13 \choose 2$ ways
Given 1, the rank of the fifth card can be selected in $11 \choose 1$ ways
Given (1,2), the suits of the higher pair can be selected in  $4 \choose 2$ ways
Given (1,2,3), the suits of the lower pair can be selected in $4 \choose 2$ ways
Given (1,2,3,4), the rank of the fifth card can be selected in $4 \choose 1$ ways

If we replace the first factor by your alternative, ${13 \choose 1}\, {12 \choose 1}$, this would count the number of ways of selecting the rank of the first pair and then the number of ways of selecting the rank of the second pair. But, there is no first and second pair, so this would double-count the number of hands (by considering, e.g., selection of $\textrm{AAKK}$ and $\textrm{KKAA}$ in the first step to be different).
An exercise
Similar thing occurs for example in 3-card hands when counting the number of pairs (pair and third card are distinct) and the number of no-pairs (symmetry between the three different ranks). To further convince yourself, you might derive similar equations for the number of pair and no-pair 3-card hands with some small deck (say, 3 ranks and 2 suits per rank) where you can also enumerate all the combinations by hand.


Perfect matching in a bipartite graph avoid increasing perfect matchings?

Question: Let $A=\{a_1,\dots,a_n\}$ and $B=\{b_1,\dots, b_n\}$ be two disjoint vertex sets.
Let $A_i=\{a_1,\dots, a_i\}$ be the set of first $i$ vertices of $A$. And $B_i=\{b_1,\dots, b_i\}$, similarly.
For $1\le i_1\le i_2\le\dots\le i_k\le n$, suppose we have $M_j$ that is a perfect matching between $A_{i_j}$ and $B_{i_j}$ for each $1\le j\le k$ satisfying
$$M_j\cap(\cup_{1\le \ell<j}M_{\ell})=\emptyset,$$
i.e., those $M_j$ are edge-disjoint.
Suppose the degree of each vertex of $A_{i_k}$ in $\cup_{1\le j\le k}M_j$ is strictly less than $i_k$. Then is it true that there is still a perfect matching between $A$ and $B$ that is edge-disjoint with $\cup_{1\le j\le k}M_j$?
I tried some small examples where the statement is true.
Answer: No, that is false. Consider $i_1 = i_2 = 2$ and $i_3 = 3$. Then $\{\,M_1, M_2\,\} = \{\,\{\,a_1b_1, a_2b_2\}, \{\,a_1b_2, a_2b_1\,\}\,\}$. And $M_3$ should contain at least one of these $4$ edges to be a perfect matching.
Comment: The vertices in $M_1\cup M_2$ already have degree $2=i_2$. This example is why I add the requirement that the degrees are less than $i_k$. But it seems still not enough.
Answer: It is false: let $M_1=\{a_1b_1,a_2b_2\}$ and $M_2=\{a_1b_2,a_2b_1,a_3b_3\}$ (so that $i_1=2,i_2=3$). If $n=3$, there is no perfect matching out of $M_1\cup M_2$, although each vertex has degree at most 2, which is less than 3.


Hadoop on Ubuntu - two different install directories?

Question: I recently installed Hadoop 1.0.3 from the .deb provided by Apache. The package installed correctly, but there seem to be two directories that have Hadoop-related files: /usr/share/hadoop has jars and the site configuration files, while /etc/hadoop has hadoop-env.sh and some other XMLs; all of the Hadoop literature I've read thus far seems to assume there is only a single Hadoop install directory; could someone please elaborate on these two directories and their respective purposes?
Thanks.
Answer: /usr/share is a platform-independent shared data directory (and hadoop is in java...) and /etc/hadoop is mostly host specific system and application configuration files.
Then, to configure hadoop, just use the following command and answer yes, yes, yes... !
sudo ./usr/sbin/hadoop-setup-single-node.sh

Then
sudo ./usr/sbin/hadoop-validate-setup.sh --user=hdfs

Complete tutorial for single node
Comment: Hi Yohann,

Thanks for your answer. Would it be accurate to say that /etc/hadoop is $HADOOP_HOME, and if so, why does it not include core-site.xml, mapred-site.xml, and hdfs-site.xml? Should I create a conf directory and copy /usr/share/hadoop/templates/conf/*-site.xml to /etc/hadoop/conf/ ?
Comment: See the edit in my post.
Comment: very helpful for me as well!


How prove this$\sum_{i=0}^{m-1}\binom{n-1+i}{i}x^ny^i+\sum_{j=0}^{n-1}\binom{m-1+j}{j}x^my^j=1$

Question: let $m,n$ be positive numbers,and $x,y>0$ such $x+y=1$,show that
$$\sum_{i=0}^{m-1}\binom{n-1+i}{i}x^ny^i+\sum_{j=0}^{n-1}\binom{m-1+j}{j}x^jy^m=1$$
My try:
$$\sum_{i=0}^{m-1}\binom{n-1+i}{i}x^ny^i=\sum_{i=0}^{m-1}\binom{n-1+i}{i}x^n(1-x)^i=\sum_{i=0}^{m-1}\binom{n-1+i}{i}x^n\sum_{k=0}^{i}(-1)^k\binom{i}{k}x^k$$
Answer: The exponents are reversed in the second summation: it should be
$$\sum_{j=0}^{n-1}\binom{n-1+j}jx^jy^m\;.$$
(You can see that the version in the problem is wrong by actual calculation with $m=n=2$ and $x=\frac14$, for instance.)
Think of $x$ as the probability of a success and $y$ as the probability of failure in a Bernoulli trial. The experiment consists of $m+n-1$ independent Bernoulli trials each with success probability $x$. Then
$$\binom{n-1+i}ix^ny^i$$
is the probability of getting the $n$-th success on the $(n+i)$-th trial (why?), so
$$\sum_{i=0}^{m-1}\binom{n-1+i}ix^ny^i$$
is the probability of getting at least $n$ successes. Similarly,
$$\sum_{j=0}^{n-1}\binom{n-1+j}jx^jy^m$$
is the probability of getting at least $m$ failures. To complete the proof, show that exactly one of these two events must occur.
Comment: Nice,Thank you,+1 I think this problem have without probability solution.
Comment: @nanchangjian: You're welcome. It probably does have an algebraic solution, but I didn't see one.
Comment: An algebraic proof using complex variables is now included on this page.
Answer: Suppose we seek to show that
$$\sum_{q=0}^{m-1} {n-1+q\choose q} x^n (1-x)^q
+ \sum_{q=0}^{n-1} {m-1+q\choose q} x^q (1-x)^m = 1$$
where $n,m\ge 1.$
We will evaluate  the second term by a contour  integral and show that
is equal to one minus the first term which is the desired result.
Introduce the Iverson bracket
$$[[0\le q\le n-1]]
= \frac{1}{2\pi i}
\int_{|z|=\epsilon} \frac{z^q}{z^n}
\frac{1}{1-z} \; dz.$$
With this bracket we may extend the sum in $q$ to infinity to get
$$\frac{1}{2\pi i}
\int_{|z|=\epsilon} \frac{1}{z^n}
\frac{1}{1-z}
\sum_{q\ge 0} {m-1+q\choose q} z^q x^q (1-x)^m\; dz
\\ = \frac{(1-x)^m}{2\pi i}
\int_{|z|=\epsilon} \frac{1}{z^n}
\frac{1}{1-z}
\sum_{q\ge 0} {m-1+q\choose q} z^q x^q \; dz
\\ = \frac{(1-x)^m}{2\pi i}
\int_{|z|=\epsilon} \frac{1}{z^n}
\frac{1}{1-z}
\frac{1}{(1-xz)^m} \; dz.$$
Now we  have three poles here at  $z=0$ and $z=1$ and  $z=1/x$ and the
residues at these  poles sum to zero, so we  can evaluate the residue
at  zero  by computing  the  negative of  the  residues  at $z=1$  and
$z=1/x.$
Observe that the  residue at infinity is zero as can  be seen from the
following computation:
$$-\mathrm{Res}_{z=0} \frac{1}{z^2}
z^n \frac{1}{1-1/z}\frac{1}{(1-x/z)^m}
\\ -\mathrm{Res}_{z=0} \frac{1}{z^2}
z^n \frac{z}{z-1}\frac{z^m}{(z-x)^m}
\\ -\mathrm{Res}_{z=0}
z^{n+m-1} \frac{1}{z-1}\frac{1}{(z-x)^m} = 0.$$
Returning to the main thread the residue at $z=1$ as seen from
$$- \frac{(1-x)^m}{2\pi i}
\int_{|z|=\epsilon} \frac{1}{z^n}
\frac{1}{z-1}
\frac{1}{(1-xz)^m} \; dz.$$
is $$-(1-x)^m \frac{1}{(1-x)^m} = -1.$$
For the residue at $z=1/x$ we consider
$$\frac{(1-x)^m}{x^m \times 2\pi i}
\int_{|z|=\epsilon} \frac{1}{z^n}
\frac{1}{1-z}
\frac{1}{(1/x-z)^m} \; dz
\\ = \frac{(-1)^m (1-x)^m}{x^m \times 2\pi i}
\int_{|z|=\epsilon} \frac{1}{z^n}
\frac{1}{1-z}
\frac{1}{(z-1/x)^m} \; dz.$$
and use the following derivative:
$$\frac{1}{(m-1)!}
\left(\frac{1}{z^n} \frac{1}{1-z}\right)^{(m-1)}
\\ = \frac{1}{(m-1)!}
\sum_{q=0}^{m-1} {m-1\choose q}
\frac{(-1)^q (n+q-1)!}{(n-1)! z^{n+q}}
\frac{(m-1-q)!}{(1-z)^{m-q}}
\\ =
\sum_{q=0}^{m-1} \frac{1}{q!}
\frac{(-1)^q (n+q-1)!}{(n-1)! z^{n+q}}
\frac{1}{(1-z)^{m-q}}
\\ =
\sum_{q=0}^{m-1} {n+q-1\choose q}
\frac{(-1)^q}{z^{n+q}}
\frac{1}{(1-z)^{m-q}}.$$
Evaluate this at $z=1/x$ and multiply by the factor in front to get
$$\frac{(-1)^m (1-x)^m}{x^m} \times
\sum_{q=0}^{m-1} {n+q-1\choose q}
(-1)^q x^{n+q}
\frac{1}{(1-1/x)^{m-q}}
\\ = \frac{(-1)^m (1-x)^m}{x^m} \times
\sum_{q=0}^{m-1} {n+q-1\choose q}
(-1)^q x^{n+q}
\frac{x^{m-q}}{(x-1)^{m-q}}
\\ = (-1)^m (1-x)^m \times
\sum_{q=0}^{m-1} {n+q-1\choose q}
(-1)^q x^{n} (-1)^{m-q}
\frac{1}{(1-x)^{m-q}}
\\ =
\sum_{q=0}^{m-1} {n+q-1\choose q}
x^{n} (1-x)^q.$$
This yields for the second sum term the value
$$1 - \sum_{q=0}^{m-1} {n+q-1\choose q}
x^{n} (1-x)^q$$
showing that when we add the  first and the second sum by cancellation
the end result is one, as claimed.
This identity generalizes an identity by Gosper to be found at this MSE link.
Remark Fri Jun 9 2017. As written this proof requires $|1/x| \gt \epsilon$ or $1/\epsilon \gt |x|$ for the pole at $1/x$ to be outside the circular contour and for the geometric series to converge. Note however that these are polynomials in $x$ of degree $n+m-1$ and hence this is sufficient to show they agree for all $x.$


ReactJS rendering multiple UI components via map iteration

Question: So I looked around and haven't really found a solution for this yet. I want to render multiple UI elements via the .map() function in React from retrieved data set through network request. I got as far as being able to populate the data value in a simple string but I can't seem to do it by calling the UI from a separate function. Here is what I have:
readMode: function(item) {
    return (

item.itemNo

item.itemRef

item.itemColor

    )
},

render: function() {
    return (

            {this.state.inventory.map(function(item){
                return this.readMode(item);
            })}

    )
}

ReactDOM.render(
    <Inventory />,
   document.getElementById("container")
);

I think I'm getting really close, bypassing all the errors but I'm left with this one at the end: TypeError: Cannot read property 'readMode' of undefined at eval
Answer: Since you're using a plain JavaScript function in your map arguments, you are creating a new context for that function. Because of this, the this value does not refer to your class but to that function's context. Two solutions are:
1) bind this to the existing map function: (function(item){ //logic}).bind(this)
2) the better alternative assuming you can use ES6, use arrow functions. Arrow functions inherit context from the parent so you automatically solve the issue:
this.state.inventory.map(item => this.readMode(item))

Answer: Addition to @ZekeDroid's answer, you can write it in more simpler way like this:
readMode: function() {
    return this.state.inventory.map((item, i)=>{
        return (

{item.itemNo}

{item.itemRef}

{item.itemColor}

        )
    })

},

render: function() {
    return (

            {this.readMode()}

    )
}

ReactDOM.render(
    <Inventory />,
   document.getElementById("container")
);

Answer: i don't think that you need to bind anything. You can just declare your function outside the component, because you don't need 'this':
const readMore = (item) => { return
{item.name}
; }

then you can just use:
this.state.inventory.map(() => {return readMore(item); });


Premature end of training in TF OD 2 API

Question: I've been playing with the Tensorflow Object Detection API 2(TF OD 2) in these days, I'm using the git head  commit ce3b7227. My aim is to find the most suitable model for my custom dataset, by using the existent DL Architecture present in theTensorFlow 2 Model Zoo. I've generated my TF Records with the following tutorial of Roboflow and I have been training it with my Laptop and Google Colab, in GPU Mode.
I've found this amazing Roboflow's Colab Notebook, while I've tried to reproduce the same steps with my dataset, by using the models/research/object_detection/model_main_tf2.py, unluckly for me, the training script always ends before it started to iterate. It didn't show any Python Error and also it show some warnings as usual. The complete output is in my Colab Notebook
I'm fine-tuning the model with the following commands.
 PIPELINE_CONFIG_PATH=models/ssd_resnet152_v1_fpn_640x640_coco17_tpu-8/pipeline.config; MODEL_DIR=training/; NUM_TRAIN_STEPS=10000; SAMPLE_1_OF_N_EVAL_EXAMPLES=1;

 python models/research/object_detection/model_main_tf2.py --model_dir=$MODEL_DIR --num_train_steps=$NUM_TRAIN_STEPS --sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES --pipeline_config_path=$PIPELINE_CONFIG_PATH --alsologtostderr

This is my pipeline.config file
model {
  ssd {
    num_classes: 90
    image_resizer {
      fixed_shape_resizer {
        height: 640
        width: 640
      }
    }
    feature_extractor {
      type: "ssd_resnet152_v1_fpn_keras"
      depth_multiplier: 1.0
      min_depth: 16
      conv_hyperparams {
        regularizer {
          l2_regularizer {
            weight: 0.00039999998989515007
          }
        }
        initializer {
          truncated_normal_initializer {
            mean: 0.0
            stddev: 0.029999999329447746
          }
        }
        activation: RELU_6
        batch_norm {
          decay: 0.996999979019165
          scale: true
          epsilon: 0.0010000000474974513
        }
      }
      override_base_feature_extractor_hyperparams: true
      fpn {
        min_level: 3
        max_level: 7
      }
    }
    box_coder {
      faster_rcnn_box_coder {
        y_scale: 10.0
        x_scale: 10.0
        height_scale: 5.0
        width_scale: 5.0
      }
    }
    matcher {
      argmax_matcher {
        matched_threshold: 0.5
        unmatched_threshold: 0.5
        ignore_thresholds: false
        negatives_lower_than_unmatched: true
        force_match_for_each_row: true
        use_matmul_gather: true
      }
    }
    similarity_calculator {
      iou_similarity {
      }
    }
    box_predictor {
      weight_shared_convolutional_box_predictor {
        conv_hyperparams {
          regularizer {
            l2_regularizer {
              weight: 0.00039999998989515007
            }
          }
          initializer {
            random_normal_initializer {
              mean: 0.0
              stddev: 0.009999999776482582
            }
          }
          activation: RELU_6
          batch_norm {
            decay: 0.996999979019165
            scale: true
            epsilon: 0.0010000000474974513
          }
        }
        depth: 256
        num_layers_before_predictor: 4
        kernel_size: 3
        class_prediction_bias_init: -4.599999904632568
      }
    }
    anchor_generator {
      multiscale_anchor_generator {
        min_level: 3
        max_level: 7
        anchor_scale: 4.0
        aspect_ratios: 1.0
        aspect_ratios: 2.0
        aspect_ratios: 0.5
        scales_per_octave: 2
      }
    }
    post_processing {
      batch_non_max_suppression {
        score_threshold: 9.99999993922529e-09
        iou_threshold: 0.6000000238418579
        max_detections_per_class: 100
        max_total_detections: 100
        use_static_shapes: false
      }
      score_converter: SIGMOID
    }
    normalize_loss_by_num_matches: true
    loss {
      localization_loss {
        weighted_smooth_l1 {
        }
      }
      classification_loss {
        weighted_sigmoid_focal {
          gamma: 2.0
          alpha: 0.25
        }
      }
      classification_weight: 1.0
      localization_weight: 1.0
    }
    encode_background_as_zeros: true
    normalize_loc_loss_by_codesize: true
    inplace_batchnorm_update: true
    freeze_batchnorm: false
  }
}
train_config {
  batch_size: 8
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  data_augmentation_options {
    random_crop_image {
      min_object_covered: 0.0
      min_aspect_ratio: 0.75
      max_aspect_ratio: 3.0
      min_area: 0.75
      max_area: 1.0
      overlap_thresh: 0.0
    }
  }
  sync_replicas: true
  optimizer {
    momentum_optimizer {
      learning_rate {
        cosine_decay_learning_rate {
          learning_rate_base: 0.03999999910593033
          total_steps: 25000
          warmup_learning_rate: 0.013333000242710114
          warmup_steps: 2000
        }
      }
      momentum_optimizer_value: 0.8999999761581421
    }
    use_moving_average: false
  }
  fine_tune_checkpoint_version: V2
  fine_tune_checkpoint: "models/ssd_resnet152_v1_fpn_640x640_coco17_tpu-8/checkpoint/ckpt-0"
  num_steps: 25000
  startup_delay_steps: 0.0
  replicas_to_aggregate: 8
  max_number_of_boxes: 100
  unpad_groundtruth_tensors: false
  fine_tune_checkpoint_type: "classification"
  use_bfloat16: true
}
train_input_reader {
  label_map_path: "datasets/UrbanTracker/urban_tracker_label_map.pbtxt"
  tf_record_input_reader {
    input_path: "datasets/UrbanTracker/urban_tracker_train.record"
  }
}
eval_config {
  metrics_set: "coco_detection_metrics"
  use_moving_averages: false
}
eval_input_reader {
  label_map_path: "datasets/UrbanTracker/urban_tracker_label_map.pbtxt"
  shuffle: false
  num_epochs: 1
  tf_record_input_reader {
    input_path: "datasets/UrbanTracker/urban_tracker_test.record"
  }
}

And this is how my model's directory looks like.
.
├── datasets
│   ├── raccoon
│   │   ├── raccoon_label_map.pbtxt
│   │   ├── raccoon_test.record
│   │   └── raccoon_train.record
│   ├── readme.md
│   └── UrbanTracker
│       ├── labels_urbantracker.txt
│       ├── urban_tracker_label_map.pbtxt
│       ├── urban_tracker_test.record
│       └── urban_tracker_train.record
├── __main__.py
├── models
│   ├── AUTHORS
│   ├── efficientdet_d1_coco17_tpu-32
│   │   ├── checkpoint
│   │   │   ├── checkpoint
│   │   │   ├── ckpt-0.data-00000-of-00001
│   │   │   └── ckpt-0.index
│   │   ├── pipeline.config
│   │   ├── saved_model
│   │   │   ├── assets
│   │   │   ├── saved_model.pb
│   │   │   └── variables
│   │   │       ├── variables.data-00000-of-00001
│   │   │       └── variables.index
│   ├── faster_rcnn_resnet101_v1_640x640_coco17_tpu-8
│   │   ├── checkpoint
│   │   │   ├── checkpoint
│   │   │   ├── ckpt-0.data-00000-of-00001
│   │   │   └── ckpt-0.index
│   │   ├── pipeline.config
│   │   ├── saved_model
│   │   │   ├── saved_model.pb
│   │   │   └── variables
│   │   │       ├── variables.data-00000-of-00001
│   │   │       └── variables.index
│   └── ssd_resnet152_v1_fpn_640x640_coco17_tpu-8
│       ├── checkpoint
│       │   ├── checkpoint
│       │   ├── ckpt-0.data-00000-of-00001
│       │   └── ckpt-0.index
│       ├── pipeline.config
│       ├── saved_model
│       │   ├── assets
│       │   ├── saved_model.pb
│       │   └── variables
│       │       ├── variables.data-00000-of-00001
│       │       └── variables.index
├── tools
│   ├── parse_polytrack.py
│   ├── polytrack_csv_to_tfrecord.py
│   ├── raccoon_labels_test.csv
│   ├── raccoon_labels_train.csv
│   ├── split_dataset.py
│   ├── urban_tracker_test.csv
│   └── urban_tracker_train.csv

I've converted my Dataset to TFRecord with the TF v1 and v2 API. Also, I've been playing with different training parameters without luck. To check my dataset, in case I've generated it wrongly, I tried with another dataset, the basic Raccoon Dataset but I got the same result.
Thank you for your attention.
Answer: Solved: For models such as efficientdet_d1_coco17_tpu-32 just change in the pipeline.config the parameter   from fine_tune_checkpoint_type: "classification" to   fine_tune_checkpoint_type: "detection", check TF Github


OSGi fragment bundle merge of headers

Question: If I create a fragment bundle and add a new header entry under the maven-bundle-plugin instractions. Will that new entry get merged with the host bundle headers list at the runtime?..
Comment: You can try it out in a couple of minutes and answer your own question ;-).
Comment: I tried that before posting the question and host bundle header list doesn't contain that, so just want to make sure its the default behavior or a mistake in my fragment bundle.. :)
Comment: Ok. I think your fragment bundle was good.
Answer: A bundle shares its classloader with its fragment bundles. That is all.
Bundle.getHeaders() always shows the headers defined for that specific bundle. The headers of fragment bundles are not merged to the headers of host bundle.

OSGi fragment bundle merge of headers


Mixed models in R : Compare measurements over several time points with missing data in 3 populations

Question: I have data that look like the example below. There are 3 different groups (g1, g2 and g3) of subjects which needed either 1, 2 or 3 procedures (p1, p2, p3) respectively according to the course of the disease. Hence, there is only p1 for the group g1; p1, p2 for the group g2; p1, p2, p3 for the group g3. There are several measurements (8 in total for the real data): some of which are continuous (such y1 in the example below) other dichotomous (such as y2 in the example below), with missing data (NA). Each measurement was performed at the time of each procedure. Moreover there is the date at which each procedure was performed and the timespan in days between each consecutive procedure of each procedure. I'm trying to compare the mean of y1 and the frequency of y2 between the several time points of the procedure.
Here is a representative simulation of the data:
library(lubridate)
library(dplyr)
library(lme4)
library(lmerTest)
library(MASS)
library(multcomp)

# define the number of subjects in each groups
n_g3 = 20
n_g2 = n_g3 * 2
n_g1 = n_g3 * 5

# 3 different groups
id_p1 = paste0("ID",1:c(n_g1 + n_g2 + n_g3))
id_p2 = paste0("ID",c(n_g1+1):c(n_g1 + n_g2 + n_g3))
id_p3 = paste0("ID",c(n_g1 + n_g2+1):c(n_g1 + n_g2 + n_g3))
id = append(append(id_p1, id_p2), id_p3)

# 3 different groups
groups = c(rep("g1", n_g1), rep("g2", n_g2), rep("g3", n_g3), rep("g2", n_g2), rep("g3", n_g3), rep("g3", n_g3))

# 1st, 2nd or 3rd procedure
procedure = c(rep("p1", n_g1), rep("p1", n_g2), rep("p1", n_g3), rep("p2", n_g2), rep("p2", n_g3), rep("p3", n_g3))

# date of procedure and timespan between procedures
date_p1 = as.Date(as.Date(rep("2012-01-30",c(n_g1 + n_g2 + n_g3))) - replicate(c(n_g1 + n_g2 + n_g3), 40*rnorm(1)) %>% round())
timespan_p1_to_px = c(rep(0, n_g1 + n_g2 + n_g3),
                  replicate(c(n_g2),500 +(40*rnorm(1))) %>% round(digit=0),
                  replicate(c(n_g3),650 +(40*rnorm(1))) %>% round(digit=0),
                  replicate(c(n_g3),1500 +(40*rnorm(1))) %>% round(digit=0))

date_construct = c(date_p1, date_p1[(n_g1+1):(n_g1 + n_g2 + n_g3)], date_p1[(n_g1 + n_g2+1):(n_g1 + n_g2 + n_g3)])

date_px = as.Date(as.Date(as.Date(date_construct)) + timespan_p1_to_px)

# measurement n°1 (continuous)
y1 = c(
  (replicate(c(n_g1 + n_g2 + n_g3), 20+(3*rnorm(1))) %>% round(digit = 1)) %>% abs(),
  (replicate(c(n_g2 + n_g3), 25+(3*rnorm(1))) %>% round(digit = 1)) %>% abs(),
  (replicate(n_g3, 26+(3*rnorm(1))) %>% round(digit = 1)) %>% abs()
  )
    y1[rbinom(c(n_g1 + n_g2 + n_g3),1,0.2) == 1] = NA

# measurement n°2 dichotomous
y2 = c(
  (rbinom(c(n_g1 + n_g2 + n_g3),1,0.8)),
  (rbinom(c(n_g2 + n_g3),1,0.5)),
  (rbinom(n_g3,1,0.3))
  )

y2[rbinom(c(n_g1 + n_g2 + n_g3),1,0.2) == 1] = NA

data = data.frame(id, groups, procedure, date_px, timespan_p1_to_px, y1, y2)

### create subset of data #####

# subset p1 to make the comparisons of all the variable at baseline
data_p1 = data[which(data$procedure == "p1"),]

# subset g2 to make comparisons within the group with 2 procedures
data_g2 = data[which(data$groups == "g2"),]

# subset g3 to make comparisons within the group with 3 procedures
data_g3 = data[which(data$groups == "g3"),]

QUESTIONS
I have the 5 following questions:
Question N°1: I would like to compare the baseline measurement at p1 for g1, g2 and g3. Those are three different groups with all different individuals. I would compare the mean of each group with an ANOVA as follow:
Question N°2: I would like to compare if there is a difference in the measurements within the group g2 between time p1 and p2. For this I think I should use a linear mixed model. I would use the following command:
Question N°3: I would like to compare if there is a difference in the measurements within the group g3 between time p1 and p2 and between p2 and p3. For this I think I should use a linear mixed model. I would use the following command:
Question N°4: How should i deal with a dichotomous variable of time y2. should I use the following lines:
Question N°5: Finally: Would it be possible to accound for the timespan elapsed between p1 and p2 in g2 and g3, and between p2 and p3 in the group g3? if yes what would be the line of code
Comment: Using a mixed model would indeed be appropriate in this case because it would give you unbiased results under the missing at random missing data mechanism (provided that the random-effects structure is correctly/flexibly specified). From your description it is was clear if you want to separately analyze y1 and y2 or jointly (e.g., are you interested in the association between y1 & y2?). And moreover why you have missing data (i.e., is it missing completely at random or missing at random)?
Comment: Thank you @DimitrisRizopoulos for your answer. So to answer your questions: 1. I am not particularly interested in association between `y1` and `y2`. What I want to do is if there is relevant difference between `y1_t1` and `y1_t2` and `y1_t3` (as well as for `y2`) between the different time points. 2. I have missing data that I assume being completely at random. Do you know some code (e.g. using `lme4`) that could model this question?
Comment: @DimitrisRizopoulos. I changed the display of the data and made the questions more precise with an attempt to answer them
Answer: Answer to question N°1:
Using an ANOVA:
mod1 = aov(formula = y1~groups, data = data_p1)
TukeyHSD(mod1)

or using a linear model and the multcomp package as described in this stackoverflow question:
mod1_2 = lm(y1~groups, data = data_p1)
mod1_2 = glht(mod1_2, linfct = mcp(groups = "Tukey"))
summary(mod1_2)

Answer to question N°2:
Using the lmerTest to get the p value in summary of the regression:
mod2 = lmer(y1 ~ procedure + (1|id), data = data_g2)
summary(mod2)

Answer to question N°3:
Performing a linear mixed models with id as random variable and performing a Tukey post-hoc test to look at all the between groups comparison using the multcomp package, as shown on this stack exchange question
mod3 = lmer(y1 ~ procedure + (1|id), data = data_g3)
summary(mod3)
summary(glht(mod3, linfct = mcp(procedure = "Tukey")), test = adjusted("holm"))

Answer to question N°4:
mod4 = glmer(y2 ~ procedure + (1|id), data = data_g2, family = binomial)
summary(mod4)

mod4_2 = glmer(y2 ~ procedure + (1|id), data = data_g3, family = binomial)
summary(mod4_2)

Answer to question N°5:
I'm really not sure about this one
mod5 = lmer(y1 ~ timespan_p1_to_px + (1|id) + (1|procedure), data = data_g3)
summary(mod5)


Contar o número de ocorrências de um valor numa list

Question: Tenho uma list com os seguintes valores:
numeros = [5, 3, 1, 2, 3, 4, 5, 5, 5]

Em Python teria alguma função para contar quantas vezes algum valor se repete?
Por exemplo: Quero saber quantas vezes o 5 repetiu.
Answer: Usando Python 2 ou 3
Só usar o método count()
numeros = [5, 3, 1, 2, 3, 4, 5, 5, 5]
numeros.count(5)

Comment: Eita, minimalista! Python ♥
Comment: No Python 2.7 também funcionou :p
Comment: Sim. Acabei de ver que funciona pros dois
Answer: Pode usar o Counter:
import collections
numeros = [5, 3, 1, 2, 3, 4, 5, 5, 5]
repetidos = collections.Counter(numeros)
print(repetidos[5])

Veja funcionando no ideone. E no repl.it. Também coloquei no GitHub para referência futura.
Se preferir a forma curta:
print(collections.Counter([5, 3, 1, 2, 3, 4, 5, 5, 5])[5])

Comment: @LuizAngioletti "bastante mais geral" se refere a poder iterar e pegar a quantidade de repetições de cada um? (essa me parece a unica diferença de ambas)
Comment: @LuizAngioletti, talvez o caso do AP seja só de fazer a contagem de um único número el específico e depois ele vai esquecer a coleção. Nesse caso, a resposta do LINQ estará mais adequada do que essa. Caso seja necessário fazer múltiplas contagens aí sim está estratégia agrega muito mais do que a outra
Comment: Bacana, o  `Counter` retorna uma estrutura semelhante a um `dict` com atribuição chave valor...
Comment: Acredito que essa deveria ser a resposta aceita, por ser bastante mais geral que a resposta do @LINQ


output inside foreach by serialize, store in database

Question: how can output of inside foreach with serialize store in database? next: called of database?
please give me example.
<code>foreach($upload_data as $file) {
    echo serialize($file['file']);
}
output: s:54:"Chrysanthemum6.jpg";s:47:"/Desert6.jpg";
</code>
With respect
Comment: Can you please extend your question a bit? I have problems to understand what you're asking for.
Answer: store the serialize in a variable?
<code>$result = '';
foreach($upload_data as $file) {
  $result .= serialize($file['file'];
}
echo $result;
</code>
Comment: Don't know, if this is part of the question, but your code will produce invalid serialized data `var_dump(unserialize(serialize('file1') . serialize('file2')));
// string(5) "file1"`
Answer: Something like this?
<code>$result = array();
foreach ($upload_data as $file) {
  $result[] = $file['file'];
}
var_dump(serialize($result));
</code>


What is the probability that the digit $0$ will appear at least once and the digit $2$ will appear at least once?

Question:
Choosing a 6 digit random number, what is the probability that the digit $0$ will appear at least once and the digit $2$ will appear at least once?

Using complement, we have the digits two or zero won't appear at all:
Zero will not appear at all: $9^6$
Two will not appear at all: $8\cdot9^5$
Finally, the general case minus the sum of the above two yields a negative number:
$9\cdot10^5-(9^6+8\cdot9^5)=-103833$
But I don't see where's my error...
Answer: You are counting numbers where zero and two don't appear twice when summing $9^6$ and $8\cdot 9^5$. To fix this, count the numbers without both, those are $8^6$. Hence, inclusion-exclusion yields
$$
9\cdot 10^5 - (9^6 + 8\cdot 9^5) + 8^6 = 158311.
$$
The main idea here is that for two sets $A$ and $B$ we have $\left|A\cup B\right| = \left|A\right|+\left|B\right|-\left|A\cap B\right|$. Here $A$ consists of numbers with no $0$-digit and $B$ consists of numbers with no $2$-digit.
Comment: Ah I didn't think of inclusion exclusion, thanks.
Comment: BTW why do you call it "don't appear twice"? the counting of $9^6$ in my question is zero not appearing at all...
Comment: @kuhaku read "you are counting (numbers where zero and two don't appear) twice". Sorry, that sentence was ambiguous.
Answer: Clue : There will cases of overlap between $9^6$ and $8.9^5$. So you have to add back the cases where both $0$ and $2$ do not appear.


Spring passing object/string between controllers (GET & POST)

Question: I've been struggling with passing some value between controller.
I have one controller like this:
@RequestMapping(value = "/add", method = RequestMethod.GET)
public String addGet(HttpServletRequest request, @ModelAttribute(value="branch") Branch branch, Model model, blahblahblah)
//What I want to pass and re use:
String loadRespond;
try{
    loadRespond= *SOME LOAD STRING METHOD*;
    branch= branchManager.convertString(loadRespond); //METHOD TO SPLIT STRING & INDUCT TO OBJECT
}catch{exception){
    //blabla
}

After I successfully inducted all the attributes into the object branch,i show them all through a binding form. What i want to do is, when i'm going to update the data/change some attribute, i want to compare the old branch to the new changed branch. This means that i have to pass the old branch object or the loadRespond string onto the POST method so that can be used. Do anyone have any idea of how to do this? Maybe to assign it to hidden type field in the jsp? and then use it on the controller with request mapping /add of method type post? Thanks..I'm a newbie..
Comment: hi,yea somethin' like that..after i pass it i want to compare it to the new branch object that has been created after i click the submit button.
Comment: so clarify for me, once branch is created you want to call another controller and pass it branch object?
Answer: Why don't you try out with session scope ?
store your old branch into the session . and when you get the new object compare with the old one (by retrieving from session)
You can save into session as any of both,
 request.getSession().setAttribute("sessionvar", "session value");
 @SessionAttributes("sessionvar")

A nice Example here to start with it.
Side-note : your question title doesnt quite expalain your problem and the solutions may vary
Answer: As San Krish notes in his answer the most common way is to use @SessionAttributes and pass objects/data using them.
This is useful if you don't worry about user moving backwards and forwards in a page, or want basic control of the object.
Now if you want to have a chain where controller 1 passes to controller 2 which may pass to controller 3 your best bet is to implement web flows.
Summary:
For short and sweet and quick: SessionAttributes is the way to go, example here http://www.intertech.com/Blog/understanding-spring-mvc-model-and-session-attributes/
For chain passing, greater control and validation use Spring Web Flows.


Strange issue - Linux server unstable for some minutes (same) every hour - I need help identifying the issue

Question: I found the following issue really interesting, I think you will too.
I have a linux server that I use for hosting some websites. It uses CENTOS 6 and DIRECTADMIN. The websites have decent traffic, its a busy box, but I had never had instability issues.
The last few days something strange is happening.
Every hour, for some minutes, the same every time (26-30) the server is un-accessible by web.  The load is increasing a lot and a lot of httpd (apache) processes are opened.
I have done a very thorough analysis of the mod_status output, and I found nothing strange (I grouped ips with open connections). It seems that those are my normal traffic but for some reason are delayed.
top gives me nothing out of the ordinary, only the load is very increased, but no process stands out.
A lot (900 httpd processes are opened) but I haven't found any malicious ips that do a lot of requests. My best guess is that something is overloading the system and http queries are delayed.
I stopped crond and waited for the next cycle, the same happened.
I have also monitored port 80 with netstats grouping ips with open connections. Nothing unusual. nginx runs on port 80 and httpd on port 8000 and I check both, 8000 has connections only from my ip and port 80 no ip stands out.
/var/log/messages has NOTHING, apache error logs have nothing.
No crons running, and I get the same overload issue every hour like a clock. And after some minutes the storm passes.
Anyone has any ideas on how to pinpoint the issue ? Any other reason that CPU is consumed that doesn't show up on top ?
Comment: Well this sounds weired. Specially because your apache - logs don't say anything. I had an issue with spamers on my sites causing me to increase the number of max clients and servers due to a repeatly sending of term - signals to http - processes that did not exit. But this you would see in the httpd -logs. So obviously, if your client processes don't exit, this will increase the load of the server permanently.
Comment: One thin to check- Some web applications (e.g. Drupal) can be scheduled to run tasks at specific times without the use of cron.
Comment: Thanks sciurus. The websites on this server are custom coded from me and I always use crons.
Answer: I would look at external services that these apache processes connect to. Maybe you have a DB backup scheduled hourly at that time and maybe that locks all of the tables.
Overall it doesn't sound like a problem on the apache server itself.
Answer: If it's that tightly scheduled, other than cross referencing all the logs for that period of time, you might consider attaching strace to the httpd PID for a more in depth look or running apache in debug mode in the foreground just before it usually occurs.
HTTPd has the -X option for that, though performance will obviously suffer so I would try to get as close to the event as possible.
"-X
    Run httpd in debug mode. Only one worker will be started and the server will not detach from the console."
For strace, you might try
strace -f -p $HTTPd_PID

and/or
strace -f -c -p $HTTPd_PID

-f to follow forks and -c to count the syscalls.


Sonata Admin blank dashboard

Question: I have a few of their projects on Symfony 2. On one of them today itch to do a total upgrade to the newest versions. Prior to that, everything worked perfectly. And the projects themselves, and sonata admin and authorization.
After the upgrade, I did not knurl old configs, and carefully read the manuals for all the vendors, and prescribed all over again. There were a few problems, but the project has started pretty quickly. The site works, authorization works, not only works admin panel. Today, a day fumbling, and can not understand why I have the admin screen blank.

in /src/Itfrogs/SiteBundle/Resources/config/services.yml
...
    sonata.admin.dictionary.group:
            class: Itfrogs\SiteBundle\Admin\Model\DictionaryGroupsAdmin
            tags:
                - { name: sonata.admin, manager_type: orm, group: Content, label: Dictionary Group }
            arguments:
                - ~
                - Itfrogs\SiteBundle\Entity\DictionaryGroup
                - ~
            calls:
                - [ setTranslationDomain, [ItfrogsSiteBundle]]
...

in main config.yml
...
sonata_block:
    default_contexts: [cms]
    blocks:
        sonata.admin.block.admin_list:
            contexts:   [admin]

        #sonata.admin_doctrine_orm.block.audit:
        #    contexts:   [admin]

        sonata.block.service.text:
        sonata.block.service.action:
        sonata.block.service.rss:
        sonata.media.block.media:

sonata_admin:
    dashboard:
        blocks:
            -
                position: left
                type: sonata.admin.block.admin_list
...

I think it's basic. Rest on a manual. The class  Itfrogs\SiteBundle\Admin\Model\DictionaryGroupsAdmin and the entity Itfrogs\SiteBundle\Entity\DictionaryGroup remained the same.
Before loading the class does not reach. I checked.
Prompt, where to search?
If any more configs need tell me - I'll post.
Answer: Provided you're using the latest Sonata Admin Bundle it should be something like this according to the documentation:
sonata_block:
    default_contexts: [cms]
    blocks:
        sonata.admin.block.admin_list:
            contexts:   [admin]

Answer: If you're using Sonata user bundle with FOSUser bundle then you may not have the right permissions to view your pages.
If you're in fact using FOSUser, Try in command line:
php app/console fos:user:create adminuser --super-admin
You can find more here
**hint: but you must configure the sonata block in your configuration
sonata_block:
    default_contexts: [cms]
    blocks:
        sonata.admin.block.admin_list:
            contexts:   [admin]

Sonata Admin blank dashboard


Prove that $\sum_{k=1}^{+\infty}a_k=\sum_{k=1}^{+\infty}\int_0^1\frac{\sin(kx)}{k^x}dx$ is divergent.

Question: I am struggling with the following series:
$$
\sum_{k=1}^{+\infty}a_k=\sum_{k=1}^{+\infty}\int_0^1\frac{\sin(kx)}{k^x}dx.
$$
The general term $a_k$ is clearly well defined for every $k\geq1$ since $\int_0^1\frac{\sin(kx)}{k^x}dx$ is well defined for every $k\geq1$. I should prove that this series is divergent.
As a second request, I should prove that hence the series of functions
$$
\sum_{k=1}^{+\infty}\frac{\sin(kx)}{k^x}
$$
does not converge uniformly. This, given the first request, is quite clear since if this series of functions was uniformly convergent, then we could write
$$
\sum_{k=1}^{+\infty}\int_0^1\frac{\sin(kx)}{k^x}dx=\int_0^1\sum_{k=1}^{+\infty}\frac{\sin(kx)}{k^x}dx=\int_0^1 S(x)dx<+\infty
$$
since $S(x)$ is continuous on $[0,1]$, being the uniform limit of a series of continuous functions on $[0,1]$.
How could I proceed with the first request?
Comment: @epsilon-emperor: No, $k^x<k$ doesn't imply the last inequality ($\sin kx$ isn't always nonnegative).
Comment: Haven't been able to solve it, but just sharing some thoughts in case you or others would be interested. Since $x\in (0,1)$, $k^x < k$ which means $$\sum_{k\ge 1} a_k \ge \sum_{k\ge 1} \int_0^1 \frac{\sin kx}{k} = \sum_{k\ge 1} \frac{1-\cos k}{k^2}$$
Comment: Oops, good catch! I overlooked that! @metamorphy
Answer: Integrating twice by parts gives
\begin{align*}
\int_0^1 {\frac{{\sin (kx)}}{{k^x }}dx} & = \frac{1}{k} - \frac{{\cos (k)}}{{k^2 }} - \frac{{\log k}}{k}\int_0^1 {\frac{{\cos (kx)}}{{k^x }}dx}
\\ & = \frac{1}{k} - \frac{{\cos (k)}}{{k^2 }} - \frac{{\sin (k)\log k}}{{k^3 }} - \frac{{\log ^2 k}}{{k^2 }}\int_0^1 {\frac{{\sin (kx)}}{{k^x }}dx} ,
\end{align*}
i.e.,
$$
\int_0^1 {\frac{{\sin (kx)}}{{k^x }}dx}  = \left( {\frac{1}{k} - \frac{{\cos (k)}}{{k^2 }} - \frac{{\sin (k)\log k}}{{k^3 }}} \right)\frac{{k^2 }}{{k^2  + \log ^2 k}}.
$$
Consequently,
$$
\int_0^1 {\frac{{\sin (kx)}}{{k^x }}dx}  \sim \frac{1}{k}
$$
for large $k$. The divergence follows from the limit comparison test.


Is my process of proving this problem correct when $2x$ & $3x$ are not acute angles?

Question: Problem:

Prove $$\cot^{-1}(\tan 2x)+\cot^{-1}(-\tan 3x)=x$$

My proof:
$$\begin{align}\text{L.H.S}&=\cot^{-1}(\tan 2x)+\cot^{-1}(-\tan3x) \\
&=\cot^{-1}(\cot (\frac{\pi}{2}-2x))-\cot^{-1}(\cot(\frac{\pi}{2}-3x))\\
&=3x-2x\\
&=x\\
&=\text{R.H.S(proved)}\end{align}$$
Will my proof still be valid if $2x$ or $3x$ is in any quadrant other than the 1st quadrant? What will be the correct proof that will hold true for any quadrant? Is this identity only true when 2x & 3x are in the first quadrant?
Comment: The identity $\cot^{-1}(\cot x)=x$ only holds when $0<x<\pi$. This is because $\cot^{-1}$ is the inverse of $\cot$ *on a restricted domain*.
Comment: Hard to read.  Please remove graphics and use [MathJax](https://math.stackexchange.com/help/notation) to format math in your question.
Comment: Invalid. By the logic in the second to third line, $\cot^{-1}(\cot 10000)=10000.$
Comment: @user2661923 I edited it.
Answer: Good question about whether your proof applies when $2x$ is outside the first quadrant.
In fact, your proof applies only when you define $\mathrm{arccot}$'s principal range as $\left(-\frac{\pi}2,0\right)\cup\left(0\frac{\pi}2\right]$ and both $\left(\frac\pi2-2x\right)$ and  $\left(\frac\pi2-3x\right)$ are within this range.
Furthermore, the given statement is actually not identically true, whichever of the two contrasting definitions of $\mathrm{arccot}$ we choose.

Let us define $\textrm{arccot}$'s principal range to be $(0,\pi).$
Then $$\text{LHS}=\mathrm{arccot}(\tan
2x)+\mathrm{arccot}(-\tan3x)\\
=\frac {\pi}2-\arctan(\tan 2x)+\left(\frac {\pi}2-\arctan(-\tan 3x)\right)\\
=\pi+\arctan(\tan 3x)-\arctan(\tan 2x)\\
=\pi+(3x+m\pi)-(2x+n\pi)\quad\text{for some }m,n \\
=x+k\pi\quad\text{for some }k.$$

For example, $$x=0.7\implies\text{LHS}=0.7=x=\text{RHS},$$ but
$$x=0.2\implies\text{LHS}=0.2+\pi\neq x=\text{RHS}.$$

On the other hand, if we define $\textrm{arccot}$'s principal range
to be $\left(-\frac{\pi}2,0\right)\cup\left(0\frac{\pi}2\right],$
LHS still doesn't identically equal $x:$
Comment: To repeat my [remark](https://math.stackexchange.com/questions/4255978/what-is-the-principal-value-of-tan-1-frac-y-x#comment8850797_4256099) from yesterday: If possible, switch to another textbook.
Comment: @tryingtobeastoic That's too bad.  That you're reading it critically is more important anyway. Keep it up.
Comment: It's not possible, unfortunately.
Comment: I think you missed a thing. My proof only applies when we pick the 1st definition of $arccot$, and when $\frac{\pi}{2}-2x$ and $\frac{\pi}{2}-3x$ are in the 1st quadrant. If these two angles are in the 4th quadrant, my proof doesn't work. If the mentioned angles are in the 4th quadrant, the 2nd line of my proof would become...
Comment: $$\cot^{-1}(-\cot (\frac{\pi}{2}-2x))-\cot^{-1}(-\cot(\frac{\pi}{2}-3x))$$

$$-\cot^{-1}(\cot (\frac{\pi}{2}-2x))+\cot^{-1}(\cot(\frac{\pi}{2}-3x))$$

$$-x$$
Comment: What do you think?
Comment: But $\cot^{-1}(x)$ is only an odd function according to WolframAlpha's definition. So, how can my 2nd line be accurate according to desmos's definition, where I treated $\cot^{-1}()$ as an odd function?
Comment: @tryingtobeastoic Ah okay, I just edited the Answer to reflect the fact that if you're using definition two (Desmos's), then yes your proof is simply invalid. Re: you new comments: 1. it's about *actual angle values*, not *quadrants*. 2. your original proof actually works under defn one and when $\left(\frac\pi2-2x\right)$ and   $\left(\frac\pi2-3x\right)$ are both in, say, -89 to -1 degs (but NOT in, say, 271-359 degs).  3. your new attempt is invalid because (a) it is incorrect to add the minus signs when taking complementary angles, (b) the LHS ought to be still be of the form "$x+k\pi$".
Comment: I understand. I assumed $(\frac{\pi}{2}-2x)$ & $(\frac{\pi}{2}-3x)$ to be in the fourth quadrant, and considered $2x$ & $3x$ to be acute angles, and then I tried to apply the method mentioned in this [question](https://math.stackexchange.com/q/4257837/768162). However, $(\frac{\pi}{2}-2x)$ & $(\frac{\pi}{2}-3x)$ can never be in the fourth quadrant if $2x$ & $3x$ are acute. One of my presuppositions is invalid. That's why I am wrong in my new attempt.
Comment: I couldn't thank you yesterday for helping me greatly! Thank you!!!!!!! By the way, I posted a [question](https://math.stackexchange.com/q/4258808/768162) based on our discussion yesterday. If you want, you can check it out!


Push Notification or Polling Service in an Android app

Question: I am developing an Android app that allows sports team coaches to update the attendance for events like training/matches. A feature I would like to add would be to display a notification on the device to remind them that they need to update the attendance for the event when it has started.
I have been reading online a bit and it seems that push is the preferred method for data that is changing. But because i know the start times of the events, would it be better to create a background service using something like the following?
https://stackoverflow.com/a/9933130/2039505
I basically want the user to receive a vibrate notification which when they click on it, it will open the events attendance screen. Hopefully someone will have some insight into which option is best!
Answer: Since all you need is a notification on a timer, the AlarmManager would be the best way to go.
If you used Push Notifications(GCM), that would require server side code and a method to store the device id to send the notification to.
Overkill if you ask me.
Here are links to the official documentation and example code:
Official documentation
Vogella's example on services


I need help with this definition of a partition of a closed interval [a,b]

Question: A partition of a closed interval I = [a,b] in R is a set of couples $(I_i,t_i)$ such that $t_i ∈ I_i$ for i = 1,...,n and I = $∪I_i$
So for [0,4] is ([0,3],{1},[2,4]{3}) a partition of [0,4]?
Is ([0,4],{2}) a partition? The reason for the first attempt is because we have $I_i$ as a subinterval of [0,4] so there can be more than one subinterval $I_i$ but I also know say $I_2$ can't be two subintervals so I am now confused.The only reasonable partitions for me here are ([0,4],$t_i$) where $t_i$ are subsets of [0,4] like {1},{2},(0,3) etc.
Is there a better explanation of this definition.
Comment: I would expect the $I_i$ to be disjoint expect perhaps at the endpoints to call it a partition. What is the context?
Comment: I am trying to calculate Reimann sums given partitions with tags at the right endpoints of the given partitions
Comment: Based on that context (but also because of the word "partition"), I am almost certain your definition is incomplete. Can you give the exact source of your definition (title, author, edition, and page number)? A link to a picture of the original definition might help too.
Comment: It is in my lecture slide from my teacher,the slide has no references and I am not able to send images as comments,is there a way i can post a link leading to it?
Comment: I got it on Google Drive https://drive.google.com/file/d/1FALIknPaYpvDmxOQNyRg5_UpWBb0qf00/view?usp=drivesdk
Comment: This kind of information does not belong in comments anyway. It belongs in the question, and you can put it there by editing the question. (Your Google drive link appears to be access locked.)
Comment: I have edited it into the question,thank you
Answer: Your first attempt,
$$
\Big( [0,3], \{1\}, [2,4]\{3\} \Big)
$$
is not a partition in the defined sense, because it is an ordered triple, not a set of ordered pairs.
Your second attempt,
$$
\Big( [0,4],\{2\} \Big)
$$
is technically not a partition, because it is an ordered pair, not a set. Moreover, the second element should be just a number, not a set. The set you'd want is
$$
\Big\{ \big( [0,4],2 \big) \Big\}
$$
Some examples of partitions of $[0,4]$ would be
$$\begin{align*}
&\Big\{ \big( [0,1], 1 \big), \big([1,3],2\big), \big([3,4],3\big) \Big\} \\
&\Big\{ \big( [0,4], 0 \big) \Big\} \\
&\Big\{ \big( [0,2], 1 \big), \big( [1,4], 3 \big) \Big\}
\end{align*}$$
at least within the sense provided. A general partition of $[a,b]$ would look like
$$
\Big\{ \big( [a_1,b_1], t_1 \big), \cdots, \big( [a_n,b_n], t_n \big) \Big\}
$$
for some positive integer $n$, with $t_i \in [a_i,b_i]$ for all $i$, with the condition
$$
[a,b] = \bigcup_{i=1}^n [a_i,b_i]
$$
If you want to evaluate a function at the right-endpoint of the partition, you would choose $t_i = b_i$.

In my experience, this is a very unusual definition. The usual definition that arises in the context of Riemann integration is as so:

A partition of $[a,b]$ is a set of finitely-many points $P := \{x_0,x_1,\cdots,x_n\}$ such that
$$
a =: x_0 < x_1 < x_2 < \cdots < x_n := b
$$

This is better, for several reasons:

You're not trying to deal with ordered pairs. In fact, this notion of a partition cleanly divides $[a,b]$ into almost-disjoint intervals,
$$
[x_0,x_1] \quad [x_1,x_2] \quad \cdots \quad [x_{n-1},x_n]
$$
("almost-disjoint" in that they only touch at the endpoints, which is more than good enough for the theory used).

Disjointness (or almost-disjointness) is a nice property to have, since we don't have to worry about overlaps.

It separates the notion of the tag from that of the partition, making it look like a freer choice. Trying to specify both at the same time just feels clunky, especially because many choices of tags $t_i$ will exist for any given partition, unless you choose to use a very narrow rule.
Comment: Okay,so given
P1 := (0, 1, 2, 4)
P2 := (0, 2, 3, 4)
P3:= (0, 1, 1.5, 2, 3.4, 4)
P4 := (0, 0.5, 2.5, 3.5, 4) if I want to find the reimann sum of $x^2$ with tags selected as P1 with the tags at the left endpoints of the subintervals how will it look like,I don't understand the tag in there and the definition of tagged partition seems similar to the first definition i gave,.
Answer: Okay,I abundoned the idea of using this definition in solving the question I had and was able to solve it now using the definition of partition as $a=x_0<x_1<...<x_n=b.$
the question I was attempting was to find the Reimann sum of $x^2$ with partitions $P_1 := (0,1,2,4)$
$P_2 := (0,2,3,4)$ and the tags are selected as indicated.
(a) P_1 with the tags at the left endpoints of the subintervals.
$\textbf{Ans}$
the subintervals in $P_1$ are [0,1],[1,2],[2,4] and tags are $t_1=0$,$t_2=1$ $t_3=2$
so the Reimann sum is $\sum_{i=1}^{n}f(t_i)(x_i -x_{i-1})$ where n=3 in this example,so evaluating that gives
f(0)(1-0)+f(1)(2-1)+f(2)(4-2)=0+1+4(2)=9
(b) $P_2$ with the tags at the right endpoints of the subintervals.
$\textbf{Ans}$
subintervals of $P_2$ are [0,2],[2,3],[3,4]
S(f,P)=f(2)(2)+f(3)(1)+f(4)(1)=4(2)+9(1)+16(1)=33.
$\textit{hope i am right}$
Comment: This seems like a good choice to me. The first definition **is** missing some information that the second definition provides. Also, the question seems to be stated using the notation of the second definition. I believe you have identified all the intervals and tags within the partitions correctly.


Problema na instalação do Phonegap

Question: Estou fora de casa e preciso manter meus projetos em dia, comprei um notebook bem simples para minha viagem, sendo que na instalação do PhoneGap esta dando um erro incomum na minha opinião pois nunca tinha visto, espero que possam me ajudar.

OBS: Já tenho instalado: NodeJS e o Git.
Comment: Provavelmente você não definiu o Git lá nas variaveis globais do sistema, por isso ele não encontra o Git na sua máquina.
Comment: Você precisa instalar o GIT globalmente através de `npm install -g git`, se mesmo assim haver problemas instale através deste pacote: http://git-scm.com/downloads
Answer: Tenha a certeza de ter o Git definida na PATH do windows...

Clique com o botão direito em Meu Computador
Clique em Configurações avançadas do sistema
Clique em Variáveis de ambiente na parte inferior da janela
Em seguida adicione o caminho do git ao final de sua Path de ambiente

;C:\Program Files (x86)\Git\bin;C:\Program Files (x86)\Git\cmd

Problema na instalação do Phonegap


linq to list and get values per row

Question: for a test I need to make a list with some parameter settings. This list is not per definition pre-defined as a type and / or what's in it.
bool[] trueOrFalse = new bool[] { true, false };
        int[] para1 = new int[] { 1, 2, 3 };
        int[] para2 = new int[] { 5, 6, 7 };
        int[] para3 = new int[] { 1, 2, 3 };
        int[] para4 = new int[] { 5, 7, 9 };

        List<object> test = (from a in trueOrFalse
                   from b in para1
                   from c in para2
                   from d in para3
                   from e in para4
                   let f = c - d
                   where c - d <= 3
                   select new { a, b, c, d, e, f }).AsEnumerable<object>().ToList();

the result is fine (this is just for a test) and I do get a list back with all the parameter combinations possible, which I can use as a parameter for another method. (I need it as a list, because, as far as I know, var test cannot be parsed as a parameter to another method.)
When I hover above test in debug mode, I can see each row and each row contains a to f as a separate field.
But how can I subtract the values?
let say I want to use a foreach loop over the list.
how can I set int myA = a etc?
regards,
Matthijs
Comment: If you need to pass this list to another method it almost certainly sounds like you should add a dedicated class - casting to `List` is just a bad workaround
Comment: You might be right. Will keep that in mind for similar problems in the future.
Answer: Well, you're explicitly removing type information by calling AsEnumerable<object>. Just get rid of that, and use var for the variable:
var test = (from a in trueOrFalse
            from b in para1
            from c in para2
            from d in para3
            from e in para4
            let f = c - d
            where c - d <= 3
            select new { a, b, c, d, e, f }).ToList();

Then you can do:
foreach (var item in test)
{
    Console.WriteLine(item.a); // Strongly typed at compile-time
}

Now, I had previously missed the "which I can use as a parameter for another method" part. (You can still use a list - but it's a list of an anonymous type.) Using anonymous types, you really pass the result to a method in a strongly-typed way.
Options:

Create a named type with all the relevant data, so you can keep it in that form in a strongly typed way.
Use type inference for the method that you're passing the data to (that may or may not be feasible; it's not clear what you're doing).
Use dynamic typing in C# 4 to get around normal static typing in C#.
Access the properties using reflection.

If you can tell us more about what you're doing in the other method, that would really help. If the method can be made generic, you may well still be able to use anonymous types.
Comment: He says that he is passing the collection as a parameter to the method so he can't use var.
Answer: If you are stripping out all of the compile time type information and can't use Dynamic objects then you will end up having to use reflection.
var i=test.FirstOrDefault();
if (i==null) {
  // NO ITEMS IN LIST!!!
  // So do something else!
}
var type=i.GetType();
var aGetter=type.GetProperty("a");
var bGetter=type.GetProperty("b");

foreach (var item in test) {
  bool myA = (bool)aGetter.GetValue(item,null);
  int myB=(int)bGetter.GetValue(item.null);
}

Comment: Hi Bob, thanks for your responds. I'll be using dynamic types for this one.
Answer: you can use reflection:
bool[] trueOrFalse = new bool[] { true, false };
        int[] para1 = new int[] { 1, 2, 3 };
        int[] para2 = new int[] { 5, 6, 7 };
        int[] para3 = new int[] { 1, 2, 3 };
        int[] para4 = new int[] { 5, 7, 9 };

        List<object> test = (from a in trueOrFalse
                             from b in para1
                             from c in para2
                             from d in para3
                             from g in para4
                             let f = c - d
                             where c - d <= 3
                             select new { a,  b,  c, d, g,  f }).AsEnumerable<object>().ToList();
        foreach (object item in test)
        {
            Response.Write(item.GetType().GetProperty("a").GetValue(item, null).ToString());
        }

you can also give names to your fields like this:
select new { aName = a, bName = b, cName = c, dName = d, gName = g, fName =  f }).AsEnumerable<object>().ToList();

and then use:
foreach (object item in test)
    {
        Response.Write(item.GetType().GetProperty("aName").GetValue(item, null).ToString());
    }

Comment: hi kleinohad, thank you as well for your responds. Dynamic types will help me out here.

linq to list and get values per row


Why is a double disk failure an issue for a 5 disk Raid 5 configuration?

Question: Based on this question about RAID-5 for 5 disks: Two disks failed simultaneously?
Sorry if I butcher some of the terminology
Why is a double disk failure an issue for a 5 disk Raid 5 configuration? Even with 2 disks offline, I still can access the sections (terminology?) A, B, C, D and E. Assuming there is no disagreements (terminology?) between (D1 and D2) OR (E1 and E2), I don't see why a double disk failure couldn't continue to work and rebuild 2 new drives.

If you lose more than a single disk in a RAID 5, your array has been irreperably damaged in some way. In most cases, the data is entirely destroyed in your case if you're not an expert at recovery, or if you are unwilling to ship it off to a recovery outfit. - SmallLoanOf1M https://serverfault.com/a/834147/348076
Comment: Your misunderstanding is thinking that A1=A2=A3=A4. They don't, they're all separate pieces of data. Think of them (for example) as the 4 parts of a single file. You've lost the fourth part, plus the parity that would have let you rebuild the 4th part.
Comment: Yeah. You have 3 of 5. Sadly that is not enough to read the data. What is the question? Because you really need to read the basics of how the data is stored.
Comment: Thank you @mfinni. That clears up it for me. I thought the data was replicated 4 times and then added parity but I now see why that doesn't make sense.
If anyone wants to explain why this is a bad question, please let me know.
Comment: That would be RAID11113, right?
Comment: It sounds like a joke but my coworkers just told my about RAID 50, so... I don't really know
Answer: You are not able to read the A4 data in the stripe in this instance. And you are also unable to read the Parity info to rebuild the data from the Parity information for stripe A.
Therefore, you have data loss and are not able to recover/read the data.
Answer: Exactly, where the confusion here is how the data is saved.
Using the graphic above, if you were to save a block of data of "ABCDEFG", then you would have "ABCD_FG".  Because the volume that the E has failed, the controller would need to "rebuild" the missing block of data from the parity drive.  However, since the parity drive is also failed, there is no way to recover the 'E' piece of data.
This is where RAID 6 comes into play to better save your data.


Composition of step function with linear one.

Question: Suppose that $s$ is a step function, and $f(x)=mx+k$. How to prove that $s(f(x))$ is step function also?
Edit:
Definition of step function:
Function $s$ defined on $[a,b]$ will called a step function is there exist a partition $P$ of $[a,b]$, $P=\{a=x_0,...,x_n=b\}$ such that $\displaystyle s_{|{\large{(x_{i-1},x_i)}}}=t_i$, for some $t_i\in \Bbb R$.
Comment: Can you state the definition of [step function](http://en.wikipedia.org/wiki/Step_function) you're using, please?
Comment: I edited my post.
Comment: I took the liberty of changing $f(x)=mx+n$ to $f(x)=mx+k$ as the $n$ might cause some confusion with the other $n$ below.
Answer: Hints: The step functions are finite linear combinations of
$$s_a\colon x\mapsto\begin{cases}0 & x<a \\ 1 & x\geq a\end{cases}$$
$$\hat{s}_a\colon x\mapsto\begin{cases}0 & x\leq a \\ 1 & x> a\end{cases}$$
$$c\colon x\mapsto 1$$
Now the only thing you have to show is that $s_a\circ f$, $\hat{s}_a\circ f$ and $c\circ f$ are step functions. In proving this you might want to consider $m<0$, $m=0$ and $m>0$ separately.
Exempli gratia:
Let $m>0$, then
$$s_a\circ f\colon x\mapsto s_a(mx+k) = \begin{cases}0 & mx+k<a \\ 1 & mx+k\geq a \end{cases} = \begin{cases}0 & mx<a-k \\ 1 & mx\geq a-k \end{cases}$$
Since $m>0$, $mx<y$ if and only if $x<\frac{y}{m}$, thus
$$s_a\circ f\colon x\mapsto \begin{cases}0 & x<\frac{a-k}{m} \\ 1 & x\geq \frac{a-k}{m} \end{cases} = s_{\frac{a-k}{m}}(x)$$
and we conclude $s_a\circ f = s_{\frac{a-k}{m}}$.
The other cases are mostly similar.
Comment: Could you show me one case where $m>0$ and $s_a\cicr f$?
Comment: Ok. I'll add it to my answer.


Let $X\subset \mathbb{R}$ Lebesgue measurable, $|X|<|\mathbb{R}|$, is it true that $X$ is null?

Question: Let $X\subset \mathbb{R}$ Lebesgue measurable, $|X|<2^{\aleph_0}$, is it true that $X$ is null?
Of course I am not assuming the Continuum Hypothesis.
EDIT: It might be helpful to know that all Borel measurable sets have cardinality either $\aleph_0$ or $2^{\aleph_0}$. Then a measurable set of cardinality strictly between those two must be Lebesgue but not Borel measurable.
Comment: Have you seen that Lebesgue measure is countably additive?
Answer: Yes. This is a trivial consequence of a theorem by Steinhaus:

Suppose that $X$ has a positive measure, then $X-X=\{x-y\mid x,y\in X\}$ contains an interval around $0$.

It is not hard to prove that if $X$ is infinite, then $X$ and $X-X$ are equipotent (there is a surjection from $X^2$ onto $X-X$, and there is an obvious injection from $X$ into $X-X$). Therefore if $X-X$ contains an interval, it has size continuum, and so must $X$.
And so it follows that if $|X|<2^{\aleph_0}$ and $X$ is measurable, then it has to have measure zero.
Comment: @JLA: Good question. The answer is not necessarily. If $\sf CH$ holds, then the answer is of course positive. In fact if $\sf CH$ fails but Martin's Axiom holds the answer is still positive. Namely, every set of size $<2^{\aleph_0}$ is null. But it is also consistent that there are sets of cardinality $\aleph_1<2^{\aleph_0}$ which are not measurable. And if they are not measurable, their outer measure cannot be $0$.
Comment: @AsafKaragila Is the theorem still true if you replace "$X$ is measurable with positive measure" with "$X$ has positive outer measure"? I'm just wondering if a set with cardinality less than the continuum is necessarily null.
Comment: Cute.${{{{}}}}$
Answer: Asaf's answer is quite fine, but perhaps too specific to Lebesgue measure.
More basically, you can use just the regularity of Lebesgue measure. This tells you that if your measurable set $X$ is not null, then it contains a closed set $C$ which is not null either.
This $C$ is certainly uncountable, and so (as for any uncountable Polish space) one can embed $\{ 0,1\}^{\mathbb N}$ into $C$ using a "Cantor-like" construction. So $C$ (and hence $X$) has cardinality at least $2^{\aleph_0}$.
Not sure, however, that this is "easier" than Asaf's answer...
Comment: It's probably not an easier theorem. It also requires heavier tools from descriptive set theory (e.g. every uncountable closed set has a copy of the Cantor set). But it's a good alternative!
Comment: @Asaf I think I agree with You : it's not easier (especially because to perform the Cantor-like construction, one first needs to show that the uncountable closed set $C$ contains a closed subset with no isolated points; and also the regularity is not easy!). Anyway, I would never have thought to use Steinhaus' Theorem here!


Groovy 1.8 :: LINQ Applied

Question: UPDATE 8/31/2011
Guillaume Laforge has nearly done it:
http://gaelyk.appspot.com/tutorial/app-engine-shortcuts#query
Looks like he's doing an AST transform to pull off the:
alias as Entity

bit.  Cool stuff, Groovy 1.8 + AST transform = LINQ-esque queries on the JVM. GL's solution needs more work as far as I can see to pull off full query capabilities (like sub queries, join using(field) syntax and the like), but for his Gaelyk project apparently not necessary.
EDIT
As a workaround to achieving pure LINQ syntax, I have decided to def the aliases. Not a huge deal, and removes a major hurdle that would likely require complex AST transforms to pull off.
So, instead of:
from   c as Composite
join   t as Teams
...

I now define the aliases (note: need to cast to get auto complete on fields):
def(Teams t,Composite c,Schools s) = [Teams.new(),Composite.new(),Schools.new()]

and use map syntax for from, join, etc.
from    c:Composite
join    t:Teams
...

To solve issue #2 (see original below), add instance level getProperty methods to each pogo alias (whose scope is limited to ORM closure in which it is called, nice). We just return the string property name as we're building an sql statement.
[t,c,s].each{Object o-> o.metaClass.getProperty = { String k-> k } }

Making "good" progress ;-)
Now to figure out what to do about "=", that is tricky since set property is void.  May have to go with eq, neq, gt, etc. but would really prefer the literal symbols, makes for closer-to-sql readability.
If interested, LINQ is doing quite a bit behind the scenes.  Jon Skeet (praise his name) has a nice SO reply:
How LINQ works internally?
ORIGINAL
Have been checking out LINQ, highly impressed.
// LINQ example
var games =
    from t in Teams
    from g in t.Games
    where g.gameID = 212
    select new { g.gameDate,g.gameTime };

// Seeking Groovy Nirvana
latest { Integer teamID->
    from   c as Composite
    join   t as Teams
    join   s as Schools on ( schoolID = {
                     from   Teams
                     where  t.schoolID = s.schoolID } )

    where   t.teamID = "$teamID"
    select  c.location, c.gameType, s.schoolName
    group   c.gameID
    order   c.gameDate, c.gameTime
}

The proposed Groovy version compiles fine and if I def the aliases c,t,s with their corresponding POGO, I get strong typed IDE auocomplete on fields, nice.  However, nowhere near LINQ, where there are no (visible) variable definitions other than the query itself, totally self contained and strongly typed, wow.
OK, so can it be done in Groovy?  I think (hope) yes, but am hung up on 2 issues:
1) How to implicitly populate alias variable without def'ing? Currently I am overriding asType() on String so in "from c as Composite", c gets cast to Composite.  Great, but the IDE "thinks" that in closure scope undefined c is a string and thus no autocomplete on POGO fields ;-(
2) Since #1 is not solved, I am def'ing the aliases as per above so I can get autocomplete.  Fine, hacked (compared to LINQ), but does the trick.  Problem here is that in "select c.location, c.gameType...", I'd like the fields to not be evaluated but simply return "c.location" to the ORM select method, and not null (which is its default value).  getProperty() should work here, but I need it to only apply to pogo fields when called from ORM scope (e.g. orm field specific methods like select, order, group, etc.). Bit lost there, perhaps there's a way to annotate orm methods, or only invoke "special" pogo getProperty via orm method calls (which is the closure's delegate in above nirvana query).
Should point out that I am not looking to create a comprehensive LINQ for Groovy, but this one particular subset of LINQ I would love to see happen.
Answer: One of the biggest reasons for Guillaume to use an AST transform is because of the problem with "=". Even if you use == for the compare as normally done in Groovy, from the compareTo method that is called for it, you cannot make out a difference between ==, !=, <=, >=, <, >. There are two possible paths for this in later versions of Groovy that are in discussion. One is to use for each of those compares a different method, the other is to store a minimal AST which you can access at runtime. This goes in the direction of C# and is a quite powerful tool. The problem is more about how to do this efficiently.
Comment: Groovy 2.0 + some flavor of LINQ will be an incredible combo, Jochen. You guys are doing great work moving Groovy along.  If static Groovy (aka Grumpy) is more than a rumour, I will be very interested to give that a try as well ;-)

Groovy 1.8 :: LINQ Applied


Two angled lines in a study score

Question: I'm studying Tchaikovsky's Symphony No. 6 and have come across two lines which look like they are intended to save paper in just showing the musical lines that are playing?

They are everywhere, the next page too.

What's the proper name for these? And are they for what I had thought initially? To prevent musical instruments not playing just displaying empty staff lines with rests? If you look at the example above it's just the Viola and Cello.
Answer: I have found an answer in the book "Behind Bars" by Elaine Gould. She writes (p.521):

A pair of thick diagonal lines, known as system dividers or separation marks, divides off two or more systems of orchestral (or ensemble) music on a page. [...] Where systems can be well separated, the dividers may be placed on the left hand side of the page only; where systems are very close together, it is visually helpful to add them on both sides of the page.
Comment: You can tell by looking at the measure numbers at the top-left of each staff system, 7 and 13 in the first example, 19 and 23 in the second.  This should tell you that these staves are not being played at the time, but in sequence, top first and then bottom.
Answer: These double oblique lines on the left side of the page are "system dividers" or "system separators".
They show where a new line of music is beginning.
They are often used in orchestral score where there are lots of different instruments in many staves.  The system dividers stand out so that it easier to find the next line of music.
The lines themselves aren't saving paper though. The paper saving is done by "Frenching" the score; omitting entire staves on lines when they only have rests.
Comment: *Many staves* is not exactly the problem, because then you would have a regular layout with one or two systems per page. You need the lines most, when there are unexpected many systems on one page, due to so many instruments being omitted (like 2nd example, where just two viola and cellos remain from a whole orchestra.)
Comment: @guidot They're also very helpful in choral music, especially when there are more than 4 staves. (I'm thinking unkindly of a certain score that really liked splitting and rejoining divisi parts.)
Comment: @guidot - Many staves is part of the problem.  In a score with only a few staves throughout, system dividers become unnecessary.

Two angled lines in a study score


How to return value from this function?

Question: This is code in my node.js project but I can't access the value of connection2. I read somewhere that .getConnection is asynchronous function so I can't get the value of connection2 this way. How can I modify this code to properly return the value?
var mysql = require('mysql');
    var connectionPool = mysql.createPool({
        host: 'localhost',
        user: 'root',
        password: '',
        database: 'rockcity_followme'
    });
exports.connExport=function() {
    var connection2;
    connectionPool.getConnection(function (err, connection) {
        connection2=connection;
    });
    return connection2;
}

Answer: You cannot return the connection here. It must be asynchronous, so you take a callback as a parameter and call the callback when you have the connection.
exports.connExport=function(callback) {
    connectionPool.getConnection(function (err, connection) {
        callback(connection);
    });
}

See this for more information about asynchronous vs synchronous functions in node.

When a function does anything asynchronously, it has to be asynchronous all the way up. To use connExport from another module, it must be used asynchronously.
var yourModule = require("yourModule");
yourModule.connExport(function(connection) { // <-- the callback function
    // you can use connection here
});

Comment: This is what I don't understand. Where should I write/define the callback function? Would you please edit your answer a bit?
Comment: @VikasKumar Does the example usage I added help?

How to return value from this function?


How to find the infimum and supremum of the set $\{x>0:3x^2-8x-3\le0\}$

Question:
Let $S=\{x>0:3x^2-8x-3\le0\}$. Find the supremum and infimum.

I want to solve this question, although I have no idea how to do it, I think I should guess the $\sup S$ and $\inf S$ first and try to prove it. I have solve the equation $3x^2-8x-3=0$ and get $x=3$. Then I found that if we have an equation $3x^2-8x-3=p$, for some $p\le0$, then we have $$x=\frac{8\pm \sqrt{64+12(3+p)}}{6}$$ and $$\frac{8\pm \sqrt{64+12(3+p)}}{6}\le3 \iff p\le0$$ Hence, I think I have proven that $3$ is a upper bound of $S$.
I know if I want to show that $3=\sup S$, I need to show that for any $\varepsilon>0$, there exists a number $t$ in $S$ such that $t>3-\varepsilon$, but I don't know how to prove that. And I have no idea how to find a lower bound of $S$ and $\inf S.$
Answer: Rewrite the inequality as follows:
$$3x^2-8x-3 \le 0 \land  x>0 $$
$$\iff(3x+1)(x-3) \le 0 \land x > 0$$
$$\iff -\frac13 \le x \le 3 \land x > 0$$
$$\iff 0 < x \le 3$$
and it should be clear that the supermum is indeed $3$ and the infimum is $0$.
Comment: Yikes, thanks for pointing that out.
Comment: Thanks, I did not realize that.


From trivariate cdf to the distribution of differences of random variables

Question: Consider a trivariate cumulative distribution function (cdf) $G$.

Is there a collection of necessary conditions on $G$ ensuring that
$$
\exists \text{ a random vector $(X_1,X_2)$ such that $(X_1, X_2, X_1-X_2)$ has cdf $G$}
$$
?
Is there a collection of necessary and sufficient conditions on $G$ ensuring that
$$
\exists \text{ a random vector $(X_1,X_2)$ such that $(X_1, X_2, X_1-X_2)$ has cdf $G$}
$$
?

Update I: Let $P$ be the probability distribution associated with $G$. We can claim that: if there exists a random vector $(X_1,X_2)$ such that $(X_1, X_2, X_1-X_2)$ has probability distribution $P$, then
$$
\int_{(a,b,c)\in \mathbb{R}^3 \text{ s.t. } c=a-b} dP=1
$$

Is this condition also sufficient? I.e., can we claim that if
$$
\int_{(a,b,c)\in \mathbb{R}^3 \text{ s.t. } c=a-b} dP=1
$$
then
there exists a random vector $(X_1,X_2)$ such that $(X_1, X_2, X_1-X_2)$ has probability distribution $P$?
Can we write
$$
\int_{(a,b,c)\in \mathbb{R}^3 \text{ s.t. } c=a-b} dP=1
$$
by using the cdf $G$
?

Update II:
If there exists a random vector $(X_1,X_2)$ such that $(X_1, X_2, X_1-X_2)$ has probability distribution $P$, then $P$ should satisfy:  for every $\begin{pmatrix}
a_1\\
b_1\\
c_1
\end{pmatrix}\leq \begin{pmatrix}
a_2\\
b_2\\
c_2
\end{pmatrix}$

If $a_2\geq b_2+c_2$
$$
\begin{cases}
P([a_1,a_2], [b_1, b_2], [c_1, c_2])= P([a_1, b_2+c_2], [b_1, b_2], [c_1, c_2])\\
P([a_2, a_3], [b_1, b_2], [c_1, c_2])= 0  & \forall a_3\geq a_2\\
\end{cases}
$$
If $b_1\leq a_1-c_2$
$$
\begin{cases}
P([a_1,a_2], [b_1, b_2], [c_1, c_2])= P([a_1,a_2], [a_1-c_2, b_2], [c_1, c_2])\\
P([a_1,a_2], [b_3, b_1], [c_1, c_2])=0 & \forall b_3\leq b_1\\
\end{cases}
$$
If $a_1 \leq b_1+c_1$
$$
\begin{cases}
P([a_1,a_2], [b_1, b_2], [c_1, c_2])= P([b_1+c_1,a_2],[b_1,b_2],[c_1,c_2])\\
P([a_3,a_1], [b_1, b_2], [c_1, c_2])=0 & \forall a_3 \leq a_1
\end{cases}
$$
If $b_2\geq a_2-c_1$
$$
\begin{cases}
P([a_1,a_2], [b_1, b_2], [c_1, c_2])= P([a_1,a_2], [b_1, a_2-c_1], [c_1, c_2])\\
P([a_1,a_2], [b_2, b_3], [c_1, c_2])=0 & \forall b_3\geq b_2
\end{cases}
$$
If $c_2 \geq a_2-b_1$
$$
\begin{cases}
P([a_1,a_2], [b_1, b_2], [c_1, c_2])= P([a_1,a_2], [b_1, b_2], [c_1, a_2-b_1])\\
P([a_1,a_2], [b_1, b_2], [c_2, c_3])=0 & \forall  c_3\geq c_2
\end{cases}
$$
If $c_1\leq a_1-b_2$
$$
\begin{cases}
P([a_1,a_2], [b_1, b_2], [c_1, c_2])= P([a_1,a_2], [b_1, b_2], [a_1-b_2, c_2])\\
P([a_1,a_2], [b_1, b_2], [c_3, c_1])=0 & \forall  c_3\leq c_1
\end{cases}
$$
These implications can be written using $G$ (as I want!). However: are these implications also sufficient? I don't know how to prove or dis-prove it.
Comment: You have supplied such conditions: to wit, when $(X_1,X_2,X_3)$ is a random variable with distribution $G,$ then almost surely $X_3=X_1-X_2.$  That's dead simple.  What other form are you hoping to express these conditions in that would be any simpler or more useful?
Comment: Thanks. I'm hoping for conditions directly imposed on $G$.
Answer: The condition in your first update is sufficient, because it implies that $X_3=X_1-X_2$ almost surely, so in particular $(X_1,X_2,X_3)=^d(X_1,X_2,X_1-X_2)$.
Edit: To be completely explicit about the dependence of $G$, this can be restated by requiring that $P(B)=0$ for any 3-dimensional box $B$ such that $B\cap \{(x,y,z):z=x-y\}=\emptyset$.
Comment: Thanks. What about update II? Remember that I'm looking for conditions that can expressed using $G$.
Comment: The measure $dP$ is defined in terms of $G$, so I'm not sure what else you want.
Comment: I want necessary and sufficient conditions explicitly involving  "boxes" in $\mathbb{R}^3$
Comment: Thanks. I think that "$P(B)=0$ for any 3-dimensional box $B$ such that $B\cap ...=\emptyset$" holds IF AND ONLY IF the bullet points in my update II hold. Is this correct?


Why do we store object in an interface reference?

Question: I have a doubt about interfaces.
I have seen some code,they are writing
Connection con = Drivermanager.getConnection("URL","User name","Pwd");

here con (reference varible of Connection Interface ) holding Oracle Connection Object.
since we know that we are creating Oracle Connection Object why cant we write
OracleConnectionClass Oraclecon = Drivermanager.getConnection("URL","User name","Pwd");

if i asked some people they say that ,they are achieving polymorphism,by using Connection Interface.
But,
what is the use.
we can do
create statement,execute query.. etc by both references (con and Oraclecon).
simillarly for List and ArrayList
Thanks in advance
Comment: this is not simple as you think, you must have to read why to use interfaces and their advantages then only to get your answer
Answer: You know the exact type you are creating right now, but that may change in the future due to your own code changing, or the implementation changing.
Polymorphism is used to abstract away details so you don't have to touch code later.  If you used an exact type instead of an interface here, and that type changed, you'd have to change your code.
The consequences of this seem obvious.  Changing code is work, work cost money, therefor less change means less cost.
But its even more expensive.  Each time you change stable code you risk introducing a bug.  That risk is low here but also worth mentioning.
This is why they use an interface.
Here's a good example of polymorphism's power.
Queue<Float> queue = new LinkedList<Float>();

We can cause a big change the behavior simply.
Queue<Float> queue2 = new BlockingQueue<Float>();

Or how about this?
Queue<Float> queue3 = new PriorityBlockingQueue<Float>();

All are queues we can use to change behavior quickly.
Answer: A short answer would be "because we don't care that it's an OracleConnection". It could be changed to a MySQLConnection and the code would still work, since both would implement the Connection class.
Another example is the List interface.
List<String> l = new ArrayList<String>();

If you need to change the list type, you need to change only one place.
List<String> l = new LinkedList<String>();

Answer: In a code written tightly coupled it achieves nothing but fancy names like I used polymorphism.
But imagine a case when You have an instance variable as
Connection con;

and a parameterised constructor accepting Connection
public businessDao(Connection con){
this.con = con;
}

//some function using connection
Now the constructor can accept any type of connection object that implements Connection interface be it a
Class.forName("oracle.jdbc.driver.OracleDriver").newInstance()

or a jndi connection as follows
dataSource = (DataSource) context.lookup("java:comp/env/jdbc/jelasticDb");
con = dataSource.getConnection();

you dont need to hard code your code with the type of connection here just pass the required type of object.
You can find actual use of polymorphism in Dependency Injection.
Comment: Please clean up your grammar.  +1 for mention of dependency injection.


plotting larger grid over image2D

Question: Does anybody know if there is any easy way to plot a larger grid over an image from image2D?
So I have a matrix of values which I currently plot using the code below:
image2D(matrix, 0:max(data$y), 0:max(data$x), border = "grey")
Which produces the following

What I want to do is just draw extra grid lines in a darker colour over the top to divide the plot into larger sections at every 10 small grid points so the whole plot will be divided into, in this instance 79 larger grid squares.
Any help with this would be much appreciated.
Answer: Ah it was me being stupid, I can just use grid
https://stat.ethz.ch/R-manual/R-devel/library/graphics/html/grid.html

plotting larger grid over image2D


How can I find file extension in ruby?

Question: How can I find file extension in ruby ?
Any example ?
Answer: Try this =>
<code>ext = File.extname(file_name)
</code>

How can I find file extension in ruby?