Comparing the accuracy of binary classifiers using iterated cross-validation

Question: Let's say that you want to compare two binary classifiers (e.g., LDA and linear SVM) for a given research question, the question being "which one will probably perform best for the problem at hand, in future studies?". To that end, we have one reasonably large sample, and want to apply iterated/repeated cross-validation to compare the accuracy of these classifiers. The goal is really to have a generalizable statement that one classifier might be regarded as more accurate than the other one for this kind of data and research question.
If you apply $K$-fold cross-validation repeated $B$ times, you get $B \times K$ accuracies for each classifier. I can guess, from various answers on SE (e.g., here and here), that using simple tests as t-tests or preferable Wilcoxon tests on those $B \times k$ accuracies should make sense. Is this a possible way, or even the canonical way, to compare the accuracy of two classifiers after such a design? If not, what would be a better way?
Also, how to report extensively the results after an iterated cross-validation? The "global", mean accuracy, is the grand mean over the $B \times K$ accuracies. Should one also report their standard deviation, or any other measure of dispersion? Similarly, is it useful or usual to give a plot (boxplot, stripchart, ...) of those $B \times K$ accuracies for each classifier?
Thanks!
Comment: The pertinent argument against the t test is [not that it is inapplicable when the data are not normally distributed](https://stats.stackexchange.com/questions/120613/p-value-test-for-comparing-classifiers/123376#comment963596_123376) (this argument is false), but the dependence structure introduced by using each observation multiple times. And we actually have *two* kinds of dependency here, which the Wilcoxon test does not account for. So if at all, it looks like we should be using a more complex mixed model with two different grouping factors.
Comment: @StephanKolassa do you have any references for a mixed-level approach for analyzing CV data? The problem is not only that there is a fold effect, but that data are dependent across folds, which I think won't be easy to model
Comment: @rep_ho Do you mean something like time series cross validation?
Comment: @Dave it doesn't matter if it's a time series or any other CV, the CV results are dependent, which is the main problem when trying to get some valid statical inference out of it. Having a 10 fold CV is not the same as testing 10 models on 10 independent samples, which would be relatively easy
Comment: Yes, the multiple dependencies is precisely what I was getting at. Unfortunately, no, I do not have anything I could point you to, or I would have posted it as an answer. Sorry!
Comment: So it seems that it's an open question :) Actually, the "testing" part is not the most important for me; I'm more interested in reporting correctly a kind of good "uncertainty"/"variability" in classification accuracy for each classifier, so as to make their comparison more useful and relevant. But even for this very simple aim, I couldn't find any good example among scientific articles in my disciplinary field.
Answer: first, mandatory disclaimer, don't use accuracy as your performance measure unless you really have to, it's quite insensitive especially for model comparison. AUC is slightly better, continuous measures like brier score or log score are much better.
Strictly speaking, you can't really make an inference about other similar datasets based on your one dataset. You can just assume they will be independent samples from the same population. Also, you are not really making statements about how good is the classifier for this problem, but how good is the classifier for this problem, with this CV setup and sample size.
As was mentioned, the problem with CV is that the folds are not independent and what to do with it is kind of an open question. The normality of residuals is not that important and that's easy to solve. If you want a quick and dirty approximate solution that is however better than a naive approach of doing nothing, you can use a correlated t-test where the t-statistic is adjusted for the assumed correlation between folds. See https://arxiv.org/abs/1606.04316 also for some other suggestions on how to deal with this issue (I wouldn't worry too much about their Bayesian preaching). Here is also a newer paper that can give you more pointers https://arxiv.org/pdf/2007.12671.pdf
You can probably also just bootstrap your errors, but I don't have a reference for that, and its easy to make it wrong. If you would care only about a p-value for one model, you can use a permutation test, I don't think that can be used easily also for model comparison https://nmr.mgh.harvard.edu/~fischl/reprints/golland-fischl-ipmi03.pdf (Golland, Fischl 2003; Permutation Tests for Classification: Towards statistical Significance in Image-Based Studies)


How to find a good estimator for $\lambda$ in exponential distibution?

Question: I have an Exponential distribution with $\lambda$ as a parameter.
How can I find a good estimator for lambda?
Comment: https://en.wikipedia.org/wiki/Exponential_distribution#Parameter_estimation
Comment: Is $\lambda$ the rate or the scale?
Comment: Notation $\lambda$ is widely used for rate, so I'm assuming rate.
Answer: The term how to find a good estimator is quite broad. Often we assume an underlying distribution and put forth the claim that data follows the given distribution. We then aim at fitting the distribution on our data. In this case ensuring we minimize the distance (KL-Divergence) between our data and the assumed distribution. This gives rise to Maximum Likelihood Estimation. We thus aim to obtain a parameter which will maximize the likelihood.
In your case, the MLE for $X\sim Exp(\lambda)$ can be derived as:
$$
\begin{aligned}
l(\lambda) =& \sum\log(f(x_i))\quad\text{where} \quad f(x_i)=\lambda e^{-\lambda x}\\
=&n\log\lambda-\lambda\sum x\\
\frac{\partial l(\lambda)}{\partial \lambda} = &\frac{n}{\lambda} - \sum x \quad
\text{setting this to } 0 \text{ and solving for the stationary point}\\
\implies \hat\lambda =& \frac{n}{\sum x} = \frac{1}{\bar x}\end{aligned}
$$
This estimator can be considered as good. But what exactly do we consider as a good estimator? Some properties for a good estimator are:

Unbiasedness - Is our estimator Unbiased?
An estimator $\hat\theta$ will be considered unbiased when $E(\hat\theta) = \theta$
In Our case:
$$
\begin{aligned}
E(\hat\lambda) = & E\left(\frac{1}{\bar X}\right) = E\left(\frac{n}{\sum X_i}\right)= E\left(\frac{n}{y}\right)\\
Recall:\quad& \sum X_i = y \sim \Gamma(\alpha=n, \beta = \lambda) \text{ where } \beta\text{ is the rate parameter}\\
\therefore E\left(\frac{n}{y}\right) = &\int_0^\infty \frac{n}{y}\frac{\lambda^n}{\Gamma(n)}y^{n-1}e^{-\lambda y}dy = n\int_0^\infty \frac{\lambda^n}{\Gamma(n)}y^{n-1-1}e^{-\lambda y}dy = n\frac{\lambda^n}{\Gamma(n)}\frac{\Gamma(n-1)}{\lambda^{n-1}}\\
=&\frac{n}{n-1}\lambda\\
\implies& E\left(\frac{n-1}{n}\hat\lambda\right) = \lambda
\end{aligned}
$$
Our estimator above is biased. But we can have a unbiased estimator $\frac{n-1}{n\bar X}$. There are many other unbiased estimators you could find. But which one is the best? We then look at the notion of Efficiency.

Efficiency
For an exponential random variable,
$$
\ln f(x \mid \lambda)=\ln \lambda-\lambda x, \quad \frac{\partial^{2} f(x \mid \lambda)}{\partial \lambda^{2}}=-\frac{1}{\lambda^{2}}
$$
Thus,
$$
I(\lambda)=\frac{1}{\lambda^{2}}
$$
Now, $\bar{X}$ is an unbiased estimator for $h(\lambda)=1 / \lambda$ with variance
$$
\frac{1}{n \lambda^{2}}
$$
By the Cramér-Rao lower bound, we have that
$$
\frac{g^{\prime}(\lambda)^{2}}{n I(\lambda)}=\frac{1 / \lambda^{4}}{n \lambda^{2}}=\frac{1}{n \lambda^{2}}
$$
Because $\bar X$ attains the lower bound, we say that it is efficient.

You could also look at Consistency, Asymptotic Normality and even Robustness.
Lastly, you would like to look at the MSE of your estimator. In this case:
$$
\begin{aligned}
MSE(\hat\lambda) =&E(\hat\lambda - \lambda)^2 = E(\hat\lambda^2) - 2\lambda E(\hat\lambda) + \lambda^2\\
=&\frac{n^2\lambda^2}{(n-1)(n-2)} -\frac{2n\lambda^2}{n-1}+\lambda^2\\
=&\frac{\lambda^2(n+2)}{(n-1)(n-2)}
\end{aligned}
$$
In the end you will still have to find a balance between the biasedness and MSE. Often a times we aim at reducing both. But usually no one estimator completely minimizes both.
Comment: I really appriciate the effort! this makes a lot of sense and intuition.
Answer: The estimation section of @StephanKolassa's Wikipedia link
has the information you need. I illustrate some of the statements
there, using a simulation in R. I use $n = 10$ and $\lambda = 1/3.$
The MLE of $\mu = 1/\lambda$ is $\hat\mu = \bar X$ and it is unbiased:
$E(\hat \mu) = E(\bar X) = \mu.$
The MLE of $\lambda$ is $\hat\lambda = 1/\bar X.$ It is biased (unbiassedness
does not 'survive' a nonlinear transformation): $E[(\hat\lambda-\lambda)] = \lambda/(n-1).$ Thus an unbiased estimator of $\lambda$ based on the MLE
is $\hat\lambda_u = \frac{n-2}{n-1}\frac{1}{\bar X}.$
<code>set.seed(1119)
a = replicate(10^6, mean(rexp(10,1/3)))
mean(a)
[1] 3.000249    # sample mean unbiased MLE for pop mean
mean(1/a)
[1] 0.3702984   # MLE for rate is biased (too large)
lam.est.u = (8/9)*(1/a)
mean(lam.est.u)
[1] 0.3291541   # unbiased MLE for rate; aprx 1/3
</code>
The Wikipedia link suggests that the (slightly biased) estimator
$\hat\lambda_m = \frac{n-2}{n}\frac{1}{\bar X}$ may have slightly
better properties.
Comment: No quarrel with that. // If you comment on unfinished answers, you seem to create extra copies.
Comment: That's what I mean.


qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found

Question: I'm getting
 qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.

after a fresh installation of qt creator in Debian 9. Unfortunately, the solutions here tell me to do ldd /usr/lib/qt/plugins/platforms/libqxcb.so but /usr/lib/qt/ doesn't even exist.
Comment: execute `export QT_DEBUG_PLUGINS=1` and execute your executable: `./your_executable`, what is the output?
Answer: sudo apt-get install --reinstall libxcb-xinerama0
Comment: Instead of posting one line of code, please consider adding more of an explanation on why you recommend this and how it answers OP's question.
Answer: Follow this two steps, it work for me with wireshark
$ sudo -s
# sudo apt-get install libcap2-bin

and reboot for more ...


Azure Function to communicate to external service via VPN

Question: I would like my azure function to be able to access a postgres db hosted on fly.io, however, to connect to the fly.io organisation I need to do it via a VPN/wireguard-tunnel as its hosted on a private network. Is this possible with an azure function, can I have an azure function connect to an external service via a VPN?
I have created a virtual network on azure and have a VM on that virtual network talking to the db via wireguard, but i am not sure how to do it from an azure function, any help would be most appreciated.
Answer: Azure Functions supports accessing resources in a VNET through Virtual Network Integration.
With this and setting up a VPN Gateway on the VNET to where your DB is connected, would allow the Azure Function to access it.


Add content to PHP WP excerpt

Question: I'm a complete PHP newbie and I need a solutionto get the most out of this stupid problem with a custom Wordpress theme.
I found the part where excerpts to on index page are 'made' and all I need is to add "..." to the end of this part.
<?php

echo wpautop(
    $post->post_excerpt
    ? $post->post_excerpt
    : athemes_limit_string(strip_tags($post->post_content), 200)
);

?>

I tried to add them after this line, but they showed up after the break, but I want them inline.
Answer: There are better ways of doing this, but to answer your question:
echo wpautop( $post->post_excerpt ? $post->post_excerpt : athemes_limit_string(strip_tags($post->post_content), 200).'...' );

Comment: So, it's not the excerpt that is being printed then. Try adding it here instead: `athemes_limit_string(strip_tags($post->post_content), 200).'...'`.
Answer: In PHP you can add anything you want in the HTML by doing it outside of PHP, that here is after the ending ?> marker:
[...] ?> ...

Compare: http://php.net/language.basic-syntax.phpmode
I'd say that is the most simple and flexible method when you're concerned about themeing. The second suggestion I would give is to add it as a string to the echo, works by using a comma (,):
echo wpautop( $p ... nt), 200) ), '...';
                                ^^^^^^^

Compare: http://php.net/echo , http://php.net/string
The function in Wordpress for this btw is the_excerpt() and if you modify a theme, you should just hook in that. Looks like the custom theme you've got is doing things out of the lines. The the_excerpt function btw. add that on it's own, you don't need to care.
Comment: Thank you. Now I understand what I'm doing a little bit more. :)
Comment: Well just digged through the code, most likely one problem you still might run into is that wpautop is doing crap again. I strongly would say not to use wpautop here, there are better plugins like fancy excerpts that are doing "the best" (tm) excerpts :) - http://wordpress.org/plugins/sem-fancy-excerpt/ - those can also be properly configured, e.g. the style of "..." you want to add here.
Comment: I'm pretty satisfied with how it looks now, I just wanted those three dots at the end - http://blog.alpner.com/ ...but I'll sure bookmark that plugin. Tnx again.

Add content to PHP WP excerpt


Linear regression: how to treat an explanatory variable that is discrete but does not have a natural zero

Question: Background/study system:
One of my MS students is studying the biomechanics of strand breakage in Spanish moss (an epiphyte--or plant that lives on other plants). Spanish moss has strands that can grow into larger clumps of strands ("festoons")...where new strands originate from renewal shoots at each node. Old strands often form a wire support from which new strands grow. The number of a nodes depends on the age of the strand, but she has found that most plants have 7 to 13 nodes (mode = 10, counting from proximal point where it joins the "wire" to the distal end of the strand).
When a single strand is pulled, it always breaks at a node (never at the inter-node) and this appears to an important mechanism by which Spanish moss reproduces (vegetatively): a strand can break off and (if it lands in a good spot) can keep growing. For each strand, she has made cuts at the midpoint around each node and examined material properties (such as "yield strength" and "work to breakage"). We want to regress these material properties against nodal position to determine if they change along a strand.
Statistical problem:
Ultimately, I would like to set this up a a mixed-effects model with nodal position as a fixed effect and strand as a random effect.
My question pertains to how the node variable should be treated in a regression analysis. Nodal position seems to be discrete (1,2,3...13) but it has no natural zero point, which would make it difficult to interpret the y-intercept. I have seen a lot of sources that discuss "truncated" variables and left-"censored" variables, but this seems to be a slightly different problem. It could also be argued that this is a form of an integer distribution without zero, but I cannot seem to find any discussion of such a probability distribution or how one would interpret regression results for such a distribution.
Alternatives:
My original thought was to treat this as an ordinal variable. My concern is that I am losing information and that the results will be harder to interpret because there will be a separate regression coefficient for each node. Adding random effects or accounting for a nonlinear response would make the results even more complicated.
I have also considered treating nodal position as a categorical variable with simple coding or something like Helmert coding, but with this approach, you are throwing away information about the order of the nodes and you again have lots of regression coefficients.
A compromise that has been suggested is defining some measure of top-middle-end, but this also comes with some loss of information and would require some sort of control for variation in strand length.
A previous study transformed the predictor by dividing the distance to the node (from the proximal end) by the total length of the strand. This simplifies the regression analysis and controls for strand length, but I am not sure if this is a good idea biologically since nodal properties for the same proportions may not be the same if the number of nodes per strand varies...and I am not sure if this is a good idea statically since you are making a naturally discrete variable continuous and, if you transform it (as described), you end up with a disproportionate number of data points at 100%.
Comment: You might be (way) overthinking this.  Simple, intuitive analogs of your situation (if I understand it correctly) would range from the number of stories in a building to the number of rings in a tree to the number of employees (including the owner) in a business.  Interpreting the intercept is rarely an issue in models that use such explanatory variables--and if it is, one can always use the simple expedient of renumbering. The real issue would appear to concern *how* to model this variable--as ordinal or with splines or linearly--as you discuss.
Comment: Yes, it's common--and it has been extensively addressed.  The question of removing an intercept is discussed in all multiple regression textbooks and has been discussed in [several threads on this site, too.](https://stats.stackexchange.com/search?q=regression+intercept+force)  Renumbering is not the same as changing the first node.  If you intend to treat the counts as *numbers,* then a valid renumbering for a linear model would be some affine transformation.  That could be as simple as subtacting 1 from each number so that the sequence begins at 0.
Comment: By renumbering are suggesting that the first node would be treated as zero? If the intercept were not important, is the simple answer to just remove it when fitting the model? Seems like a common problem (indeed), but one that has not been addressed!
Comment: I have see a lot of discussion about the validity of removing the intercept, but not in this context..and these threads don't necessarily discuss the problem stated above. Are you saying that if my goal is to test the effect (of say nodal position on yield strength) the y-intercept does not matter?
Comment: I wasn't saying that, but if indeed that's your objective, then the y-intercept would appear to be irrelevant.
Answer: If all you really care about is whether the properties "change along a strand," then the value of the intercept itself is unimportant. You are correct in your comment: "if [the] goal is to test the effect (of say nodal position on yield strength) the y-intercept does not matter."  However you model the node position, the change along a strand will be indicated by the coefficient(s) for node position.
The idea to simply subtract 1 from each node position, proposed in comments by @whuber, would give a simple interpretation to the intercept: it would be the (predicted) value of a mechanical property at the very first node along the strand. (Think about this as having chosen to start numbering from 0 instead of from 1, as some computer languages do for indices into arrays.) In a simple random effects model for strands, the random effect (for an intercept) would then pick up the variation among strands in that property at the first node. This is simple, will be interpretable for all strands with nodes, and is easy to explain to a reviewer.
Note that if you were to treat node number as purely categorical (or ordinal), you would still have to choose a reference category value against which the other categories are compared. Subtracting 1 from all your node numbers, if they are treated as numerical predictors in some way, is simply choosing node 1 as the reference.
The bigger problem, as @whuber emphasized, is how to deal with the node numbers: as purely categorical, ordinal, numerical (possibly transformed in some way, or modeled with splines). Should node numbers perhaps be corrected for the total number of nodes along a strand? You knowledge of the underlying subject matter, and exploratory plots of properties as functions of node number, stratified by the total number of nodes in the strands, should indicate useful ways to proceed.


This has to be in the lit somewhere. Can someone point me to this in any accessible book or lit?

Question: It's just a big trig, sinusoidal, Fourier series thing:
$$\begin{align}
 y(t) &= \sum_{k=0}^{K} a_k \big( A \cos(\omega t) \big)^k \\
\\
     &= \sum_{n=0}^{K} b_n \cos(n \omega t) \\
\end{align}$$
The result I got (I think I fixed my error with the limits in swapping summations) is:
apparently, the DC component ($n=0$) of the Fourier cosine series is:
$$ b_0 =  a_0 + \sum_{k=1}^{\left\lfloor \tfrac{K}{2} \right\rfloor} a_{2k} \frac{(2k)!}{(k!)^2} \left(\frac{A}{2}\right)^{2k}  $$
The coefficient for the even harmonic ($n$ even) terms:
$$ b_{n} = 2 \sum_{k=\frac{n}{2}}^{\left\lfloor \tfrac{K}{2} \right\rfloor} a_{2k} \frac{(2k)!}{\left(k+\frac{n}{2}\right)! \left(k-\frac{n}{2}\right)!} \left(\frac{A}{2}\right)^{2k}  $$
And the coefficient for the odd harmonic ($n$ odd) terms:
$$ b_{n} = 2 \sum_{k=\frac{n-1}{2}}^{\left\lfloor \tfrac{K-1}{2} \right\rfloor} a_{2k+1} \frac{(2k+1)!}{(k+\frac{n+1}{2})!(k-\frac{n-1}{2})!} \left(\frac{A}{2}\right)^{2k+1}  $$
where $\lfloor u \rfloor$ is the floor() function, which is the largest integer no larger than the argument $u$.
$$ \lfloor u \rfloor \le u  < \lfloor u \rfloor + 1 \qquad \qquad \lfloor u \rfloor \in \mathbb{Z}, u \in \mathbb{R}$$
or
$$  u-1 < \lfloor u \rfloor \le u  \qquad \qquad \lfloor u \rfloor \in \mathbb{Z}, u \in \mathbb{R}$$
Comment: What, actually, is your question?
Comment: i want an expression for $b_n$, in terms of $K$, $A$, and all of the $a_k$ for $0 \le k \le K$.  this **has** to have been done before by someone.
Comment: we can set $A=1$ with no loss of generality.  but even with $A$ removed from the problem, i would still like to see a prior reference to this simple question and the non-so-simple solution.
Answer: Since the OP wants a reference, I would say the use the sums found in Prudnikov, Integrals and Series, vol. 1. 22.214.171.124 and 188.8.131.52. Ultimately they are not any more profound than the binomial theorem. In this particular situation they are, with a little algebra to clean them up
$$ \cos^{2n}(x)=2^{1-2n}\sum_{k=1}^{n}\binom{2n}{n-k}\cos{(2\,k\,x)} + 2^{-2n}\binom{2n}{n}$$
$$\cos^{2n+1}(x)=2^{-2n}\sum_{k=0}^{n}\binom{2n+1}{n-k}\cos{((2\,k+1)x)} $$
The OP's procedure basically amounts to inserting these expressions in the original sum and interchanging the summations.
Comment: Sorry, I don't have a more relevant reference, but I would think a description of the technique is sufficient for most publications.
Comment: right.  but i was looking for a reference that also does that.
Comment: if i can get a reference that says
$$\sum_{k=0}^{K}a_k \big(\cos(\theta) \big)^k=\sum_{n=0}^{K} b_n \cos(n \theta)$$where$$\begin{align} b_0&=a_0+\sum_{k=1}^{\left\lfloor \tfrac{K}{2} \right\rfloor}a_{2k} \frac{(2k)!}{(k!)^2}(\tfrac12)^{2k}\\ \\ b_{2m}&=2\sum_{k=m}^{\left\lfloor \tfrac{K}{2} \right\rfloor}a_{2k} \frac{(2k)!}{\left(k+m\right)!\left(k-m\right)!} (\tfrac12)^{2k}\\ \\ b_{2m-1}&=2 \sum_{k=m-1}^{\left\lfloor \tfrac{K-1}{2} \right\rfloor}a_{2k+1}\frac{(2k+1)!}{(k+m)!(k-m+1)!} (\tfrac12)^{2k+1}\end{align}$$
where $m\in\mathbb{Z}\ge1$,
 that is the result i am looking for.
Comment: or, equivalently, if i can get a reference that says
$$\sum_{k=0}^{K}a_k \, x^k=\sum_{n=0}^{K} b_n T_n(x)$$ where $T_n(x)$ is the $n$th-order Tchebyshev polynomial of the first kind and the $b_n$ coefficients are as above, then i will feel that i found a prior reference.
Comment: thank you skbmoore.  i dunno if/when i will ever write anything up.  but this is about a slight generalization (but not totally general) to the notion of *"waveshaping"* where a memoryless non-linearity is modeled as a finite-order polynomial, a sinusoid goes into the polynomial, and what comes out is a periodic function with an upper limit to the harmonics and we can compute the strength of each harmonic without doing the traditional Fourier series.  it's just seems to me to be a problem that someone else would have explicitly solved decades or longer ago.
Comment: if no one else weighs in with an answer, i think i'll give the bounty to you just because i hate waste.
Answer: Note that the Chebyshev polynomials $T_n$ are such that $T_n(\cos\theta) = \cos n\theta$; see the wikipedia article for details.  So you are given a polynomial $A(x)$ with known coefficients and want to find $b_k$ so that $A(x)=\sum_k b_k T_k(x)$.  A recipe is given here where it says "with inverse $x^n = 2^{1-n}\sum\dots$".  Multiply both sides of that by your $a_n$ and sum;
the coefficient of $T_k(x)$ will be your desired $k$.
Added: A reference for the needed identity is Corollary 12 in this paper; I'm sure there are others.
Comment: i've known and understood Tchebyshev polynomials for 4 decades.  i know exactly why $T_n(\cos \theta) = cos(n \theta)$.  and this can get us to another result if the polynomial is, instead, framed as a sum of Tchebyshev's (like a different, but equivalent, set of basis functions).  Corrolary 12 looks like it might be useful (not complete) except that i cannot decode what "$T_0 \binom{2n}{n}$" means.
Comment: It means the polynomial $T_0$ times the binomial coefficient $\binom{2n}{n}$.
Comment: $T_0(x)$ evaluated at what value of $x$?


Correct Use of XML datatype in SQL Server?

Question: I have an application that allows my users to make temporary changes to an ongoing booking.  I have recenlty read about the new XML datatype in SQL server, and the corresponding SqlXml type.
I am thinking of using the new type to store the previous values of the object so that I can later revert to them.
Something like:
Booking Table:
BookingID int
SomeField1 String
SomeField2 String
SomeField3 String
RevertValues XML
RevertDateTime DateTime2

This is a cut down version of my table, but I plan to store the current values in the ReverValues column and then revert when the RevertDatetime is reached.
My application uses a SQL Server 2008 database,  Linq-to-SQL model, with a MVC front end.
Any and all advice would be appreciated as I've never done anything like this before!
Thanks
Answer: Sounds like a good idea to me - I'm doing similar things in SQL Server auditing where I keep audit entries in a table, along with two XML fields - ValuesBefore and ValuesAfter - to document and track changes.
What you might want to do is split out the "RevertValues" into a separate table that has a foreign key relationship to your bookings, so that you could keep track of several revisions of the booking - something like a BookingHistory table:
BookingHistory
BookingID            INT   (FK to Booking table)
DateOfModification   DATETIME
BookingData          XML

or something along those lines.
Marc
Comment: Thanks Marc. How to you store/extract your object in the column?  Serialise/Deserialise methods?  Maybe put xml attributes in my partial class?
Comment: In my case, I simply serialize the object in question to XML and store that. Works fine. I wouldn't store that in your object - but provide a method on the class to render the class contents as XML (return a string)


H2 data base remote connection with mixed mode (Auto_server= true)

Question: I need to install same application on two computers. I want to use h2 database in mixed mode i.e AUTO_SERVER= TRUE. But my problem is that how to connect to this database from computer 2 while database files are stored in Computer 1.
as per docs, url ="jdbc:h2:/dbfile;AUTO_SERVER= TRUE"
I am not able to get the DBPATH working for both the computers.
Please help.
Answer: You have to use Server mode.
Basically what you are trying to do is network communication. There should be an address, so you need to make Computer1 with static IP (local).
In my understand The meaning of mixed mode is you can access same db with two or more Apps not different computers.


Change keyboard shortcut to comment out lines in Spyder

Question: I recently started using the Spyder IDE (Python 3.6) and I'm having some difficulties getting used to the keyboard shortcuts, which is complicated due to me using an azerty keyboard. The ctrl + 1 keyboard shortcut to comment out lines becomes ctrl + shift + 1 due to this (azerty keyboards only read numbers when holding the shift key). Therefore I was wondering if there was any way to change the keyboard shortcuts in Spyder.
Thank you!
Answer: (Spyder maintainer here) To change our shortcuts you need to go to
Tools > Preferences > Keyboard shortcuts
and introduce the one you want to use.
Comment: I see how to change shortcuts there, but is it possible to add new shortcuts?
Comment: Unfortunately, that's not possible. A shortcut needs to be declared in Spyder's source code and exposed in the Preferences entry I mentioned above so users can modify it.


Phonegap App crashing on Galaxy Note when orientation changes

Question: I have a couple of complete phonegap apps, fully functional. I've tried in the whole set of devices I have access to, and it doesn't run on my galaxy note. It's a brand new device, I can run native apps, but whenever I run any phonegap app i get a black screen and the app closes. No crash, no alert, no nothing.
(I remove the logs here because they were not relevant)
EDIT
I've tracked down the issue and it works when the orientation changes. As soon as I stat my activity it sets the orientation to portrait.
setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_PORTRAIT);

So I removed that lane, activated the option to change the orientation when I rotate the device. Now when I start the app it runs, but when I rotate it, it crashes. I can't figure out why. I have several alerts in my js code and none of them is fired. The interesting thing is the cordova default app also crashes when I rotate the device. Below you can find both logs.
Any suggestions on how to fix this?
DroidgapExample log:
11-27 12:41:15.750: D/DroidGap(9129): Paused the application!
11-27 12:41:15.750: D/CordovaWebView(9129): Handle the pause
11-27 12:41:15.750: D/CordovaWebView(9129): >>> loadUrlNow()
11-27 12:41:15.750: D/webkit(9129): Firewall not null
11-27 12:41:15.750: D/webkit(9129): euler: isUrlBlocked = false
11-27 12:41:15.760: D/DroidGap(9129): onDestroy()
11-27 12:41:15.760: D/CordovaWebView(9129): >>> loadUrlNow()
11-27 12:41:15.760: D/webkit(9129): Firewall not null
11-27 12:41:15.765: D/webkit(9129): euler: isUrlBlocked = false
11-27 12:41:15.765: D/CordovaWebView(9129): >>> loadUrlNow()
11-27 12:41:15.765: D/webkit(9129): Firewall not null
11-27 12:41:15.765: D/webkit(9129): euler: isUrlBlocked = false
11-27 12:41:15.775: D/DroidGap(9129): DroidGap.onCreate()
11-27 12:41:15.780: I/webclipboard(9129): clipservice: android.sec.clipboard.ClipboardExManager@4149bc80
11-27 12:41:15.785: D/CordovaWebView(9129): Origin to allow: http://127.0.0.1*
11-27 12:41:15.785: D/CordovaWebView(9129): Origin to allow: .*
11-27 12:41:15.785: I/CordovaLog(9129): Found log level DEBUG
11-27 12:41:15.785: I/CordovaLog(9129): Changing log level to DEBUG(3)
11-27 12:41:15.785: I/CordovaLog(9129): Found preference for useBrowserHistory=false
11-27 12:41:15.785: D/CordovaLog(9129): Found preference for useBrowserHistory=false
11-27 12:41:15.785: I/CordovaLog(9129): Found preference for exit-on-suspend=false
11-27 12:41:15.785: D/CordovaLog(9129): Found preference for exit-on-suspend=false
11-27 12:41:15.785: D/JsMessageQueue(9129): Set native->JS mode to 1
11-27 12:41:15.785: D/DroidGap(9129): DroidGap.init()
11-27 12:41:15.790: D/CordovaWebView(9129): >>> loadUrl(file:///android_asset/www/index.html)
11-27 12:41:15.790: D/PluginManager(9129): init()
11-27 12:41:15.795: D/CordovaWebView(9129): >>> loadUrlNow()
11-27 12:41:15.800: D/DroidGap(9129): Resuming the App
11-27 12:41:15.800: D/CordovaLog(9129): JSCallback Error: Request failed with status 0
11-27 12:41:15.800: D/CordovaLog(9129): file:///android_asset/www/cordova-2.1.0.js: Line 3743 : JSCallback Error: Request failed with status 0
11-27 12:41:15.800: I/Web Console(9129): JSCallback Error: Request failed with status 0 at file:///android_asset/www/cordova-2.1.0.js:3743
11-27 12:41:15.800: D/JsMessageQueue(9129): Set native->JS mode to 0
11-27 12:41:15.800: D/WML_SISO(9129): InitPasteboardJni
11-27 12:41:15.805: D/SoftKeyboardDetect(9129): Ignore this event
11-27 12:41:15.810: D/SoftKeyboardDetect(9129): Ignore this event
11-27 12:41:15.810: D/JsMessageQueue(9129): Set native->JS mode to 1
11-27 12:41:15.810: D/DroidGap(9129): onMessage(onPageStarted,about:blank)
11-27 12:41:15.820: W/webcore(9129): java.lang.Throwable: EventHub.removeMessages(int what = 107) is not supported before the WebViewCore is set up.
11-27 12:41:15.820: W/webcore(9129):    at android.webkit.WebViewCore$EventHub.removeMessages(WebViewCore.java:2681)
11-27 12:41:15.820: W/webcore(9129):    at android.webkit.WebViewCore$EventHub.access$13000(WebViewCore.java:1365)
11-27 12:41:15.820: W/webcore(9129):    at android.webkit.WebViewCore.removeMessages(WebViewCore.java:2793)
11-27 12:41:15.820: W/webcore(9129):    at android.webkit.WebView.sendOurVisibleRect(WebView.java:4046)
11-27 12:41:15.820: W/webcore(9129):    at android.webkit.ZoomManager.setZoomScale(ZoomManager.java:1024)
11-27 12:41:15.820: W/webcore(9129):    at android.webkit.ZoomManager.access$2100(ZoomManager.java:57)
11-27 12:41:15.820: W/webcore(9129):    at android.webkit.ZoomManager$PostScale.run(ZoomManager.java:1595)
11-27 12:41:15.820: W/webcore(9129):    at android.os.Handler.handleCallback(Handler.java:605)
11-27 12:41:15.820: W/webcore(9129):    at android.os.Handler.dispatchMessage(Handler.java:92)
11-27 12:41:15.820: W/webcore(9129):    at android.os.Looper.loop(Looper.java:137)
11-27 12:41:15.820: W/webcore(9129):    at android.app.ActivityThread.main(ActivityThread.java:4517)
11-27 12:41:15.820: W/webcore(9129):    at java.lang.reflect.Method.invokeNative(Native Method)
11-27 12:41:15.820: W/webcore(9129):    at java.lang.reflect.Method.invoke(Method.java:511)
11-27 12:41:15.820: W/webcore(9129):    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:995)
11-27 12:41:15.820: W/webcore(9129):    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:762)
11-27 12:41:15.820: W/webcore(9129):    at dalvik.system.NativeStart.main(Native Method)
11-27 12:41:15.820: D/SoftKeyboardDetect(9129): Ignore this event
11-27 12:41:15.825: I/GATE(9129): <GATE-M>DEV_ACTION_COMPLETED</GATE-M>
11-27 12:41:15.830: V/(9129): NotificationPresenterImpl::setSettingsValue 0
11-27 12:41:15.830: V/(9129): NotificationPresenterImpl::setSettingsValue 0
11-27 12:41:15.835: D/webkit(9129): Firewall not null
11-27 12:41:15.835: D/webkit(9129): euler: isUrlBlocked = false
11-27 12:41:15.845: D/Cordova(9129): onPageFinished(about:blank)
11-27 12:41:15.845: D/DroidGap(9129): onMessage(onPageFinished,about:blank)
11-27 12:41:15.845: D/DroidGap(9129): onMessage(exit,null)
11-27 12:41:15.900: I/GATE(9129): <GATE-M>DEV_ACTION_COMPLETED</GATE-M>
11-27 12:41:16.245: D/DroidGap(9129): onMessage(onPageStarted,file:///android_asset/www/index.html)
11-27 12:41:16.245: D/Cordova(9129): onPageFinished(file:///android_asset/www/index.html)
11-27 12:41:16.245: D/CordovaWebView(9129): >>> loadUrlNow()
11-27 12:41:16.245: D/webkit(9129): Firewall not null
11-27 12:41:16.245: D/webkit(9129): euler: isUrlBlocked = false
11-27 12:41:16.245: D/DroidGap(9129): onMessage(onNativeReady,null)
11-27 12:41:16.245: D/DroidGap(9129): onMessage(onPageFinished,file:///android_asset/www/index.html)
11-27 12:41:16.250: D/DroidGap(9129): Paused the application!
11-27 12:41:16.250: D/CordovaWebView(9129): Handle the pause
11-27 12:41:16.250: D/CordovaWebView(9129): >>> loadUrlNow()
11-27 12:41:16.360: D/DroidGap(9129): onMessage(networkconnection,wifi)
11-27 12:41:16.370: D/DroidGap(9129): onMessage(spinner,stop)
11-27 12:41:16.375: D/webkit(9129): Firewall not null
11-27 12:41:16.375: D/webkit(9129): euler: isUrlBlocked = false
11-27 12:41:16.380: D/CordovaLog(9129): Received Event: deviceready
11-27 12:41:16.380: D/CordovaLog(9129): file:///android_asset/www/js/index.js: Line 47 : Received Event: deviceready
11-27 12:41:16.380: I/Web Console(9129): Received Event: deviceready at file:///android_asset/www/js/index.js:47
11-27 12:41:17.635: D/DroidGap(9129): onDestroy()
11-27 12:41:17.635: D/CordovaWebView(9129): >>> loadUrlNow()
11-27 12:41:17.640: D/webkit(9129): Firewall not null
11-27 12:41:17.640: D/CordovaWebView(9129): >>> loadUrlNow()
11-27 12:41:17.640: D/webkit(9129): euler: isUrlBlocked = false
11-27 12:41:17.640: D/webkit(9129): Firewall not null
11-27 12:41:17.640: D/webkit(9129): euler: isUrlBlocked = false
11-27 12:41:17.650: D/CordovaLog(9129): JSCallback Error: Request failed with status 0
11-27 12:41:17.650: D/CordovaLog(9129): file:///android_asset/www/cordova-2.1.0.js: Line 3743 : JSCallback Error: Request failed with status 0
11-27 12:41:17.650: I/Web Console(9129): JSCallback Error: Request failed with status 0 at file:///android_asset/www/cordova-2.1.0.js:3743
11-27 12:41:17.650: D/JsMessageQueue(9129): Set native->JS mode to 0
11-27 12:41:17.650: D/JsMessageQueue(9129): Set native->JS mode to 1
11-27 12:41:17.660: D/DroidGap(9129): onMessage(onPageStarted,about:blank)
11-27 12:41:17.670: I/GATE(9129): <GATE-M>DEV_ACTION_COMPLETED</GATE-M>
11-27 12:41:17.670: D/Cordova(9129): onPageFinished(about:blank)
11-27 12:41:17.670: D/DroidGap(9129): onMessage(onPageFinished,about:blank)
11-27 12:41:17.670: D/DroidGap(9129): onMessage(exit,null)
11-27 12:41:18.255: D/DroidGap(9129): onMessage(spinner,stop)

My App log:
11-27 12:36:41.810: D/DroidGap(10500): Paused the application!
11-27 12:36:41.810: D/CordovaWebView(10500): Handle the pause
11-27 12:36:41.810: D/CordovaWebView(10500): >>> loadUrlNow()
11-27 12:36:41.810: D/webkit(10500): Firewall not null
11-27 12:36:41.810: D/webkit(10500): euler: isUrlBlocked = false
11-27 12:36:41.815: D/DroidGap(10500): onDestroy()
11-27 12:36:41.850: D/CordovaWebView(10500): >>> loadUrlNow()
11-27 12:36:41.850: D/webkit(10500): Firewall not null
11-27 12:36:41.850: D/CordovaWebView(10500): >>> loadUrlNow()
11-27 12:36:41.850: D/webkit(10500): euler: isUrlBlocked = false
11-27 12:36:41.855: D/webkit(10500): Firewall not null
11-27 12:36:41.860: D/webkit(10500): euler: isUrlBlocked = false
11-27 12:36:41.865: D/DroidGap(10500): DroidGap.onCreate()
11-27 12:36:41.875: I/webclipboard(10500): clipservice: android.sec.clipboard.ClipboardExManager@4138e4e0
11-27 12:36:41.880: I/CordovaLog(10500): config.xml missing, reverting to cordova.xml
11-27 12:36:41.880: D/CordovaWebView(10500): Origin to allow: http://127.0.0.1*
11-27 12:36:41.880: I/CordovaLog(10500): Found log level DEBUG
11-27 12:36:41.880: I/CordovaLog(10500): Changing log level to DEBUG(3)
11-27 12:36:41.880: I/CordovaLog(10500): Found preference for classicRender=true
11-27 12:36:41.880: D/CordovaLog(10500): Found preference for classicRender=true
11-27 12:36:41.880: D/JsMessageQueue(10500): Set native->JS mode to 1
11-27 12:36:41.895: D/DroidGap(10500): DroidGap.init()
11-27 12:36:41.900: D/CordovaWebView(10500): >>> loadUrl(file:///android_asset/www/page_A_0.html)
11-27 12:36:41.900: D/PluginManager(10500): init()
11-27 12:36:41.900: I/PluginManager(10500): Using plugins.xml instead of config.xml.  plugins.xml will eventually be deprecated
11-27 12:36:41.905: D/CordovaWebView(10500): >>> loadUrlNow()
11-27 12:36:41.910: D/DroidGap(10500): Resuming the App
11-27 12:36:41.915: I/GATE(10500): <GATE-M>DEV_ACTION_COMPLETED</GATE-M>
11-27 12:36:41.915: D/CordovaLog(10500): exception firing pause event from native
11-27 12:36:41.915: D/CordovaLog(10500): null: Line 1 : exception firing pause event from native
11-27 12:36:41.915: I/Web Console(10500): exception firing pause event from native at null:1
11-27 12:36:41.915: D/CordovaLog(10500): exception firing destroy event from native
11-27 12:36:41.915: D/CordovaLog(10500): null: Line 1 : exception firing destroy event from native
11-27 12:36:41.915: I/Web Console(10500): exception firing destroy event from native at null:1
11-27 12:36:41.915: W/System.err(10500): java.net.SocketException: Socket closed
11-27 12:36:41.915: W/System.err(10500):    at libcore.io.Posix.accept(Native Method)
11-27 12:36:41.915: W/System.err(10500):    at libcore.io.BlockGuardOs.accept(BlockGuardOs.java:55)
11-27 12:36:41.915: W/System.err(10500):    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:98)
11-27 12:36:41.915: W/System.err(10500):    at java.net.ServerSocket.implAccept(ServerSocket.java:202)
11-27 12:36:41.915: W/System.err(10500):    at java.net.ServerSocket.accept(ServerSocket.java:127)
11-27 12:36:41.920: W/System.err(10500):    at org.apache.cordova.CallbackServer.run(CallbackServer.java:195)
11-27 12:36:41.920: W/System.err(10500):    at java.lang.Thread.run(Thread.java:856)
11-27 12:36:41.920: D/DroidGap(10500): onMessage(onPageStarted,about:blank)
11-27 12:36:41.920: D/WML_SISO(10500): InitPasteboardJni
11-27 12:36:41.920: D/SoftKeyboardDetect(10500): Ignore this event
11-27 12:36:41.920: V/(10500): NotificationPresenterImpl::setSettingsValue 0
11-27 12:36:41.920: V/(10500): NotificationPresenterImpl::setSettingsValue 0
11-27 12:36:41.925: D/webkit(10500): Firewall not null
11-27 12:36:41.925: D/webkit(10500): euler: isUrlBlocked = false
11-27 12:36:41.930: D/SoftKeyboardDetect(10500): Ignore this event
11-27 12:36:41.935: D/Cordova(10500): onPageFinished(about:blank)
11-27 12:36:41.935: D/DroidGap(10500): onMessage(onPageFinished,about:blank)
11-27 12:36:41.935: D/DroidGap(10500): onMessage(exit,null)
11-27 12:36:41.970: D/chromium(10500): Unknown chromium error: -6
11-27 12:36:42.315: D/DroidGap(10500): onMessage(onPageStarted,file:///android_asset/www/page_A_0.html)
11-27 12:36:42.315: D/SoftKeyboardDetect(10500): Ignore this event
11-27 12:36:42.360: D/DroidGap(10500): Paused the application!
11-27 12:36:42.360: D/CordovaWebView(10500): Handle the pause
11-27 12:36:42.360: D/CordovaWebView(10500): >>> loadUrlNow()
11-27 12:36:42.555: W/IInputConnectionWrapper(10500): showStatusIcon on inactive InputConnection
11-27 12:36:43.465: D/DroidGap(10500): onDestroy()
11-27 12:36:43.465: D/CordovaWebView(10500): >>> loadUrlNow()
11-27 12:36:43.465: D/CordovaWebView(10500): >>> loadUrlNow()
11-27 12:36:43.480: E/WindowManager(10500): Activity co.uk.precedent.tesco.CordovaTuxedoActivity has leaked window com.android.internal.policy.impl.PhoneWindow$DecorView@41449660 that was originally added here
11-27 12:36:43.480: E/WindowManager(10500): android.view.WindowLeaked: Activity co.uk.precedent.tesco.CordovaTuxedoActivity has leaked window com.android.internal.policy.impl.PhoneWindow$DecorView@41449660 that was originally added here
11-27 12:36:43.480: E/WindowManager(10500):     at android.view.ViewRootImpl.<init>(ViewRootImpl.java:414)
11-27 12:36:43.480: E/WindowManager(10500):     at android.view.WindowManagerImpl.addView(WindowManagerImpl.java:282)
11-27 12:36:43.480: E/WindowManager(10500):     at android.view.WindowManagerImpl.addView(WindowManagerImpl.java:218)
11-27 12:36:43.480: E/WindowManager(10500):     at android.view.WindowManagerImpl$CompatModeWrapper.addView(WindowManagerImpl.java:143)
11-27 12:36:43.480: E/WindowManager(10500):     at android.view.Window$LocalWindowManager.addView(Window.java:548)
11-27 12:36:43.480: E/WindowManager(10500):     at android.app.Dialog.show(Dialog.java:337)
11-27 12:36:43.480: E/WindowManager(10500):     at android.app.AlertDialog$Builder.show(AlertDialog.java:932)
11-27 12:36:43.480: E/WindowManager(10500):     at org.apache.cordova.CordovaChromeClient.onJsAlert(CordovaChromeClient.java:119)
11-27 12:36:43.480: E/WindowManager(10500):     at android.webkit.CallbackProxy.handleMessage(CallbackProxy.java:628)
11-27 12:36:43.480: E/WindowManager(10500):     at android.os.Handler.dispatchMessage(Handler.java:99)
11-27 12:36:43.480: E/WindowManager(10500):     at android.os.Looper.loop(Looper.java:137)
11-27 12:36:43.480: E/WindowManager(10500):     at android.app.ActivityThread.main(ActivityThread.java:4517)
11-27 12:36:43.480: E/WindowManager(10500):     at java.lang.reflect.Method.invokeNative(Native Method)
11-27 12:36:43.480: E/WindowManager(10500):     at java.lang.reflect.Method.invoke(Method.java:511)
11-27 12:36:43.480: E/WindowManager(10500):     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:995)
11-27 12:36:43.480: E/WindowManager(10500):     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:762)
11-27 12:36:43.480: E/WindowManager(10500):     at dalvik.system.NativeStart.main(Native Method)

Comment: any luck fixing the problem? I have a similar problem on Google Nexus S phone.
Answer: Firstly, it isn't a device compatibility issue - I have a Galaxy Note here that runs PhoneGap apps without any problems.
Have you tried running the default Cordova template app to rule out more complex app-specific issues?
The firewall messages suggest that your app is trying to load something over the network during startup - your question doesn't give any information on what that might be, but it shouldn't be too hard to track down.
Comment: Thanks for the suggestion. I've updated my question with more details on my problem!
Answer: I've sorted this a while ago, with this answer.
adding the screenSize property to my config changes attribute in every activity solved the problem. Bear in mind it was added in API level 13.
android:configChanges="orientation|screenSize|keyboardHidden"


Show that the Lie algebra of $\operatorname{Aut}(\mathfrak{g})$ is given by $\operatorname{Der}(\mathfrak{g})$.

Question: Notation:
Let $\mathfrak{g}$ be a Lie algebra, $\operatorname{Aut}(\mathfrak{g})$ denotes the set of automorphisms. $\operatorname{Der}(\mathfrak{g})$ denotes the set of derivations of $\mathfrak{g}$. That is is if $T\in \operatorname{Der}(\mathfrak{g})$ then $T[X,Y]=[T(X),Y]+[X,T(Y)]$.
I'm trying to show that the Lie algebra of $\operatorname{Aut}(\mathfrak{g})$ is given by $\operatorname{Der}(\mathfrak{g})$, I am stuck on a specific step in the proof which I will mark in bold.
Let $T$ be in the Lie algebra of $\operatorname{Aut}(\mathfrak{g})$. Then $\exp(tT)\in \operatorname{Aut}(\mathfrak{g})$.
Therefore $\exp(tT)[X,Y]=[\exp(tT)X,\exp(tT)Y]$. Taking  the derivative derivative at $t=0$ gives $T[X,Y]$ on the left-hand side but I am not sure how to take the derivative of the right hand side.
Obviously the answer should be $[TX,Y]+[X,TY]$, this is easy to see if $[\,,]$ was a commutator bracket but for general Lie algebras this is not always the case.
Answer: $${\Bbb d\over {\Bbb dt}}\big[\exp(tT),\exp(tT)Y)\big]=\left[{\Bbb d\over {\Bbb dt}}\exp(tT)X,\exp(tT)Y\right]+\left[\exp(tT)X,{\Bbb d\over{\Bbb dt}}\exp(tT)Y\right]$$
Since the function $b:(X,Y)\rightarrow [X,Y]$ is bilinear, $\Bbb db_{(X,Y)}(U,V)=[U,Y]+[X,V]$ and $t\rightarrow [\exp(tT),\exp(tT)Y)]$ is the composition of $b$ and $t\rightarrow (\exp(tT)X, \exp(tT)Y)$, the result follows from the chain rule.


Watermarking in Spark Structured Streaming 2.3.0

Question: I read data from Kafka in Spark Structured Streaming 2.3.0. The data contains information about some teachers, there is teacherId, teacherName and teacherGroupsIds. TeacherGroupsIds is an array column which contains ids of the group. In my task I have to map the column with group ids to column containing information about group names([1,2,3] => [Suns,Books,Flowers]). The names and ids are stored in HBase and can change everyday. Later I have to send the data to another Kafka topic.
So, I read data from two sources - Kafka and HBase. I read data from HBase using shc library.
First, I explode the array column (group ids), later I join with the data from HBase.
In next step I would like to aggregate the data using teacherId. But this operation is not supported in Append Mode which I use.
I have tried with watermarking, but at the moment it doesn't work. I added a new column with timestamp and I would group by this column.
Dataset<Row> inputDataset = //reading from Kafka

Dataset<Row> explodedDataset = // explode function applied and join with HBase

Dataset<Row> outputDataset = explodedDataset
.withColumn("eventTime", lit(current_timestamp()))
.withWatermark("eventTime", "2 minutes")
.groupBy(window(col("eventTime"), "5 seconds"), col("teacherId"))
.agg(collect_list(col("groupname")));

Actual results show empty dataframe at the output. There is not any row.
Comment: I use "Append mode" to write data to another Kafka topic.
Answer: The problem is current_timestamp().
current_timestamp returns the timestamp in that moment, so, if you create a dataframe with this column and print the result, you print the current timestamp, but if you process the df and you print the same column, you print the new timestamp.
This solution works locally, but sometimes in a distributed system it fails because the workers when receiving the order to print the data, this data is already outside the timestamp range.


Split tunneling through two VPNs simeltaneously

Question: I want to run two OpenVPN client instances on an Ubuntu Server 14.04. I have both .conf files that can work independently (both are set to different interfaces - tun0 and tun1). I want to run both at the same time, and route traffic from one application into one VPN (Private Internet Access) and all other traffic into the other VPN (An OpenVPN server I have set up on another machine).
I've been doing some research but haven't found a way to make this work. If this helps, I've included my current routing table and one of the server .confs. OpenVPN starts them automatically on boot.
Private Internet Access .conf
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.160.1.5      188.8.131.52       UG    0      0        0 tun0
default         126.96.36.199  0.0.0.0         UG    0      0        0 eth0
10.160.1.1      10.160.1.5      255.255.255.255 UGH   0      0        0 tun0
10.160.1.5      *               255.255.255.255 UH    0      0        0 tun0
64-237-37-119.c 220.127.116.11  255.255.255.255 UGH   0      0        0 eth0
126.96.36.199       10.160.1.5      220.127.116.11       UG    0      0        0 tun0
220.127.116.11    *               255.255.254.0   U     0      0        0 eth0

Comment: What networks are tun0 and tun1 connected to?
Answer: You need to add routes for your applications via the tunnels.
For example, if application A is at 10.70.82.5, and you want to route application A via Private Internet Access, you should run this command:
route add 10.70.82.5 gw "IP address of gateway at PIA"

Then, to route all other traffic via other VPN, you need to do two things:

Route traffic to the other VPN server via your normal default gateway
route add "IP address of other VPN server" gw 220.127.116.11

Here I assume this is your normal default gateway, based on your routing table in your post.

Make a default route via other VPN server gateway
route add default gw "Other VPN server gateway IP"

Remember that both OpenVPN servers need to do NAT in order for return packets to arrive correctly via the VPN.
Comment: This might sound stupid, but how do I find the IP address of the application? I'm sure I understand that. I have a Deluge server, would I just use localhost:port to forward the traffic through PIA?
Comment: While running the application, you can run `netstat -np` in a shell to see where the application connects to. Then you can route that destination IP via one VPN.
Comment: Okay, so this is my deluge daemon:

`tcp        0      0 127.0.0.1:58846         127.0.0.1:48582         ESTABLISHED -               `

Would I then use
`route add 127.0.0.1:58846 gw "IP address of gateway at PIA"`? Or would I use the foreign address?
Comment: This deluge daemon doesn't communicate anywhere outside your network, so you cannot route it anywhere with OpenVPN. It seems that your original question lacks a lot of information in order to answer it properly.
Comment: It does send traffic outside the network, but it's all bitorrent traffic, it doesn't go to a single address.
Comment: Not sure I fully grok what you are trying but these may help: http://www.lartc.org/howto/lartc.rpdb.multiple-links.html http://linux-ip.net/html/adv-multi-internet.html http://unix.stackexchange.com/questions/104830/block-specific-application-with-iptables


Como fazer para esconder e habilitar uma linha de uma table?

Question: Não sei o que acontece, mas quando inicio a tela com o display block no id impar, tudo ok
porém quando desabilito, e habilito novamente, os campos das colunas aparecem tudo na
primeira coluna

function Mudarestado(el) {
  var display = document.getElementById(el).style.display;
  if (display == "none") {
    document.getElementById(el).style.display = 'block';
  } else {
    document.getElementById(el).style.display = 'none';
  }
}
table {
  width: 100%;
  height: 100%;
}

thead tr {
  background-color: ActiveCaption;
  color: CaptionText;
}

th,
td {
  vertical-align: top;
  font-family: "Tahoma";
  font-size: 10pt;
  text-align: left;
  cursor: pointer;
}

#impar {
  display: none;
}

      COLUNA 1
      COLUNA 2
      COLUNA 3

      CAMPO DA COLUNA 1
      CAMPO DA COLUNA 2
      CAMPO DA COLUNA 3

      CAMPO OBSERVACAO DA COLUNA 1
      CAMPO OBSERVACAO DA COLUNA 2
      CAMPO OBSERVACAO DA COLUNA 3

Answer: Isso acontece porque o valor de display do elemento  não é block, mas sim table-row.
Esta ajuste na função Mudarestado() resolve o problema:

function Mudarestado(el) {
  var display = document.getElementById(el).style.display;
  if (display == "none") {
    document.getElementById(el).style.display = 'table-row';
  } else {
    document.getElementById(el).style.display = 'none';
  }
}

        COLUNA 1
        COLUNA 2
        COLUNA 3

        CAMPO DA COLUNA 1
        CAMPO DA COLUNA 2
        CAMPO DA COLUNA 3

        CAMPO OBSERVACAO DA COLUNA 1
        CAMPO OBSERVACAO DA COLUNA 2
        CAMPO OBSERVACAO DA COLUNA 3

Edit: removi o CSS, e adicionei o display: none de forma inline na linha #impar, pois estava causando um problema no primeiro clique, já que o element.style.display somente lê estilos inline.
Comment: Muito obrigado!!!
Answer: Olá, no lugar do 'block', coloca o 'table-row'. Espero ter ajudado
document.getElementById(el).style.display = 'table-row';

function Mudarestado(el) {
  var display = document.getElementById(el).style.display;
  if (display == "none") {
    document.getElementById(el).style.display = 'table-row';
  } else {
    document.getElementById(el).style.display = 'none';
  }
}
table {
  width: 100%;
  height: 100%;
}

thead tr {
  background-color: ActiveCaption;
  color: CaptionText;
}

th,
td {
  vertical-align: top;
  font-family: "Tahoma";
  font-size: 10pt;
  text-align: left;
  cursor: pointer;
}

#impar {
  display: none;
}

      COLUNA 1
      COLUNA 2
      COLUNA 3

      CAMPO DA COLUNA 1
      CAMPO DA COLUNA 2
      CAMPO DA COLUNA 3

      CAMPO OBSERVACAO DA COLUNA 1
      CAMPO OBSERVACAO DA COLUNA 2
      CAMPO OBSERVACAO DA COLUNA 3

Comment: Agora sim!!
Muito obrigado, era isso que tava faltando


SQL Like is accepting only "context"

Question: Hi I have problem in my DB Query.
$q = $this->db->query("SELECT * FROM (`knihy`)
                      WHERE `stav` = 1
                      AND (Autor1 LIKE '$vyraz'
                      OR Autor2 LIKE '$vyraz'
                      OR Autor3 LIKE '$vyraz'
                      OR Autor4 LIKE '$vyraz')
                      ORDER BY `id` desc LIMIT $limit OFFSET $offset ");

The problem is that, when I am searching for example for "lorem" and I have in my table for example in row Autor1 content: "lorem ipsum" nothing will be found. But if I am searching for lorem ipsum there will be found those one field. Where is problem please?
Comment: add for LIKE params '%' around
Answer: You're not using the right syntax for LIKE:
Autor1 LIKE '$vyraz'

Should be:
Autor1 LIKE '%$vyraz%'

Without the wildcards, you're just doing an exact match.
Comment: Remember that using an initial wildcard will throw out your index.  Be careful!
Comment: @AlainCollins - you know, that hadn't ever occurred to me. That's something new I've learned today!
Comment: what you mean with "initial wildcard will throw out your index" ?
Comment: @Geril - if you're using an index on the column, it won't work if you have a wildcard at the start of your LIKE. Your index contains all the values in that column, sorted - so if you have a wildcard at the start of your LIKE, the database will still have to search the column. You'll still get the results, but the performance won't be as fast.

SQL Like is accepting only "context"


how to deal with string.split by position

Question: I'd like to ask one question about String.Split
For example:
char[] semicolon=new [] {';'};
char[] bracket=new [] {'[',']'};
string str="AND[Firstpart;Sndpart]";

I can split str by bracket and then split by semicolon.
Finally,I get the Firstpart and Sndpart in the bracket.
But If str="AND[AND[Firstpart;Sndpart];sndpart];
How can I get AND[Firpart;Sndpart] and sndpart?
Is there a way to tell c# to split by second semicolon?
Thanks for your help
Comment: I think you'll need to implement a parser for this, `string.Split` isn't really appropriate for handling syntax rules.
Comment: one way. when you get inside bracket `[` hide the specific characters. how to hide? change its char with something not really in use. example: `AND[AND\u9999Firstpart\u9997Sndpart\u9998;sndpart];`. then unhide after splitting.
Comment: @ Charles Mager could you explain a little bit your idea about Parser?
Comment: @  M.kazem Akhgary it's true that bracket can be hidden. May I ask you one questions? how can I replace bracket by \u?
Comment: Ill give answer ASAP when i come back home. Thats not \u. Its \uXXXX and X is number in base 16. Represents a unicode character.
Answer: One way is to hide characters inside bracket with a character that is not used in any of your strings.
Method HideSplit: This method will change separator characters inside brackets with fake ones. Then it will perform split and will give back the result with original characters.
This method maybe an overkill if you want to do this many times. but you should be able to optimize it easily if you got the idea.
    private static void Main()
    {
        char[] semicolon = new[] { ';' };
        char[] bracket = new[] { '[', ']' };
        string str = "AND[AND[Firstpart;Sndpart];sndpart]";

        string[] splitbyBracket = HideSplit(str, bracket);
    }

    private static string[] HideSplit(string str,char[] separator)
    {
        int counter = 0; // When counter is more than 0 it means we are inside brackets
        StringBuilder result = new StringBuilder(); // To build up string as result

        foreach (char ch in str)
        {
            if(ch == ']') counter--;

            if (counter > 0) // if we are inside brackets perform hide
            {
                if (ch == '[') result.Append('\uFFF0'); // add '\uFFF0' instead of '['
                else if (ch == ']') result.Append('\uFFF1');
                else if (ch == ';') result.Append('\uFFF2');
                else result.Append(ch);
            }
            else result.Append(ch);

            if (ch == '[') counter++;
        }

        string[] split = result.ToString().Split(separator); // Perform split. (characters are hidden now)

        return split.Select(x => x
            .Replace('\uFFF0', '[')
            .Replace('\uFFF1', ']')
            .Replace('\uFFF2', ';')).ToArray(); // unhide characters and give back result.
                                               // dont forget: using System.Linq;
    }

Some examples :
string[] a1 = HideSplit("AND[AND[Firstpart;Sndpart];sndpart]", bracket);
// Will give you this array { AND , AND[Firstpart;Sndpart];sndpart }

string[] a2 = HideSplit("AND[Firstpart;Sndpart];sndpart", semicolon);
// Will give you this array { AND[Firstpart;Sndpart] , sndpart }

string[] a3 = HideSplit("AND[Firstpart;Sndpart]", bracket);
// Will give you this array { AND , Firstpart;Sndpart }

string[] a4 = HideSplit("Firstpart;Sndpart", semicolon);
// Will give you this array { Firstpart , Sndpart }

And you can continue splitting this way.
Answer: You can use Regex.Split, which is a more flexible form of String.Split:
string str = "AND[AND[Firstpart;Sndpart];sndpart]";

string[] arr = Regex.Split(str, @"(.*?;.*?;)");

foreach (var s in arr)
    Console.WriteLine("'{0}'", s);

// output: ''
//         'AND[AND[Firstpart;Sndpart];'
//         'sndpart]'

Regex.Split splits not by chars, but by a string matching a regex expression, so it comes down to constructing a regex pattern meeting particular requirements. Splitting by a second semicolon is in practice splitting by a string that ends in a semicolon and that contains another semicolon before, so the matching pattern by which you split the input string could be for example: (.*?;.*?;).
The returned array has three elements instead of two because the splitting regex matches the beginning of the input string, in this case the empty string is returned as the first element.
You can read more on Regex.Split on msdn.
Answer:
Is there a way to tell c# to split by second semicolon?

There is no direct way to do that, but if that is precisely what you want, it's not hard to achieve:
string str="AND[AND[Firstpart;Sndpart];sndpart];
string[] tSplits = str.Split(';', 3);
string[] splits = { tSplits[0] + ";" + tSplits[1], tSplits[2] };

You could achieve the same result using a combination of IndexOf() and Substring(), however that is most likely not what you'll end up using as it's too specific and not very helpful for various inputs.
For your case, you need something that understands context.
In real-world complex cases you'd probably use a lexer / parser, but that seems like an overkill here.
Your best effort would probably be to use a loop, walk through all characters while counting +/- square brackets and spliting when you find a semicolon & the count is 1.


Difference between Hidden Markov models and Particle Filter (and Kalman Filter)

Question: Here is my old question
I would like to ask if someone knows the difference (if there is any difference) between Hidden Markov models (HMM) and Particle Filter (PF), and as a consequence Kalman Filter, or under which circumstances we use which algorithm. I'm a student and I have to do a project, but first I have to understand some things.
So, according to bibliography, both are State Space Models, including hidden (or latent or unobserved) states. According to Wikipedia (Hidden_Markov_model) "in HMM, the state space of the hidden variables is discrete, while the observations themselves can either be discrete (typically generated from a categorical distribution) or continuous (typically from a Gaussian distribution). Hidden Markov models can also be generalized to allow continuous state spaces. Examples of such models are those where the Markov process over hidden variables is a linear dynamical system, with a linear relationship among related variables and where all hidden and observed variables follow a Gaussian distribution. In simple cases, such as the linear dynamical system just mentioned, exact inference is tractable (in this case, using the Kalman filter); however, in general, exact inference in HMMs with continuous latent variables is infeasible, and approximate methods must be used, such as the extended Kalman filter or the particle filter."
But for me this is a bit confusing… In simple words does this mean the follow (based also to more research that I have done):

In HMM, the state space can be either discrete or continuous. Also the observations themselves can be either discrete or continuous. Also HMM is a linear and Gaussian or non-Gaussian dynamical system.
In PF, the state space can be either discrete or continuous. Also the observations themselves can be either discrete or continuous. But PF is a non-linear (and non-Gaussian?) dynamical system (is that their difference?).
Kalman filter (also looks like the same to me like HMM) is being used when we have linear and Gaussian dynamical system.

Also how do I know which algorithm to choose, because to me all these seem the same... Also I found a paper (not in English) which says that PF although can have linear data (for example raw data from a sensor-kinect which recognizes a movement), the dynamical system can be non-linear. Can this happen? Is this correct? How?
For gesture recognition, researchers can use either HMM or PF, but they don't explain why they select each algorithm… Does anyone know how I can be helped to distinguish these algorithms, to understand their differences and how to choose the best algorithm?
I'm sorry if my question is too big, or some parts are naive but I didn't find somewhere a convincing and scientific answer.
Thank you a lot in advance for your time!
Here is my NEW question (according to @conjugateprior's help)
So with further reading, I would like to update some of my parts of my previous comment, and to make sure that I understood a bit more what is going on.

Again in simple words, the umbrella is Dynamic Bayesian networks under which the models of HMM and State space are included (subclasses) (http://mlg.eng.cam.ac.uk/zoubin/papers/ijprai.pdf).
Furthermore, the initial difference between the 2 models is that, in HMM the hidden state variables are discrete, while the observations can either be discrete or continuous. In PF the hidden state variables are continuous (real-valued hidden state vector), and the observations have Gaussian distributions.
Also according to @conjugateprior each model has the 3 following tasks: filtering, smoothing and prediction. In filtering, the model HMM uses for discrete hidden state variables the method Forward algorithm, state space uses for continuous variables and linear dynamic system the Kalman Filter, etc.
However, HMM can also be generalized to allow continuous state spaces.
With these extensions of HMM, the 2 models seems to be conceptually identical (as it is also mentioned in Hidden Markov Model vs Markov Transition Model vs State-Space Model...?).

I think that I'm using a bit more accurate the terminology, but still everything is blurry to me. Can anyone explain to me what is the difference between HMM and State Space model?
Because I really cannot find an answer that can fit to my needs..
Thank you once more!
Comment: If your school's library has this book: https://www.crcpress.com/Time-Series-Modeling-Computation-and-Inference/Prado-West/9781420093360 I would take a look at it. It does a good job of explaining all three topics which, I might mention, are three very distinct topics.
Comment: I just checked that library doesn't have this book, unfortunately.. so if you could send me the parts that you believe that answer my question or help me to also distinct these topics it would be great! : )
Answer: It will be helpful to distinguish the model from inference you want to make with it, because now standard terminology mixes the two.
The model is the part where you specify the nature of: the hidden space (discrete or continuous), the hidden state dynamics (linear or non-linear) the nature of the observations (typically conditionally multinomial or Normal), and the measurement model connecting the hidden state to the observations.  HMMs and state space models are two such sets of model specifications.
For any such model there are three standard tasks: filtering, smoothing, and prediction.  Any time series text (or indeed google) should give you an idea of what they are.  Your question is about filtering, which is a way to get a) a posterior distribution over (or 'best' estimate of, for some sense of best, if you're not feeling Bayesian) the hidden state at $t$ given the complete set of of data up to and including time $t$, and relatedly b) the probability of the data under the model.
In situations where the state is continuous, the state dynamics and measurement linear and all noise is Normal, a Kalman Filter will do that job efficiently.  Its analogue when the state is discrete is the Forward Algorithm.  In the case where there is non-Normality and/or non-linearity, we fall back to approximate filters.  There are deterministic approximations, e.g. an Extended or Unscented Kalman Filters, and there are stochastic approximations, the best known of which being the Particle Filter.
The general feeling seems to be that in the presence of unavoidable non-linearity in the state or measurement parts or non-Normality in the observations (the common problem situations), one tries to get away with the cheapest approximation possible.  So, EKF then UKF then PF.
The literature on the Unscented Kalman filter usually has some comparisons of situations when it might work better than the traditional linearization of the Extended Kalman Filter.
The Particle Filter has almost complete generality - any non-linearity, any distributions - but it has in my experience required quite careful tuning and is generally much more unwieldy than the others.  In many situations however, it's the only option.
As for further reading: I like ch.4-7 of Särkkä's Bayesian Filtering and Smoothing, though it's quite terse.  The author makes has an online copy available for personal use.  Otherwise, most state space time series books will cover this material.  For Particle Filtering, there's a Doucet et al. volume on the topic, but I guess it's quite old now.  Perhaps others will point out a newer reference.
Comment: "What is the difference between HMM and State Space model?" Basically: By convention, HMMs have discrete state. Also by convention, 'state space models' denote things with continuous state.
Comment: @Dnaiel Add as much as you like, the OP has given up or taken what they needed.
Comment: @JeeyCi to the extend I get what you're asking, I think the answer is: measurement models like KF and HMM are like regular GLMs except that the independent variable is not observed (that's the hidden state). So when you fit the parameters of the model there's an extra step where you need to infer the hidden state (which is a variable not a parameter). A classic way to maximise the likelihood of the data under one of these models is the EM algorithm which (roughly) infers the expected state, then fits the GLM parameters as if the state were known, then repeats until convergence. (cont)
Comment: Posteriors are relevant because the GLM embedded in these models is P(data | hidden state, params) - in a regular GLM it would be P(Y | X, params), but here although Y is observed, X is the hidden state - so if we want P(hidden state | data, params), because we think of it as a measurement, but also because the EM algorithm demands its expected value, then we need to assume some prior P(hidden state) and use Bayes theorem to update that in the light of data. Hope that helps
Comment: First of all thank you very much for your answer. Please check that I edited the question above in order to be more coherent and accurate with the terminology. I also rephrase my whole question.
Comment: In your rewrite it's not quite true that "In PF the hidden state variables are continuous (real-valued hidden state vector), and the observations have Gaussian distributions."  PF is a filter.  The model *for which* it is a filter must have a continuous state space (with some Markovian structure or other) but is otherwise unconstrained: any distributions, any dynamics, and any measurement process.
Comment: You could, for example, use a PF to filter an ordinary linear Gaussian state space model.  It would work just fine.  You just don't have to since the Kalman filter is exact.
Comment: Thank you so much! So this difference is happened by convention (and according to bibliography). But in general, we can choose whichever model we want. Is that correct? The model that I'll choose depends on whether will I have better results?
Comment: @conjugateprior.  I'd add that in addition to prediction, filtering and smoothing one could also assume that the parameters are unknown and then the baum-welch/EM/SMC methods would prove useful in addition to the forward, forward-backward and Kalman filtering methods mentioned here.
Comment: @ conjugateprior:  could you make more exact for me, please, are these "posterior distribution & probability of the data" calculations also used in Generalized Linear Models or there is the other algorithm used in GLM, then Kalman filter ??


SOAPUI Groovy Script calls to Command Line

Question: I would like to run something like
ipconfig /all | find "IPv4"
that fetches the IP addresses for all the interfaces I'm connected to. This is straight forward in pure Java or C#, and they even have a managed API for this. I could run a 'command line' command or access managed objects to fetch the IP address of the particular interface I'm looking for.
I have SOAPUI Pro and am aware the API is here, but I cant find out how do either make a command line call and fetch the result- nor a direct way to get the IP addresses with that API.
Ultimately I want to store the runtime IP address in a project property.
Answer: This script runs the ipconfig /all command and gets the result into a variable. But the parsing to get the IP remains to be done. Currently the Groovy step will return the entire ipconfig result.
<code>def result = "ipconfig /all".execute().text

//Do stuff with the result, probably RegEx your way to the IpAdress

return result;
</code>


Android Studio shows wrong layout in layout preview (ActionBarOverlayLayout?!)

Question: I have a nice FrameLayout as main container and some other views inside it's hierarchy.
But the preview shows a simple ActionBarOverlayLayout.
What is that? Why is it here?
I have Android Studio 3.0.0
I have tried:
Restart Android Studio. Refresh the preview by resizing. Changed the preview device, changed the SDK of the preview, changed the blueprint\design options, pressed "force refresh layout" Button.
XML:
<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
android:layout_width="match_parent"

android:layout_height="wrap_content"
android:orientation="vertical"

android:paddingLeft="8dp"
android:paddingRight="8dp"
android:paddingTop="8dp">

<com.makeramen.roundedimageview.RoundedImageView
    android:layout_width="match_parent"
    android:layout_height="@dimen/walk_list_item_height"
    android:adjustViewBounds="false"
    android:cropToPadding="false"
    android:scaleType="centerCrop"
    app:layout_heightPercent="25%"
    app:layout_marginLeftPercent="0%"
    app:layout_marginTopPercent="0%"
    app:layout_widthPercent="100%"
    app:riv_corner_radius="@dimen/corner_radius"
    android:id="@+id/walk_iv" />

<LinearLayout
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:layout_gravity="bottom"
    android:background="@drawable/gradient"
    android:orientation="vertical"
    android:paddingBottom="16dp"
    android:paddingTop="16dp">

    <TextView
        android:id="@+id/walk_name_tv"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_gravity="center"
        android:gravity="center_vertical"
        android:padding="16dp"
        android:text="New Text"
        android:textAppearance="?android:attr/textAppearanceMedium"
        android:textColor="#fff" />

    <LinearLayout
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:orientation="horizontal"
        android:paddingLeft="16dp"
        android:paddingRight="16dp">

        <ImageView
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            app:srcCompat="@drawable/ic_duration_icon" />

        <TextView
            android:id="@+id/duration_tv"
            android:layout_gravity="center"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_marginLeft="8dp"
            android:gravity="center|left"
            android:text="TextView"
            android:textColor="#fff" />

        <ImageView
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            app:srcCompat="@drawable/ic_marker"
            android:layout_marginLeft="16dp" />

        <TextView
            android:id="@+id/stops_tv"
            android:layout_gravity="center"

            android:layout_marginLeft="8dp"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:gravity="center"
            android:text="TextView"
            android:textColor="#fff" />

    </LinearLayout>
</LinearLayout>

</FrameLayout>

Comment: I created a new xml in my project using your xml and everything works fine. Have you tried cleaning the project and rebuilding it?
Comment: @Ibrahim Exactly where?
Comment: share your build.gradle
Comment: There is nothing wrong with your code, I have created a new layout with your XML and it is working fine. Try clean build and share your `build.gradle`
Comment: I've also tried your code and it works fine. Go ahead and run it it's probably working fine.
Comment: Is this a framelayout that will be call by a fragment? And i think in your Activity layout, you have a `FrameLayout` that will call this fragment layout. Check again that activity
Answer: That is because there is a broken build in your earlier build. It is coming from the intermediate files which is because the files are not proper in your xml files. There might be a broken link or a - in the drawables names.
Until you resolve this you cannot proceed to see the layout of your app in the layout-editor.
Moreover Android Studio will not show you any error in the error log or any other terminal.
Start from the styles/themes of the app, it might contain something which is missing from the gradle or resources of the project. Due to the support library still in the project it partially showcases it as missing control of XML files, but it will not let you load the xml editor visually for the project.
You have see that by yourself, start first from drawables and then move to values folder, if not check your gradle and then come back to Java files for the error.
Answer: Had to raise

buildToolsVersion to 26.0.0

previous was bugged (25.0.2)
Answer: I faced the same problem and, In my case, it was simpler than that.  I just received a warning message recommending to remove the explict reference to the BuildToolsVerison from build.gradle file so I just did it, then saved all file, Menu >> Run >> Make a project and it worked fine.


Marathon kills container when requests increase

Question: I deploy Docker containers on Mesos(0.21) and Marathon(0.7.6) on Google Cloud Engine.
I use JMeter to test a REST service that run on Marathon. When the concurrent requests are less than 10, it works normal, but when the concurrent requests are over 50, the container is killed and Mesos start another container. I increase RAM, CPU but it still happens.
This is log in /var/log/mesos/
<code>
E0116 09:33:31.554816 19298 slave.cpp:2344] Failed to update resources for container 10e47946-4c54-4d64-9276-0ce94af31d44 of executor dev_service.2e25332d-964f-11e4-9004-42010af05efe running task dev_service.2e25332d-964f-11e4-9004-42010af05efe on status update for terminal task, destroying container: Failed to determine cgroup for the 'cpu' subsystem: Failed to read /proc/612/cgroup: Failed to open file '/proc/612/cgroup': No such file or directory
</code>
Answer: The error message you're seeing is actually another symptom, not the root cause of the issue. There's a good explanation/discussion in this Apache Jira bug report:
https://issues.apache.org/jira/browse/MESOS-1837
Basically, your container is crashing for one reason or another and the /proc/pid#/ directory is getting cleared without Mesos being aware, so it throws the error message you found when it goes to check that /proc directory.
Try setting your allocated CPU higher in the JSON file describing your task.


Javascript stopped working after moving html to parent site

Question: I have seen a few questions that are similar but they didn't seem to have the answers that helped me. So, I come to you with hat in hand.
My site was working perfectly under a test URL and then when I moved all the files to the parent url, no code changed, just a simple copy and paste with all the links updating themselves, all of a sudden I get no javascript plugins working.
Everything else came over just fine, it is only the javascript that is giving me problems. When I look at the site through firebug, it doesn't seem to show any problems. I am absolutely pulling my hair out on this one.
This is the setup I have right now:

    <link type="text/css" href="../style.css" rel="stylesheet" media="all" />
    <link href='http://fonts.googleapis.com/css?family=Open+Sans:400,700' rel='stylesheet' type='text/css' />

    <link rel="stylesheet" type="text/css" href="../_layout/js/imagebox/imagebox.css" />

    <link rel="stylesheet" type="text/css" href="../_layout/js/flexslider/flexslider.css" />

    <link rel="stylesheet" type="text/css" href="../_layout/js/carousel/carousel.css" />

    <link rel="stylesheet" type="text/css" href="../_layout/js/validity/css.validity.css" />

Again, everything worked before the move, so I don't think that there is anything wrong with the individual javascript files themselves. If you don't see anything wrong here, then it might be the DOM? Full disclosure, I know nothing about the DOM, if anybody has a good source for learning about configuring the DOM for beginners, please do let me know.
I appreciate any time that any of you can give to this problem.
Comment: Are the JS files in the same place?
Comment: `../` references a level above the folder the file exists in. Have you tried changing the paths?
Comment: Can you give us a link?
Comment: When you say that you've pulled up the JS files successfully, what do you mean? How are you testing that?
Comment: The JS files were moved with the rest of the site but they are being called correctly. I have pulled them up successfully.
Comment: j08691,  yes, I have tried changing paths by taking out the ../ and it made no difference.
Comment: I'm sorry, I can't. The client has said that they don't want their website to be found on a support site. I am very sorry about that.
Comment: The javascript files show up in tact under the URL path when I click on them through the page source in the browser.
Answer: I suspect the problem is caused by the fact that you are using relative URLs, and perhaps your new web server setup has changed.  Have you tried going to the NET tab in Firebug to see if all of the JavaScript files load successfully (make sure you arent getting an HTTP 404 error)?
Comment: Thank you for your answer, I completely forgot to mention that. The URLs work perfectly and yes I am able to load the javascript files successfully.
Comment: If you go to the console tab of firebug and re-load the page, do you see any JavaScript errors at all?
Comment: The console typically will show messages, although they are not necessarily errors, they could be informational messages or log messages as well.  Usually a JavaScript error will show up in red.  When you go to the Script tab in firebug and click the drop-down (it may say 'inline'), do all of your files appear in the list?
Comment: When I reload the page it still doesn't show any errors, which now that I think of it is rather strange because there is always some sort of error, even if it is a small one, right?
Comment: Aha! No, I didn't realize it before but the flexslider javascript files don't show up in the list.
Comment: Okay, that is probably the issue, but now we need to figure out why it isnt loading properly.  Do you see the rest of the js files in that list?  Can you verify that the JavaScript file was successfully copied to the _layout\js\flexslider\ folder?
Comment: Yes the flexslider file is definitely there as I can see it and bring it up through the source code in the browser. I went to the console and it tells me that there is an error in this line -                         "/>':"  "+b+" ",a.controlNavScaffold.append(""+g+" "),b++;a.controlsContainer?d(a.controlsContainer).append(a.controlNavScaffold):a.append(a.controlNavScaffold);f.controlNav.set();f.controlNav.active();a.controlNavScaffold.delegate("
Comment: It says that it is missing a colon but I don't see where and it isn't clear where. (oh, this is in the flexslider js file)
Comment: It also says that flexslider is not a function on this line, nextText: "Next" //String: Set the text for the "next" directionNav item                    in the plugins.js file.
Comment: let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/50796/discussion-between-mdiesel-and-aceofhearts)
Answer: Make sure you're pointing to correct location. If you were in /web_root/test/ and you just moved everything to /web_root/, you'll need to update the src attribute of all your imported scripts to reflect this change. Assuming you just moved your files up one directory, you can simply get rid of the first ../. For example,

Would be

It would be helpful to know the directory structure for your site, specifically the directory you were in while testing everything, and the directory you moved everything into.
Comment: Thank you, I forgot to mention that I have tested that the javascript files are being called correctly. I also tried getting rid of the ../ at the beginning and this made no change whatsoever.
Comment: Like I said, it would be immensely helpful to know exactly what you changed, and where, including the directory structure.
Comment: Well, the only thing that changed was moving everything as it was under a sub URL under the parent URL and the links all changed to reflect that. Other than that everything stayed exactly the same. The folder that the javascript files were under is still there, just under the parent URL instead of a sub URL.
Comment: I'm still not convinced the links are correct. Do the JS files show up in Firebug or Chrome Developer Tools?
Comment: I was just made aware that one of my scrips is not showing up on Firebug. The flexslider javascript file doesn't show up.


Can't understand this behaviour

Question: I have a simple notes app and delete isn't working properly, even though state is correctly updated.
The state is being updated correctly as I can see on the console.
But all the notes including and after that note that I click delete on are getting deleted on the DOM for some reason.
For example if I have 3 notes ["hello","hi","hey"], if I delete the second note("hi"), the state shows the correct notes ["hello","hey"] but  both "hi" and "hey" are deleted on the page not just "hi" like it was supposed to.
I can't understand where I've gone wrong, so I'd like to correct it.
App.js:
handleDelete = (note_id) => {
    const id = 'display-' + note_id;
    console.log(id);
    document.getElementById(id).style.display = 'none';//remove element

    //delete the note and update state
    const newNotesList = this.state.notesList;
    newNotesList.splice(note_id,1);
    this.setState({
      notesList : newNotesList
    })
    console.log(this.state.notesList)
  }

Display.js:
render(){
    const notesList = this.props.notesList;
    const displayNotes = notesList.map( (note,note_id) =>

      {/*some code*/}
      <button type="button" className="delete-button"
     onClick = { () => this.props.handleDelete(note_id) } > Delete </button>

 );

    return
{displayNotes}
;
  }

Comment: why, oh, why you have two ways of removing notes from the view and you apply both at the same time?
Answer: do like this
// don't mutation object

// App.js
handleDelete = (note_id) => {
    //delete the note and update state
    const newNotesList = this.state.notesList.filter((item, index)=> index !== note_id)
this.setState({
  notesList : newNotesList
})
  }

// Display.js
render(){
    const notesList = this.props.notesList;
    const displayNotes = notesList.map( (note,note_id) =>

      {/*some code*/}
      <button type="button" className="delete-button"
     onClick = { () => this.props.handleDelete(note_id) } > Delete </button>

 );

    return
{displayNotes}
;
  }

==== here is the reason ========
at first the state.note is ["hello","hi","hey"], in the function of handleDelete you delete "hi" and make the id of dispaly-1's display become to hidden, so when react render the state.note === ["hello","hey"] the element of "hey"'s id become dispaly-1 so "hey" will be hidden. you will only see "hello"
handleDelete = (note_id) => {
    // this.state.notesList === ["hello","hi","hey"]
    const id = 'display-' + note_id;
    console.log(id);
    // the problem is u hidden the next element

    // 1. newNotesList.splice(note_id,1);
    // 2. document.getElementById(id).style.display = 'none'
    // two methods choose one or you will hidden two elements

    document.getElementById(id).style.display = 'none';//remove element

    //delete the note and update state
    const newNotesList = this.state.notesList;

    newNotesList.splice(note_id,1);
    this.setState({
      notesList : newNotesList
    })
    console.log(this.state.notesList)

   // for example
   // the `this.state.notesList` new is ["hello","hey"]
  }

notesList.map( (note,note_id) =>
      // `note` is ["hello","hey"]  , u don't need hidden the `2rd` element
      //you have been delete 'hi' ` the id of `display-1`'s display ==='hidden'
     // now "hey"'s id is `display-1`

      {/*some code*/}
      <button type="button" className="delete-button"
     onClick = { () => this.props.handleDelete(note_id) } > Delete </button>

 );
``

Comment: Avoiding mutation is certainly good, but why exactly would this solve the problem of multiple notes apparently getting deleted, do you know?
Comment: actually the problem is he delete 2rd element hidden `3rd` element @CertainPerformance
Comment: Curious, I thought that since no `key` was provided, the rendered ``s (and their contents) would be re-created on every render call - so the hiding of `'display-' + note_id` wouldn't do anything, since the element would not exist on the next render. Wonder what's up.
Comment: actually the point is the element didn't bind with the `id` name, but the `id` name connected with index. as we know, `index` is "eval" ; )  so every render call, it will effect the `index` element's style   @CertainPerformance

Can't understand this behaviour


Infield Fly Rule

Question: During the 2012 NL Wild Card game between the St. Louis Cardinals and the Atlanta Braves, a fly ball was hit by Atlanta. St. Louis let the ball drop due to miscommunication, but an umpire called "infield fly rule."
The ball clearly lands deep in the outfield, near the red circle below.

Do the umpires have grounds to call "infield fly rule" when a fly ball will clearly land that deep in the outfield? What are the rules regarding the "infield fly rule?"
EDIT: Interesting post by the MLB (now removed)
Answer: It's entirely up to the umpire's discretion:

An INFIELD FLY is a fair fly ball (not including a line drive nor an
  attempted bunt) which can be caught by an infielder with ordinary
  effort, when first and second, or first, second and third bases are
  occupied, before two are out. The pitcher, catcher and any outfielder
  who stations himself in the infield on the play shall be considered
  infielders for the purpose of this rule. When it seems apparent that a
  batted ball will be an Infield Fly, the umpire shall immediately
  declare "Infield Fly" for the benefit of the runners. If the ball is
  near the baselines, the umpire shall declare "Infield Fly, if Fair."
  The ball is alive and runners may advance at the risk of the ball
  being caught, or retouch and advance after the ball is touched, the
  same as on any fly ball. If the hit becomes a foul ball, it is treated
  the same as any foul. If a declared Infield Fly is allowed to fall
  untouched to the ground, and bounces foul before passing first or
  third base, it is a foul ball. If a declared Infield Fly falls
  untouched to the ground outside the baseline, and bounces fair before
  passing first or third base, it is an Infield Fly. -- MLB

What constitutes ordinary effort? How do you define that as anything but an arbitrary judgement for the umpire?
The point of the infield fly rule is to protect runners from a forced double play if a infielder purposefully drops an easy pop-up. This is why the rule is written so loosely and is generally left up to the umpire to define as to what "can be caught by an infielder with ordinary effort."
Technically, any ball hit high enough could be an infield fly even if it goes as far as the warning track. However, in this case, the umpire simply made a bad call. Balls that fly that deep are rarely called an infield fly.
Answer: Well, here's is what a comment in the official rules (pdf) says:

On the infield fly rule the umpire is to rule whether the ball could
  ordinarily have been handled by an infielder—not by some arbitrary
  limitation such as the grass, or the base lines. The umpire must rule
  also that a ball is an infield fly, even if handled by an outfielder,
  if, in the umpire's judgment, the ball could have been as easily
  handled by an infielder. The infield fly is in no sense to be
  considered an appeal play. The umpire's judgment must govern, and the
  decision should be made immediately.

While I've never seen a hit like this declared an infield fly, any shortstop would've caught this ball without much effort under normal circumstances.
EDIT:  Here is a video of another infield fly that is similar.
EDIT AGAIN:  This analysis from Harold Reynolds, with other examples, states why it was the correct call.
Comment: I agree. However, with the miscommunication...made a big difference!
Answer:
(source: baseballpilgrimages.com)
I've never seen a infield fly called that deep before.  I've noticed a lot of Turfed baseball stadiums have a white line just beyond the base path.  I was always under the impression that this was the cut off for an infield fly.  The rule stats that "ordinarily have been handled by an infielder" arguments could be made that this wasn't an ordinary play for the SS.  Also  "decision should be made immediately." which it wasn't.  The argument that he didn't know until that moment makes me feel that it should not have been called at all.

Infield Fly Rule


Switching between camera not working in android

Question: My Video Recoding app record video and it is working from back camera now i want to use front camera also to record video. So users can switch between camera. User can select by which camera he want to record the video.
Initially when my activity loaded back camera is open and preview is start, but when i click on switch camera button then it gives exception that java.lang.RuntimeException: Fail to connect to camera service.
Below is my code to switch camera :
private void initRecorder(Surface surface) throws IOException {

   try{
        if (mCamera != null) {
            mCamera.lock();
            mCamera.unlock();
            mCamera.stopPreview();
            mCamera.setPreviewCallback(null);
            mCamera.release();
            mCamera = null;
            mHolder.addCallback(null);
            mHolder = null;
        }

        mHolder = mSurfaceView.getHolder();
        mHolder.addCallback(this);
        mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);

        if(cameraToOpen==1){
            mCamera = Camera.open(Camera.CameraInfo.CAMERA_FACING_BACK);

        } else if(cameraToOpen==2){
            mCamera = Camera.open(Camera.CameraInfo.CAMERA_FACING_FRONT);
        }

        mCamera.lock();
        mCamera.setDisplayOrientation(90);
        mCamera.setPreviewDisplay(mHolder);
        mCamera.startPreview();
        mCamera.unlock();

    }catch(Exception e){

      }
 }

When user click on switch camera button then mCamera = Camera.open(Camera.CameraInfo.CAMERA_FACING_FRONT);
gives exception.
How to work with both cameras.
Edited:
When i open front camera when activity start then it is working fine. SO i can open both camera and record from both only when initial that camera is selected. But When i switch between camera then it gives exception unable to connect to camera service.
Comment: Try this approach: http://stackoverflow.com/a/17117023/2944070
Comment: Still not found solution to my problem.
Comment: Have u tried the method mentioned there? I was getting same error, and it fixed the issue for me.
Comment: Yes i have try this but still got error unable to start preview when call mCamera.startPreview();
Answer: You can do the switch between cameras as stated below :
Button otherCamera = (Button) findViewById(R.id.OtherCamera);

otherCamera.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View v) {
if (inPreview) {
    camera.stopPreview();
}

//NB: if you don't release the current camera before switching, you app will crash
camera.release();

//swap the id of the camera to be used
if(currentCameraId == Camera.CameraInfo.CAMERA_FACING_BACK){
    currentCameraId = Camera.CameraInfo.CAMERA_FACING_FRONT;
}
else {
    currentCameraId = Camera.CameraInfo.CAMERA_FACING_BACK;
}
camera = Camera.open(currentCameraId);

setCameraDisplayOrientation(CameraActivity.this, currentCameraId, camera);
try {
    camera.setPreviewDisplay(previewHolder);
} catch (IOException e) {
    e.printStackTrace();
}
camera.startPreview();
}

For recording video while switching the camera you can follow more here switching between cameras using mediarecorder.