Pysnmp openServerMode with IPv6

Question: I try to start a SNMP Agent with pysnmp.
for IPv4 and IPv6 bind, it works pretty fine with localhost ('127.0.0.1' and '::1')
but when I try to use other IPv6 IP which I fetched from interface, it failed due to
[vagrant@test SOURCES]$ sudo python snmp_agent.py enp0s8
Traceback (most recent call last):
  File "snmp_agent.py", line 172, in <module>
    master_agent_startup(ifname=sys.argv[1])
  File "snmp_agent.py", line 101, in master_agent_startup
    (get_ipv6_address(interface_name), SNMP_AGENT_PORT))
  File "/usr/lib/python2.7/site-packages/pysnmp/carrier/asyncore/dgram/base.py", line 50, in openServerMode
    raise error.CarrierError('bind() for %s failed: %s' % (iface, sys.exc_info()[1],))
pysnmp.carrier.error.CarrierError: bind() for ('fe80::a00:27ff:fe9e:9c16', 8001) failed: [Errno 22] Invalid argument

This is the output from the interface 'enp0s8':
[vagrant@test SOURCES]$ ifconfig enp0s8
enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.20.26  netmask 255.255.255.0  broadcast 172.20.20.255
        inet6 fe80::a00:27ff:fe9e:9c16  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:9e:9c:16  txqueuelen 1000  (Ethernet)
        RX packets 874053  bytes 115842841 (110.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 862314  bytes 114652475 (109.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This is the code piece I used for IPv6 bind:
def get_ipv6_address(ifname):
    return netifaces.ifaddresses(ifname)[netifaces.AF_INET6][0]['addr'].split('%')[0]

config.addSocketTransport(snmpEngine, udp.domainName,
                          udp.UdpTransport().openServerMode(
                              (get_ipv4_address(interface_name), SNMP_AGENT_PORT))
                          )
config.addSocketTransport(snmpEngine, udp6.domainName,
                          udp6.Udp6SocketTransport().openServerMode(
                              (get_ipv6_address(interface_name), SNMP_AGENT_PORT))
                          )

From pysnmp sample, it seems the parameter inside "openServerMode()" is just a tuple of IP and port.
And from output error I suppose there is no error for given IP and port.
so why it failed due to Invalid argument?
Could you @Ilya Etingof or some other pysnmp expert help me with it?
Thanks.

UPDATE: I try to bind it with given suggestion, but is still doesn't work.
  the bind command was run from a new installed CentOS. but it still failed:
      [root@test ~]# ifconfig

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.20.4  netmask 255.255.255.0  broadcast 10.10.20.255
        inet6 fe80::f816:3eff:fee1:5475  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:e1:54:75  txqueuelen 1000  (Ethernet)
        RX packets 12242  bytes 962552 (939.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12196  bytes 957826 (935.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@test ~]# python
Python 2.7.5 (default, Oct 11 2015, 17:47:16)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0)
>>> addr_and_port = ('fe80::f816:3eff:fedb:ba4f', 8001)
>>> s.bind(addr_and_port)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 22] Invalid argument
>>>
[1]+  Stopped                 python
[root@test ~]# netstat -anp | grep 8001
[root@test ~]#

One more update:
  I suppose the bind is failed due to my environment has some issue with IPv6 configuration. As I'm only able to get one IPv4 address by using "socket.getaddrinfo()" method.

Br,
-Dapeng Jiao
Answer: You could get this error if attempted to bind the same socket more than once. But I can't see that is the case in your code.
That .openServerMode() method does no magic -- it just calls .bind() on socket object. For inspiration, does this work in at your Python prompt?
from pysnmp.carrier.asyncore.dgram import udp6

addr_and_port = ('fe80::a00:27ff:fe9e:9c16', 8001)
udp6.Udp6SocketTransport().openServerMode(addr_and_port)

or even:
import socket

s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0)
addr_and_port = ('fe80::a00:27ff:fe9e:9c16', 8001)
s.bind(addr_and_port)

My hope is that tests like these may help you figuring out the problem...
Comment: Thx for ur quick answer. Both of this two try give the same error.

Not sure how to paste all the error output here. So just give some lines:
'Traceback (most recent call last):
  File "", line 1, in
  File "/usr/lib/python2.7/site-packages/pysnmp/carrier/asyncore/dgram/base.py", line 50, in openServerMode
    raise error.CarrierError('bind() for %s failed: %s' % (iface, sys.exc_info()[1],))
'
Comment: '>>> s.bind(addr_and_port)
Traceback (most recent call last):
  File "", line 1, in
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 22] Invalid argument'
Answer: When you use an IPv6 link-local address, you must always use the scope along with it. The link-local address is not valid without its scope, thus you will receive an Invalid argument error.
For example, instead of using fe80::a00:27ff:fe9e:9c16 you must use fe80::a00:27ff:fe9e:9c16%enp0s8.

Pysnmp openServerMode with IPv6


'Edit' tab does not appear on user's account page?

Question: On my site when a user navigates to their account page there is no 'Edit' button for them to be able to edit their details. From the documentation online it seems this should appear automatically.
Is there any way this could have been disabled by accident or are there any modules that could have interfered with it?
A user is able to edit their profile if they add /edit to the link to their profile page, so it is not that they don't have the 'edit' permission - they definitely do and are able to edit their details successfully if they go to this link.
Answer: Oceanescence, some times people have disabled or moved the tabs block. If you are using a custom theme try switching to Bartik and see if it pops up.
If not try digging into the permissions. Just log in as Admin 1 and see if the tabs show up.
Comment: Glad it worked. Have fun Druaplin!
Comment: Thank you, the tab block was indeed not enabled


Is the cumulative incidence function just an inverted Kaplan-Meier survival curve?

Question: Is CIF just an inverted Kaplan-Meier survival curve, and nothing more?
Comment: Check out this paper here for some useful reading on the topic: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2410013/
Answer: If you're talking about the cumulative incidence function that arises from a Kaplan-Meier estimator, then it's just $1 - S(t)$ where $t$ is time. In R:
<code>library(survival)
fit <- survfit(Surv(time, status) ~ x, data = aml)
plot(fit) # the standard survival curve
plot(fit, fun="event") # the cumulative incidence curve
</code>
But, CIF is more commonly used in other contexts, for example in competing risk analysis using cause-specific Cox regression. There, the CIF is not necessarily equal to $1-S(t)$ because it takes into account the competing risk, estimating the risk of an event happening given survival up to time $t$ and also that a competing risk hasn't occurred up to time $t$.
Comment: If you're referring to the no-competing risk scenario, then I'd just call it cumulative risk.
Comment: GREAT THANKS! COULD I CALL (IN THE ARTICLE, IN WHICH I ANALYZE GROUP OF PATIENTS) CIF AS A "CUMULATIVE FREQUENCY OF RISK" (ASSOCIATED WITH SOME FACTOR) OR SIMPLER: "CUMULATIVE RISK"?


Why doesn't estimating Shannon entropy with a histogram converge to its true value?

Question: I'm following the third recipe of this answer to estimate the Shannon entropy of my samples using histograms.  My expectation was, increasing the sample size should lead to a better estimation of the true entropy. To test that, I sampled from a Gaussian distribution $N(0, \sigma^2)$ with known entropy $H(X) = 0.5[1 + \log(2\pi \sigma^2)]$:
import numpy as np
from scipy.stats import entropy # by default in natural log
import matplotlib.pyplot as plt

sigma = 2
H0 = 0.5*(1 + np.log(2*np.pi*sigma**2)) # theoretical value in nats

Hs = []
ns = np.logspace(2,6,5, dtype=int)
for n in ns:
    X = np.random.normal(0, sigma, size=n)
    nbins = int(n/20)  # <--- This is causes divergence, read UPDATE 2
    hist, bin_edges = np.histogram(X, nbins, density=True)
    Hs.append(entropy(hist)) # will be automatically normalized to sum to 1

plt.plot(ns, Hs, '-x', label='estimation')
plt.hlines(H0, ns[0], ns[-1], linestyle='--', color='k', label='exact H')
plt.legend()
plt.yticks(np.arange(11))
plt.xscale('log')
plt.xlabel('num samples')
plt.ylabel('sample entropy')

To my surprise, they diverge:

Am I missing something very simple?
I found a relevant question that is concerned about the case where the probability distribution is either "peaked" or "flat". The upshot of the most voted answer is that one has to normalize the entropy by $\log n$ where $n$ is the number of samples. Although I find this normalization sensible, I still observe an offset between the theoretical value and the estimation:
plt.plot(ns, np.array(Hs)/np.log(ns), '-x', label='normalized estimation') # normalized

What causes this discrepancy?
As @sextus-empiricus noted below, scaling the number bins with the sample size leads to a coarse density estimation. In fact, fixing the number of bins, resolves the divergence. However, no matter what number of bins one chooses, there's still some offset between the estimated (normalized or non-normalized) entropy and the theoretical value, as you can see below:

What is the reason behind this gap? The curve displacement by nbins clearly suggest that there's a relationship between the two, but I found the reason elusive.
Answer: Here is an image that I made with the R-code below. The computation is with n-bins = 50.
There is in this simulation also an offset but smaller and with more bins I get a smaller offset. This might be because you approach the values of a discretized normal distribution and not the values of a continuous normal distribution. So one should not expect the same.
For small sample sizes there is lower entropy. This might be counter intuitive because one would imagine a small sample to be more variable and more chaotic. But, the state described by the histogram of a small sample is a very particular state and has more order than the average of many different samples. The increase of the sample size can be seen as blending multiple states and that creates an average that blends out to a smooth histogram, a histogram that looks like a more homogeneous sample.

set.seed(1)
n_range = 10^seq(2,6,0.2)
entropy_range = c()
n_bins = 50

for (n in n_range) {
  x = rnorm(n, mean = 0, sd = 2)
  h = hist(x, breaks = n_bins, freq = 0)
  p = h$density[h$density > 0]
  bin_size = h$mids[2]-h$mids[1]
  entropy = -1*sum(log(p)*p*bin_size)
  entropy_range = c(entropy_range, entropy)
}

plot(x= n_range, y= entropy_range, log = "x")
lines(x= c(1,10^6), y= c(1,1)*(0.5*log(8*pi)+0.5), lty = 2)

A tricky part is that the computation of entropy is different for continuous distributions
$$\int f(x) \log \left( f(x) \right) \text{d} x \approx \sum_{k = 1}^{n_{bins}} f_k \log \left( f_k \right) \cdot \text{bin_size}$$
When you just apply the bin probabilities $p_k = f_k \cdot \text{bin_size}$ and use the formula for discrete distributions then you get a different values.
$$\sum_{k = 1}^{n_{bins}} f_k \cdot \log \left( f_k \right) \cdot \text{bin_size} \neq \sum_{k = 1}^{n_{bins}} f_k \cdot \text{bin_size} \cdot  \log \left( f_k \cdot \text{bin_size} \right) = \sum_{k = 1}^{n_{bins}} p_k \cdot  \log \left( p_k \right)$$
You will be wrong with a factor $\log(\text{bin_size})$ and that explains the different curve displacements shown in your second edit.
In short: entropy is the expectation of the logarithm of the probability mass function or the probability density function $E[\log p(X)]$ and $E[\log f(X)]$. Problems arrise when we mix those formulas and functions.
Comment: It is funny that in my previous version of the post I made the mistake discussed at the end, but because I used 20 bins which made binsizes of 1, it didn't make an influence. The weird discrete step that I observed there was due to the bin-sizes changing when the sample size increases.
Comment: @arash and the divergence that you had earlier was due to bin_size scaling with sample size. With the correct formula for entropy you can let the n_bins increase with n.
Comment: That's right. Fixing the number of bins solves the problem of divergence. Yet, the offset persist. (I'll update my question to include this).
Comment: Thanks for posting this! I could reproduce your result once I set `nbins=20`, but not for other values of `nbins`. Please see my last figure, I just updated it. At this point, I'm not sure how `nbins` effects the gap between theoretical and estimated entropy values.
Comment: Thank you very much for this update! Sure enough, adding $\log (\text{bin_size})$ fixes the gap.


How to find the sides of a triangle if all angles of the triangle are known

Question: In a triangle, if all angles are known, how is it possible to find all the 3 sides, using just this much information?
Comment: Knowing the three angles only defines the triangle up to scale.  If you declare one side to be $1$, say, you can use the Law of Sines to get the other two.
Comment: Every equilateral triangle has $60^\circ$ angles. But the side lengths can be any length whatsoever.
Answer: You can't. When you dilate a triangle, its angles stay the same. Thus, given all three angles, the sides can be anything as long as they stay in proportion.
However, we can find the exact proportions of the sides given the angles.
For this, the law of sines is useful.
$$\frac{A}{\sin \alpha}=\frac{B}{\sin \beta}=\frac{C}{\sin \gamma}$$
Where $A$ is the length of the side opposite angle $\alpha$, $B$ is the length of the side opposite angle $\beta$, and $C$ is the length of the side opposite angle $\gamma$.


How can I extend my male chest voice range higher when singing piano/pianissimo?

Question: I sing tenor in an SATB choir and currently I can reach F#4 when singing loudly at the top of my chest voice - sometimes G4 depending on the 'shape' of the note. For all the pieces we've done this is fine but the ecclesiastical music we sing often has quiet, high phrases for the tenors and I have to switch to falsetto around D4-E4.
Can anyone tell me how I can specifically improve my range when singing quietly? Singing a little run around F#/G4 pp is really nerve-wracking!
Comment: Possibly easier to extend falsetto downwards?
Comment: It would be good for you to stop belting high notes with your chest voice and learn how to used your mixed voice.
Answer: Singing in a soft dynamic in our extreme vocal ranges is a combination of good vocal technique and breath support.
An example exercise for improving range could be starting with a 5-4-3-2-1 descending pattern on the middle range and working upwards.  The important part is to remember how relaxed, easy, and open the middle range is and try to transfer as much of that feeling as you can as you ascend.
There will always be tension in the higher range so be mindful of where that tension is and use it to your advantage.  A quick google search can tell you about laryngeal position while singing higher.  Most vocalists advocate for a lower larynx position.
Once the technique is there then focus on using your breath to support the dynamic.  Practice soft singing in the middle range.  Depending on the sound you want will depending on the resonance you are going for.  Raspy, strident, dark and so on.  I would focus on a warm and resonant sound as singing in that manner will help the voice to stay healthy.


how do chmod a directory in drupal

Question: I want to enable pictures for users in drupal http://drupal.org/node/22271
and I am not sure how to chmod the directory so that users can save their pictures in it.
Thank you for your help in advance.
Answer: You want to go through this guide.  It discusses in depth the appropriate filesystem permission settings you should have on every part of the Drupal installation (public files, private files, etc.)  If you follow that guide to the letter your filesystem problems will sort itself out, and you'll have a more secure Drupal installation.  Let me know if you have any questions.
Answer: Go to the parent of the directory where you want to save the pictures or create your own directory with mkdir and cd command.
cd to change the directory .
mkdir to make a directory.
Then do sudo chmod 755 directory-name

how do chmod a directory in drupal


What (discrete) distribution might fit best here?

Question: I have some count data I need to fit to a distribution.  I have tried several distributions but none of them start out steep enough.
Here is the fit of a geometric distribution:

My question is: Which distribution should I fit here?  (And (how) is that possible in R?)
In case you want to play:
Due to its size I put the data on pastebin.  You can read it into R using the pastebin package:
## library("devtools")
## devtools::install_github("hrbrmstr/pastebin")
library("pastebin")
ttt <- get_paste("hkuUg0m3")$text
ttt <- as.numeric(strsplit(ttt, "\r\n")[[1]])

I fit the geometric distribution using the fitdistrplus package:
library("fitdistrplus")
fit <- fitdist(ttt, distr = "geom")

And this is what I use for visualization:
viz_fitdist(fit, distr = "geom")

using this function:
viz_fitdist <- function(fit, distr="geom", annotation_colour="#326a97")
{
  library("ggplot2")
  library("grid")
  library("gridExtra")

  data <- data.frame(dat=fit$data)

  plotdata <- as.data.frame(table(data))
  plotdata <- as.data.frame(table(data)/nrow(data))
  plotdata$data <- as.numeric(as.character(plotdata$data))
  p <- plotdata %>% ggplot(aes(x=data, y=Freq)) + geom_bar(stat="identity")

  if (distr=="geom") {
    xsupport <- 1:max(data$dat)
    } else {
      xsupport <- unique(data$dat)
    }
  if (distr!="none") {
    dfun <- paste0("d", distr)
    y <- do.call(dfun, c(list(x = xsupport), as.list(fit$estimate)))
  }

  if (distr!="none") {
    fitdata <- data.frame(x=xsupport, y=y)

    p <- p +
      geom_point(aes(x=x, y=y), data=fitdata, colour=annotation_colour)

    p <- p + geom_line(aes(x=x, y=y), data=fitdata, colour=annotation_colour, size=1)

    p <- p +
      annotation_custom(grob=textGrob(sprintf("Parameter: %.2f", fit$estimate),
                                      x=1-0.07,
                                      y=1-0.07,
                                      just=c("right", "top")))
  }

  p + scale_x_continuous("Data", breaks=xsupport) + ylab("Frequency")
}

EDIT: more distributions
Here is the fit of a negative binomial:
ttt %>%
  fitdist(distr = "nbinom") %>%
  viz_fitdist(distr = "nbinom")

And a zero inflated poisson:
library("extraDistr")
ttt %>%
  fitdist(distr = "zip", start = list(lambda = 1.6, pi = 0.5), lower = c(-Inf, 0), upper = c(Inf, 1)) %>%
  viz_fitdist(distr = "zip")

And a Weibull:
ttt %>%
  fitdist(distr = "weibull") %>%
  viz_fitdist(distr = "weibull")

Comment: Welcome to [stats.se]!  Please take a moment to view our [tour].  There are several possible answers to this.  How closely does the model need to match your data?  What are you using as your measure of "fit?"
Comment: The point of fitting a distribution to your data is obscure.  How exactly will you use it?  What do these data represent?
Comment: One purpose of the fit is to have a means to quantify 'unexpectedness' for high counts.  For that I guess the model does not need to be very close.  My current measure of 'fit' are my eyes...
Comment: I want to identify unexpectedly high counts in the spirit of [alphaOutlier](https://cran.r-project.org/package=alphaOutlier) for which I need a probability model.
Answer: Let me open by saying that searching for and identifying a model that appears to describe the data is not where I would normally begin my modelling; it begins with the information whuber and I were seeking on how the model will be used, with what is being measured - and further, to consider any associated theoretical or expert practical understanding of its properties, the performance of models on similar data sets, and so on. Considerations like these are essential to the choice of an appropriate model.
Nevertheless, as requested, I will attempt to describe some steps one could take in examining data like this and seeing if it could suggest a model.
If you're going to use the resulting distributional model for inference (hypothesis tests, or confidence intervals, or prediction intervals for example), there's a problem that you chose the model based on your data so the nominal properties of the inference will no longer hold. (If you really can't use the characteristics of the variable and subject area knowledge, having no other basis than the sample itself, then it's better to pull off a random subset and use that for model identification.)
Given the heavy tail, one option is to look at the way the log of the empirical survivor function relates to $x$. My first plots were to plot $\log(\hat{S}(x))$ against $\log(x)$ (not shown; linear at the top but then somewhat curved near the middle) and then against $\sqrt x$. The appearance of these then suggested to me that a fair first model might be a zeta distribution. However, they can also suggest a number of other possibilities (such as perhaps directly modelling the seeming linear relationship in the first plot below; it might provide a somewhat better fit, at the expense of being less familiar)
So I then looked at the empirical pmf, plotting $\log(\hat{p}(x))$ against $\log{x}$ as a check on the potential for fitting a zeta function.

That indeed suggests a fairly straight line relationship, and it looks like a parameter in the ballpark of 2 or perhaps a bit more would be a reasonable first model -- note that the most precisely estimated proportions are the first few, and fitting those accurately will matter more for a lot of applications -- but not all. However, it's not a perfect fit - with a large sample size like you have, I believe you would easily reject a pure zeta model (or indeed any other simple model).

(NB the parameter value here has not been optimized; it's just a reasonable guess at a suitable value)
So if I had to suggest a simple model on the basis of only looking at the data, the zeta distribution would be an obvious candidate; if an initial guess at the parameter value is needed, I'd start at 2.
On fitting power laws, such as the zeta distribution, also see Cosma Shalizi's essay So you think you have a power law? which debunks some of the enthusiasm for power laws (a lot of things will look more or less power-law like) and the paper it discusses, which has a lot of useful information.
Comment: Thanks for this answer!  I appreciate your comment about fitting meaningful distributions.  But even more I appreciate you answering my question of which simple distribution might fit the given data.  And most of all, I appreciate your very educational explanations on how you got there.  So, +3 would be appropriate.  Thanks!
Answer: Have you tried a Negative Binomial distribution or a zero-inflated Poisson?
 These distributions are often used to take care of overdispersion.
You have been suggested below a Weibull distribution, which may fit the data but it is not appropriate for count data.  A Weibull is a distribution for continuous measurements, and not for discrete data.  Negative Binomial distributions are routinely used for RNAseq and Microbiome data, among other types of data. So are the zero-inflated Poisson distribution. They are standard tools and are easily implemented in R packages  See Cameron-Trivedi (2013, Regression Analysis of Count Data). Now, there are also Discrete Weibull distributions, but they are not so much used. See the R package "DiscreteWeibull" and related manuscripts.
The negative binomial fit above is similar to the Weibull and it is more appropriate, since it takes into account the nature of the data.
If you really wanna be flexible, mixtures of Poisson-Gamma may help you to capture overdispersion in a Bayesian fit. They are of course not trivial. But if the question is about fitting the data, then all the options should be proposed.
Now, your data show that histogram because there are probably bumps at the right end of the x-axis. No parametric model can capture those bumps correctly. That's why I was suggesting mixtures of parametric models. And in such case, the Bayesian fit is probably the way to go.
The right model depends of course on the objective of your inference, which is not clear from your question.
Comment: A Weibull distribution (which is continuous) makes little sense for *counted* data!
Comment: @Tavrock Weibull is continuous. Data is (mostly small) positive integers (>90% of values are 1,2 or 3). Makes no sense to use Weibull for that. You could perhaps make an argument for modeling the top 2 or 3 percent of the distribution as continuous but that seems unnecessary when so much of it is very small integers.
Comment: Many of those categories have probably zero frequency. Except a few. Again, these type of situations/needs for model fit are standard in the analysis of sequencing data. But we haven't been told what these data represent.
Comment: @Andreas It's quite easy to find a discrete distribution that fits this data pretty well (a couple of quick plots was sufficient), but that's going about the problem backwards. You need to talk about what you're modelling (what did you count?\*) and what the model will be used for. $\quad$ \* while I could probably make a fair guess at the general kind of thing you're counting, we shouldn't guess
Comment: A well chosen Weibul could be made to fit this as well.
Comment: I have tried negative binomial and zero-inflated poisson. I have not done a mixture of Poisson-Gamma distributions in a Bayesian fit and I would not know how to do that.
Comment: @Tavrock I have not tried a Weibull either.
Comment: @Tawrock I have now done a first Weibull fit.  See the edit.  While not bad, also that seems not 'steep' enough to my eyes.
Comment: @whuber, if the OP only had seven or so categories, I would definitely agree with you, but if the OP really does have 56 distinct categories, then I feel it *can* be appropriate to try a continuous distribution.
Comment: @Glen_b Good to hear that this is easy.  I am not experienced enough to achieve this with a 'couple of quick plots' (see my edits).  I want to use that model to identify unexpectedly high counts (e.g. how likely are the numbers I see for 8 and 9?)
Comment: I'm not sure I understand -- if you fit a model, but some of your counts are substantially higher than the model suggests, it would be the model that's wrong.
Comment: Sure.  I just trust the low numbers generally more.  That is why I am unhappy about all my fits so far.
Comment: @MG Thanks for the only answer so far, which is informative. I am not really interested in the bumps at the right, but I am more concerned about the first counts.  But none of my fits (including your suggestions negative binomial and zero inflated poisson) seem to be able to capture the steep slope  in the low numbers (or the enrichment of '1').  Can you comment?
Comment: @Glen_b Thanks for your comments.  Could you provide your proposed fits as an answer?  If you think the problem is not stated well, feel free to comment in which situation the fits would be ill-suited.
Comment: Andreas, if you consider a transformation of your data so that ttt=ttt-1, then a zip would fit better. That's not probably what you want, in general.  I am looking into packages for fitting mixtures of Poisson or Negative Binomial. That might help, but I cannot do it right away.


Understanding database structure for auto-migration

Question: I am trying to come up with a good way to get this handed down Drupal project to work as smooth as possible. Ideally, my goal is to do changes local, and when that looks okay, I should be able to git push this to the server. Earlier I've made myself able to push structure changes through git hooks/post-receive, CodeIgniter Migrate and some bash scripts. This was worked out really neat, and I'm interested in similiar solutions for my Drupal project.
Ideally, I would like to upon git push have my php script generate a .sql file, send it to the server and let the server execute it so that it replaces live tables as ideal. Since our main page will handle products, orders and such that can't be overwritten, I can't simply just clone the database.
The thing I'm mostly puzzled out is, how do I tell what tables do what in my Drupal database? I have around 300 tables, and am not fully sure how dependent the tables are of each other, which that handles structure changes such as enabling and disabling modules and so on. Does anyone share an insight regarding this?
Maybe there's some easy solution for what I'm after that I've missed?
Comment: Integrating git with MySQL is not Drupal-specific. If your question isn't really about that, consider changing title and making it clearer.

Question about complete Drupal's database documentation [was asked already](http://drupal.stackexchange.com/questions/21096/looking-for-a-complete-documentation-for-drupals-database-schema-need-to-have)
Comment: Noted and changed the title of the question.
Answer: You can skip migrating the following tables:

accesslog
cache_*
sessions
watchdog

All the above contains material that you generally don't want to migate across sites.
The contents of the tables for built-in search will be regeneated by cron if you don't migrate them, but it may take up a lot of resouces if you've a lot content. But if this is not a concern, you can also skip migrating:

search_*

As for table names, Drupal relies heavily on namespace dicipline to avoid collisions.  So a module named foobar should only create tables with a name that starts with foobar_ (or just foobar if it only creates a single table).  In other words, you can tell what tables in the database that "belongs" to what module by their name.  This goes for both core modules and for regular contributed modules.  As for custom modules (i.e. modules that is managed outside the Drupal.org framework, you may find exceptions to this naming convention - but only if they're written by bozos :-).
If you look at your database, you may discover a lot of tables with names starting with field_.  These are of course from the (core) Field module, but I mention these specifically because these are the tables used for custom content types that are created using "Fields in core" (a key Drupal feature).
As for interdependencies between tables, looking at the fields in the tables that belong to the same module is usually enough.  Getting familiar with the Schema API is also helpful.  But to be honest: Sometimes you need to sit down and read source code.
As for creating and dropping tables, well behaved modules create datebase tables using hook_schema (found inside the module's .install-file) upon installation, and automatically drop the same tables upon un-installation.
Note that just pushing a module with git doesn't install the module.  It just makes it available for installation.  To install it, you either do it manually (using the administrator's GUI), or script it using Drush.
It is also useful to know that there are clear rules in the Drupal community of practice that a module should only write to its own tables.  If a client module want to write to another module's table, it must do so through the module's API, not by interacting with the database.  Module interdependency is tracked in the module's .info-file.  This means, for instance, that you'll not be able to disable a service module a client module relies on (provided their interdependency has been correctly declared).
Comment: Thanks for the input! Generally I'm mostly interested in applying the changes done from the local code. As far as I know, the most relevant part is to transfer whatever data that checks what modules are enabled. I want to keep data such as products and order history intact. Maybe I could execute Clear Cache upon `git push` aswell. Does new modules ideally create database structure in some way which I don't have to worry about, and maybe just execute some install .php file upon `push`? Will try to read documentation, but Drupal's is hard for me to learn from.
Comment: Many thanks for this post. Upvoted for now, and I'll try to get my head into this Drupal project which hopefully follows these rules, but judging by the code I've seen so far, may not be the case. Drush sounds as something worth checking out! Will answer-mark you once I've confirmed that this indeed was what I'm after.


Merging two dataframes in a loop change original dataframe in loop

Question: Assuming I have a list of dataframes
dataframes = [a, b, c, d]

and one dataframe additionalInformation  containing information I need for the merge.
Is there a way to join the dataframes in a loop and overwrite the original dataframe?
for index, df in enumerate(dataframes):
    dataframes[index] = pd.merge(df, additionalInformation, how="left", left_on="cat", right_on="cat")

However, this is not updating the dataframes.
When I do a
a.columns

The columns from additionalInformation are not merged... When I perform a
a = pd.merge(a, additionalInformation, how="left", left_on="cat", right_on="cat")

It works.
How would I merge dataframes in a loop and overwrite the original dataframe?
Comment: Never change the content of a loop on its elements. This is a general rule in Python (and in general). The solution: work on a copy, or create a new output and merge it later.
Comment: When you assign something to `a` you make `a` point towards this object, whether or not `a` pointed to something else is irrelevant an other objects won't be modified. If you do `a = 1 ; a = 2`, do you expect `1` to be equal to `2`?
Comment: check the `dataframes[0]`.  `pd.merge` returns a new object and won't mutate the original `a`.
Answer: If you want to modify the DataFrame, you have to assign to the dataframe's content, not the variable.
If the cat values in additionalInformation are unique (thus making the left-merge keep the same number of row), you can use:
for index, df in enumerate(dataframes):
    merged = pd.merge(df, additionalInformation, how="left", on="cat")
    dataframes[index].loc[:, merged.columns] = merged

If the values are not unique, this will truncate the output


What if the rewards are revealed in a bandit problem

Question: Take a n-armed bandit setup. When agent pulls an arm, the environment reveals rewards of all arms at that time step.
My question is will existing algorithms work well in this setting ? Can we take advantage of all the information revealed to us ?
Answer: Yes, I think many algorithms take the advantage of all the information revealed by all the other arms at a time stamp.
Let's take RL (Reinforcement Learning) for e.g
There is a trade off of exploration and exploitation in RL. We apply e-greedy policy in which our agent chooses the action that corresponds to the largest expected value with e probability. In this we can easily see our agent is exploiting information given by different arms to continue to learn more about them.


Basic finite dimensional distribution question

Question: I'm having trouble wrapping my head around the basic idea of a finite dimensional distribution.  Suppose $(\Omega, \Bbb P, \mathcal{F})$ is a probability space.
Let $(X_{t})_{t \geq 0}$ be a stochastic process, i.e., $X_{t}: \Omega \to \Bbb R $ is a random variable for each $t$.
Let $\mu_{t}$ be the distribution measure of $X_{t}$ on $\mathcal{B}(\Bbb R)$, i.e., $\mu_{t}(A) = \Bbb P(X^{-1}(A))$ for each $A \in \mathcal{B}(\Bbb R)$.
Given $t_{1}, t_{2}, \dots, t_{n} \geq 0$, we can talk about the finite dimensional distribution of the vector $(X_{t_{1}},X_{t_{2}},\dots,X_{t_{n}})$.  Specifically, this is the measure $\mu$ defined by:
$$\mu(D) = \Bbb P \{ \omega \in \Omega \mid (X_{t_{1}}(\omega), X_{t_{2}}(\omega), \dots, X_{t_{n}}(\omega)) \in D \} $$ where $D \subseteq \Bbb R^{n}$ is a Borel set.  So $\mu$ is a measure on $\mathcal{B}(\Bbb R^{n})$.
For some reason, I'm having trouble understanding what this measure is doing.  For example, what if we have a measurable rectangle $A_{1} \times \cdots \times A_{n}$?  My gut is telling me that the measure of this is just the product of the distribution measures, i.e., $\mu(A_{1} \times \cdots \times A_{n}) = \mu_{t_{1}}(A_{1})\mu_{t_{2}}(A_{2}) \cdots \mu_{t_{n}}(A_{n})$.  Is that right?  Can anyone give me more information about this distribution?  I'm very new to this material.
Comment: If the $X_{t_k}$ are independent that would be correct, but that was not part of the hypotheses...
Comment: @copper.hat Thanks for the observation.  Can you give me any additional insight into this measure?
Comment: Not really. I think of it as 'sampling', but that doesn't add insight.
Answer: A finite-dimensional distribution is just the joint distribution of a random vector. You pick various time points $t_1,t_2,\ldots,t_n$, observe the stochastic process at those time points, and ask yourself "what's the distribution of this random vector?"
To define a FDD it is sufficient to specify the joint distribution on rectangles $A_1\times A_2\times \cdots\times A_n$ (as you've suggested), or  on products of half-infinte intervals $(-\infty,a_1]\times(-\infty,a_2]\times\cdots\times(-\infty,a_n]$ (to generalize the one-dimensional CDF of a scalar random variable). In general, these joint distributions won't correspond to product measure unless the selected time points yield independent variables; typically you get a joint distribution of correlated random variables. For example, if the stochastic process is Brownian motion, then the finite-dimensional distributions will be correlated multivariate gaussian.
The simplest finite-dimensional distribution occurs when $n=1$: you get just the distribution of $X_t$, the process observed at a single time point.
Comment: What is a "joint distribution" of a vector?  I think if I understood that, it would really help me, but I'm not sure what it is.
Comment: If the random vector is $d$-dimensional, its joint distribution is the probability distribution on $\mathbb R^d$ of the values of the vector. You can usually specify the joint distribution using a $d$-dimensional density function, if the vector is continuous. Usually we say "joint" for the distribution of the whole vector, to distinguish it from the "marginal" distribution of each component of the vector.
Comment: Example: To specify the joint distribution on half-infinite intervals you need to report $P(X_{t_1}\le  a_1, X_{t_2}\le a_2, \ldots, X_{t_n}\le a_n)$ for all choices of $(a_1,\ldots,a_n)$.
Comment: I want to add my thoughts about this and see if you agree with them before I take any further action on this question.  Here is my (potentially incorrect) idea:  The reason we call the measure $\mu(A) = \Bbb P(X^{-1}(A))$ the *distribution* of the random variable $X$ is because it gives us information on where $X$ is "likely" to $B$.  In other words, it tells us how likely $X$ will fall within the set $A$, so that if we somehow were to represent all possible locations of $X$, we would get a picture of how $X$ is "distributed" throughout $\Bbb R$.  Is the in any way a correct interpretation...
Comment: ...of distribution?  If not, can you give me a better intuitive explanation of what a distribution is?
Comment: Yes, your intuition is correct. $P(X^{-1}(A))$ is the probability that $X$ falls within set $A$. The distribution is the collection of all such probabilities for all possible $A$.


How do I change taxonomy page callback for a specific vocabulary only?

Question: Following Clive's answer to this question, I would like to change taxonomy page callback and redirect users to another page just for a specific vocabulary.
function MYMODULE_taxonomy_page_callback($term) {
  if (/* Check if the vocabulary is the one we need to implement hook to */) {
    drupal_goto('somewhere');
  }
}

How can I achieve this?
Answer: Since the callback is getting the taxonomy term object as argument, you can check the value in $term->vocabulary_machine_name to get the machine name of the vocabulary, and see if it is the one you are interested in.
function MYMODULE_taxonomy_page_callback($term) {
  if ($term->vocabulary_machine_name == 'MYVOCABULARY') {
     drupal_goto('somewhere');
  }
  else {
    return taxonomy_term_page($term);
  }
}


Classic ASP check if an element in array exist

Question: I have the below code, i want to check if index 3 in array is existing or not, but i always  got this error :

Microsoft VBScript runtime error '800a0009'
  Subscript out of range: '[number: 0]'

<code>urlArray=Split(url1,"/")

If (not isNull(urlArray(3))) then
    If (urlArray(3)="site") Then
        newUrl=urlArray(0) &"/"& urlArray(1) &"/"& urlArray(2) &"/m/" & urlArray(4) & "/" & urlArray(5)
    Else
        newUrl= url1
    End If
Else
    newUrl= url1
End If
</code>
Answer:
Use function "IsArray" to make sure that variable is array, see here:

IsArray returns True if the variable is an array; otherwise, it
  returns False. IsArray is especially useful with variants containing
  arrays.

Use function "UBound" to check upped bound of array, see here:

Returns the largest available subscript for the indicated dimension of
  an array. UBound(arrayname[, dimension])
  The lower bound for any dimension is always 0.
Comment: thanks I used UBound to check if the array has elements or not :) .


How should I understand $\frac{\partial^2 v_i}{\partial x_j\partial x_j}(x)=\frac{\partial p}{\partial x_i}(x)$?

Question: The formula is from the first paragraph in the paper "Second Kind Integral Equation Formulation of Stokes' Flows Past a Particle of Arbitrary Shape" by Power and Miranda:

... the governing equations for the auxiliary perturbed fluid velocity $\vec{v}=(v_1,v_2,v_3)$ and pressure $p$ can be approximated by the creeping motion and continuity equations
  $$
\begin{align}
&\frac{\partial^2 v_i}{\partial x_j\partial x_j}(x)=\frac{\partial p}{\partial x_i}(x)\\
&\frac{\partial v_i}{\partial x_i}(x)=0
\end{align}
$$

Here are my questions:

How should I understand the first formula? Is the Einstein summation convention applied here, i.e.,
$$
\sum_j\frac{\partial^2 v_i}{\partial x_j\partial x_j}(x)=\frac{\partial p}{\partial x_i}(x)?
$$
Is there any reference for the derivation of the first formula? (I don't find any in the paper.)
Comment: http://en.wikipedia.org/wiki/Creeping_motion

Einstein summation convention is indeed implied.  In other words, the equation can be written as
$$
\nabla ^2\mathbf{v}=\mathbf{\nabla}p.
$$

As for the derivation, according to the article, it follows directly from the Navier-Stokes equations, but as I don't know what they mean by ". . . the inertial forces are assumed to be negligible . . .", I'm afraid I can't explain this.
Comment: @JonathanGleason, Thanks for your comment. It is the answer I'm looking for. When I read your $\nabla^2 v=\nabla p$, things are clear to me. May I suggest you to write it down as an answer?
Comment: The second formula in the question, I think, means $\nabla\cdot v=0$, instead of $\nabla v=0$.
Answer: See the Wikipedia article Creeping Motion.
Einstein summation convention is indeed implied. In other words, the equation can be written as $\nabla ^2\mathbf{v}=\mathbf{\nabla}p$.
As for the derivation, according to the article, it follows directly from the Navier-Stokes equations, but as I don't know what they mean by ". . . the inertial forces are assumed to be negligible . . .", so I'm afraid I can't explain this, although perhaps this will help.
Note:  The original reason I did not want to post this as an answer was because I don't know how to completely answer your second point.  In any case, if you are happy with this answer, then I am certainly happy to post it as one :).
Comment: The "inertial forces" neglected in the Stokes equation are simply the time derivative terms. Navier Stokes has the form "Forces = mass* acceleration" as in classical mechanics, but formulated for a fluid element and "mass * acceleration" is what they call inertial forces. The reason this is neglected is that the limit of Reynolds number $ \to \infty$ is taken. Another way of deriving this equation is by minimizing the dissipation $\int_V (\partial_i v_j) (\partial_i v_j) \mathrm{d}V$ for an incompressible flow.


Merge differencing and parent vhd into new vhd

Question: Using DISKPART and the "VDISK MERGE" option, is it possible to specify that you want to keep the existing child/parent disks and do a merge into a new vhd file? That is; I still want to keep my original child/parent disks. As far as I can tell from the resources I've found, the only option is to select a differencing vdisk and merge it with it's parent(s).
Answer: Take a copy of the parent and child first, do the merge, and then rename the copies back.
Comment: Yes, that's what I ended up doing too. Find it a bit strange that there's no option for specifying an 'output file' in the VDISK MERGE command though.
Comment: If you want to duplicate a disk image, which is essentially what you're suggesting, you don't want to merge, you want to mirror.  Mount the child VHD, create a mirror and then break it once it's complete.
Answer: Cant find any other useful information on that sorry, my suggestion: boot the machine with with some WinPE or Dos from cd/floppy and use norton ghost or altiris to clone the disk that the machine sees to an other VHD drive or file share location.
I guess linux and some "dd" magic will do to.


400 Http Errors Using Jsoup in Multithreaded Program

Question: I've created a program that parses html pages. I use jsoup connect function within a callable class inside ThreadPool. The problem is that I'm connecting to the same website and with a thread pool size of 5+, I get IO Exceptions - 400 errors.
How do I not make that happen?
Answer: If you're getting a 400 HTTP response, check the content of the response for an error message. A 400 means a bad request of some kind: you didn't include all the required information or included malformed information. Some people also use it as kind of a catch-all for when the client did something they don't like. If you're making lots of different requests, examine the ones that caused 400's to see if there's something wrong with them. If they all look right or if you're sending the same requests repeatedly, then maybe the site you're hitting has some kind of rate limiting that disallows you from making too many concurrent requests or too many requests within a particular timeframe. If it's something like that, I'd expect there to be a message in the response telling you what's going on.


Plastic ring on bow?

Question: Does anyone know why the plastic ring in the photo below would be put on the bow for an instrument? I thought maybe it would be to help shuck off broken strings, or somehow vary the tension, but neither of these seem plausible. Maybe it was put simply to keep the strings safer during shipping?
Does anyone have any idea what the purpose of the little ring could be?
Comment: What instrument is this for?
Comment: @Dave It's called a kokyu- https://en.wikipedia.org/wiki/Koky%C5%AB
Answer: Your last guess is almost certainly correct: the ring is just to keep the hairs from getting jumbled during shipping.  You can safely remove it.

Plastic ring on bow?


Apollo useQuery gives null result for the subscription in react app

Question: I am using Apollo Client (React) 3.0. The backend is sending the subscriptions correctly.
I am getting a null value for the result "prev" when I receive 2 subscriptions (received and sent) at the same time from the backend.
My use case here is auto-reply on certain keywords. For eg:
Received: Help ( sub: MESSAGE_RECEIVED_SUBSCRIPTION )
Sent: "Auto-response" (sub: MESSAGE_SENT_SUBSCRIPTION )
In this case, "prev" is set on received subscription but not on sent subscription.
Here is my code:
<code>const { loading, error, data, subscribeToMore } = useQuery<any>(GET_CONVERSATION_QUERY, {
    variables: queryVariables,
  });

  // handle subscription for message received and sent
  const getMessageResponse = useCallback(() => {
    // message received subscription
    subscribeToMore({
      document: MESSAGE_RECEIVED_SUBSCRIPTION,
      variables: queryVariables,
      updateQuery: (prev, { subscriptionData }) => {
        console.log('calling message received sub');
        ...
      },
    });

    // message sent subscription
    subscribeToMore({
      document: MESSAGE_SENT_SUBSCRIPTION,
      variables: queryVariables,
      updateQuery: (prev, { subscriptionData }) => {
        console.log('calling message sent sub');
        ...
      },
    });
  }, [subscribeToMore, queryVariables, updateConversations]);

  useEffect(() => {
    getMessageResponse();
    // we should call useEffect only once hence []
  }, []);
</code>
Any help is appreciated.
Answer: I found out the issue. There were differences in the returned values for query and subscription hence, it was failing and maybe triggering the query again resulting in the null result on the second run.
Hope this helps!


Remover caracteres especiais e espaços de uma string?

Question: Tenho um problema, em um app estou pegando os contatos da agenda telefônica, porém quero fazer um tratamento nos números dos contatos que podem vir assim:
(99) 9999-9999
9999-9999
9999 9999

e entre outras coisas, o único tratamento que fiz foi usar um SubString para cortar e pegar apenas os últimos 8 caracteres do número (para tirar operadora etc), agora como faço para remover caracteres que não sejam números dessas string? No exemplo acima todos os números ficariam assim: 99999999
Comment: Só para deixar anotado, existem números de 9 dígitos também, em algumas regiões. Logo, pegar os últimos 8 caracteres pode não ser a melhor opção dependendo do uso de seu app.
Comment: Aqui peguei os 8 dígitos pois o app é para Windows Phone (que não possui nenhuma ferramenta que coloca 9 digito) e estou tratando isso depois.
Answer: Alguém vai dar uma solução com RegEx, mas eu prefiro assim, utilizando método de extensão:
var texto = "(99) 9999-9999";
foreach (var chr in new string[] {"(", ")", "-", " " }) {
    texo = texto.Replace(chr, "");
}

Veja funcionando no ideone. E no .NET Fiddle. Também coloquei no GitHub para referência futura.
Obviamente se quiser guardar isto na variável precisa armazenar de novo o resultado do método na variável.
Dá para fazer de forma mais otimizada. Exemplo:
public static class StringExt {
    public static string Replace(this string str, string newValue, params char[] chars) {
        var sb = new StringBuilder();
        foreach (var chr in str) {
            if (!chars.Contains(chr)) {
                sb.Append(chr);
            } else {
                sb.Append(newValue);
            }
        }
        return sb.ToString();
    }
}

Veja funcionando no ideone. E no .NET Fiddle. Também coloquei no GitHub para referência futura.
Comment: O que quer dizer com otimizada? Usa menos recurso de memória? ou o que ?
Comment: Memória e processamento.
Answer: Como foi previsto pelo @Maniero, vou mostrar o método utilizando Regex.Replace() substituindo tudo em uma só linha, fique a vontade para escolher como deseja trabalhar:
string strTexto = "(12) 3456-7890";
strTexto = Regex.Replace(strTexto, "[\\(\\)\\-\\ ]", "");
Console.WriteLine(strTexto);

Rodando no .NETFiddle: https://dotnetfiddle.net/0cid0m


Filtering on a field from field collection in a view

Question: I have several node types (merchandise vendor, food vendor, festival vendor) which all have an embedded field collection called equipment, which can contain multiple values. The field collection has a taxonomy field called event and some others.
For example, a merchandise vendor can have equipment for event 1 (a taxonomy term), and different equipment for event 2 (a different taxonomy term).
My view can sort out the vendors for event 1 just fine. I can include the equipment for the events in the view, but I can't seem to filter it to just event 1.
I want to be able to visit the view page for event 1, and have it show all the vendors for event 1, and the equipment each vendor needs for event 1. Right now, it shows all the equipment for the vendor for all the events.
Is there a way to filter a field collection in a view?
Comment: Not a proper answer but an idea. Perhaps the Field Collection Views module can be of help: http://www.youtube.com/watch?v=_GlfYhsk3X0 https://drupal.org/project/field_collection_views
Comment: The Field Collection Views module is for creating very nice content formatters via Views, and within views, not exactly what OP is looking for, but also very useful.
Comment: True Renee, my bad.
Answer: You can create a Relationship to the field collection, and then you'll have access to all the field collection's fields within the collection from both "Fields" and "Filters".
Comment: Have you tried messing with the delta on the relationship?
Comment: The delta is "All." If I take the relationship away from the contextual filter and set the delta to 1, it seems to work. This is entirely unintuitive.
Comment: I added the relationship to the field collection, and all the collection's fields are available in fields and filters. If I add the relationship to the contextual filter, then I can't get vendors for event 1 who don't have equipment for event 1. Without the relationship in the filter I get them, but I also get their equipment for event 2, when I only want event 1.


What principled arguments are made with data?

Question: Consider the frequentist argument: "I will believe A since not-A makes the data improbable".
Consider the Bayesian argument: "I will believe A since given my beliefs on A and the data generating process, it is more probable than not-A".
Are there any other argument types and how would you construct an argument? (Fiducial? Fuzzy logic? Dempster-Shafer? Possibility theory? Bounds? ...)
Answer: Sure, you can make one that doesn't use hypothesis tests (or any statements about A and NOT A) at all. Something like:

We had evidence of a relationship between A and B because of (cite
  literature) but we thought that this needed further exploration
  because (give reasons). When we did (cite analysis done) we found
  (cite relationship found) which (strengthens/weakens/alters) our
  notion of the relationship. Reasons for this are (give more reasons)
Comment: Thank you Peter. I agree this does qualify as a data driven argument. I guess I had a more quantitative approach in mind.


Android create new instance of the application

Question: Is there a possible way to create a new instance of the whole app?
What i'm doing is this,
I'm taking a photo, sending it to the server, and then send a mail. (without internet or 3g).
The 2nd time i'm trying to take the picture and send it to the server to send a mail and turn on my internet i'm getting two mails.
One with the last picture made and one with an empty picture (0 bytes).
Now there are two option:

create a mail sending queue.
or if possible create a whole new instance of the application so the old data won't be overwritten.

I hope the last one is available with some kind of intent-flag or something?
So is it? or isn't it? because then i'll need to build a queue for sending.
Answer:
Is there a possible way to create a new instance of the whole app?

No, sorry.

because then i'll need to build a queue for sending.

That would be recommended.


Transition matrix generated by throwing 5 dice

Question:
You throw five dice and set aside those dice that show sixes. Throw the
  remaining dice and then again set aside the sixes. Continue until you
  get all sixes. Exhibit the transition matrix for the associated Markov
  chain, where $X_n$ is the number of sixes after $n$ throws.

I figured that the matrix should have $6$ states, that is state space $S=\{0,1,2,3,4,5\}.$ Just by thinking about the first entry, $P_{00}$ we can think "starting with $0$ sixes, what is the probability that on the first throw, we throw $0$ sixes?". To do that, we simply need to throw each die with probability $5/6$, that is $P_{00}=(5/6)^5\approx 0.4018$, which is correct.
Same reasoning for $P_{01}$ gives $P_{01}=(1/6)^1\cdot (5/6)^4\approx 0.08037.$ This is incorrect, for the correct answer here one should apparently multiply the answer with 5. Why? Why did we not need to do that in $P_{00}?$
The general matrix element is given by
$$P_{ij}={5-i\choose j-i}\left(\frac{1}{6}\right)^{j-i}\left(\frac{5}{6}\right)^{5-j}, \quad \forall \quad 0\leq i,j\leq 5. \tag{1}$$
I'm trying to build this formula step by step, intuitivly, but I fail to understand the binomial. Can someone give me an explanation on how to construct $(1)?$
Comment: The $P_{01}$ that you've computed is the probability that a _specific_ die shows a six. That one die could be any one of the five dice, though, so you've undercounted.
Answer: Look at the case for just two dice:  then you have a transition matrix where $X_n \in \{0, 1, 2\}$.  What is the dimension of this matrix?  Then if $X_n = 0$, what is the chance that $X_{n+1} = 0$?  What kind of outcomes correspond to such an event?  Similarly, what is the chance that $X_{n+1} = 2$ if $X_n = 0$?  Finally, based on these previous results, what is the chance that $X_{n+1} = 1$ if $X_n = 0$?
Then do the same for $X_n = 1$:  Is it possible that $X_{n+1} = 0$ if $X_n = 1$?
Finally, consider $X_n = 2$.  Is this an absorbing state?
If you answer these correctly, then think about how this generalizes to more than two dice.


How do these two contorted functions, $F(x)(f)=f(x)$ and $\mathscr{F}(f)=(x\mapsto (t\mapsto f(t)(x)))$, work?

Question: First off: I haven't taken a functional analysis course yet, so please keep that in mind when explaining. I just rather randomly read about this stuff and it started to interest me.
I'm having trouble understanding how the following contorted mappings work. I don't want any more details than necessary; just the plane explanation which functions they take to which space and why it makes sense to define the mapping in the way they are defined (maybe like a diagram in words) and so on.
1) In this Wikipedia article it was mentioned, that there is a natural map $F$ from a vector space $V$ over a field $F$ to $V''$ (the dual of the dual, meaning the space of all linear functionals that take themselves linear functionals as  arguments and return values from $F$, if I understood it correctly ), by $F(x)(f)=f(x)$.
(The notation $F(x)(f)$ would imply that $F(x)$ is itself a function taking as an argument another function $f$. But what would be the domain and range of this $f$ ? And why does it make sense to assign the value of $F(x)(f)$ the value of $f$ at $x$)
2) Similar question (maybe more difficult), to be found in these lecture notes,page 61 bottom: Let $I$ be compact and $E,F$  Banach spaces and consider the mapping $\mathscr{F}:C(I,L(E,F)) \rightarrow L(E,C(I,F))$ defined by $\mathscr{F}(f)=(x\mapsto (t\mapsto f(t)(x)))$. Now I understand that to $f$ there is a map associated, lets call it $\mathscr{G}:E \rightarrow C(I,F)$, such that  $\mathscr{G} (x)$ is a continuous map from $I$ to $F$, $t\mapsto f(t)(x)$. But what I don't understand is, why the author says this mapping simulates the change of variables.If it did, why did the constraints to work in the space of continuous respectively linear maps had to be imposed?  And  does why this (latest) mapping from  $I$ to $F$ takes the value $f(t)(x)$ ?
Comment: All of what you're confused about can be understood on the level of sets; it isn't actually about functional analysis. See http://en.wikipedia.org/wiki/Currying .
Answer:
$V$ is a vector space; $V^*$ is the dual. So the elements of $V^*$ are functions $\mathbf{f}\colon V\to K$ ($K$ the field). The double dual $V^{**}$ is the dual of $V^*$, so the elements of $V^{**}$ are functions $\mathbf{x}$ with domain $V^*$ and range $F$; i.e., $\mathbf{x}\colon V^*\to K$.
Since $F$ is a function from the vector space $V$ to the double dual $V^{**}$, that means that for each vector $x\in V$, $F(x)$ is in $V^{**}$. So $F(x)$ is a function whose domain is $V^*$ and whose range is $K$. If $\mathbf{f}\in V^*$, then $[F(x)](\mathbf{f})$ (that is, the function whose name is "$F(x)$" evaluated at $\mathbf{f}\in V^*$) is a scalar. Now, $\mathbf{f}$ is itself a function with domain $V$ and images in $K$, so $\mathbf{f}(x)$ "makes sense" (we are evaluating a function with domain $V$ at an element of $V$), and is in fact a scalar. So we can define $[F(x)](\mathbf{f})$ as $\mathbf{f}(x)$, because both sides are scalars, which is what the value of $F(x)$ at an element of $V^*$ has to be.
Added. There still seems to be some confusion here, so let's make sure everything works explicitly here. We have now seen that in fact the formula "makes sense" in that both sides are what they ought to be: scalars. Now, why does this define a linear function from $V$ to $V^{**}$?
Let $x\in V$. We want $F(x)$ to be an element of $V^{**}$, so $F(x)$ needs to be a function with domain $V^*$. How do we describe a function with domain $V^*$? By saying what it does to every vector in $V^*$. So take a vector $\mathbf{f}\in V^*$; then $F(x)$ is the function that is defined by the rule $[F(x)](\mathbf{f}) = \mathbf{f}(x)$.  This defines a function, called $F(x)$, with domain $V^*$ and images in $K$. In order to verify that $F(x)$ is in fact an element of $V^{**}$, we need to verify that this function is in fact a linear function with domain $V^*$. To that end, let $\mathbf{f},\mathbf{g}\in V^*$, and let $\alpha\in K$. We need to show that
$$[F(x)](\mathbf{f}+\alpha\mathbf{g}) = [F(x)](\mathbf{f}) + \alpha[F(x)](\mathbf{g}).$$
Indeed, the left hand side is, by definition of $F(x)$ and by the definition of what it means to add linear transformations, equal to:
$$[F(x)](\mathbf{f}+\alpha\mathbf{g}) = (\mathbf{f}+\alpha\mathbf{g})(x) = \mathbf{f}(x) + \alpha\mathbf{g}(x).$$
The right hand side is equal to, by definition of $[F(x)]$,
$$[F(x)](\mathbf{f}) + \alpha[F(x)](\mathbf{g}) = \mathbf{f}(x) + \alpha\mathbf{g}(x).$$
So they are indeed equal, hence $[F(x)]$ is a linear function with domain $V^*$ and image in $K$; that is, $[F(x)]$ is an element of $V^{**}$, as desired.
Finally, we need to check that the map $F\colon V\to V^{**}$ is itself a linear transformation. That is, we need to check that for all $x,y\in V$ and all $\beta\in K$,
$$F(x+\beta y) = F(x) + \beta F(y).$$
Now, this is an equality of functions; both sides are functions with domain $V^*$ and codomain $K$, so in order to verify that the two functions are equal we need to check that they have the same value at every element of the domain, which is $V^*$. So let $\mathbf{g}\in V^*$. We need to check that $[F(x+\beta y)](\mathbf{g}) = [F(x)](\mathbf{g}) + \beta[F(y)](\mathbf{g})$. And indeed, we have:
$$\begin{align*}
[F(x+\beta y)](\mathbf{g}) & = \mathbf{g}(x+\beta y) &&\text{(by definition of }F\text{)}\\
&= \mathbf{g}(x) + \beta\mathbf{g}(y) &&\text{(because }\textbf{g}\text{ is linear)}\\
&= [F(x)](\mathbf{g}) + \beta[F(y)](\mathbf{g}) &&\text{(by definition of }F\text{)}
\end{align*}$$
So $F$ is linear, and thus we have a linear transformation $F\colon V\to V^{**}$ with domain $V$ and coddomain $V^{**}$.
$\mathcal{C}(I,L(E,F))$ is the set of continuous functions whose domain is $I$, and whose range is an element of $L(E,F)$ (the continuous linear functions from $E$ to $F$). So if $\mathbf{f}\in\mathcal{C}(I,L(E,F))$, and $a\in I$, then $\mathbf{f}(a)$ is a continuous linear function from $E$ to $F$.
Now, let $f\in \mathcal{C}(I,L(E,F))$. We want to define $\mathscr{F}(f)$ to be an element of $L(E,\mathcal{C}(I,F))$. That is, we want $\mathscr{F}(f)$ to be a function with domain $E$ and whose values are continuous functions from $I$ to $F$. How do we describe a function? By saying what its values are at a point in the domain. So let $x\in E$. The notation $x\mapsto A$ means that we want the function to send $x$ in the domain to $A$. So
$$\mathscr{F}(f) = \Bigl( x \longmapsto \bigl(t\mapsto f(t)(x)\bigr)\Bigr)$$
is telling you that $\mathscr{F}(f)$ is a function that will take $x$ to... well, the image of $x$ is a function (from $I$ to $F$). How do you describe a function? By saying what it does to every point in the domain. So I need to tell you what $[\mathscr{F}(f)](x)$ does to each $t\in I$. What it does is to send $t$ to $f(t)(x)$. This makes sense because $f(t)$ is a linear function from $E$ to $F$, so evaluating it at $x\in E$ gives an element of $F$.
In general, one can think of a function of two variables as a function of single variable whose values are functions of single variables. For example, given $f(x,y) = x^2-y^2$, you can think of it instead as a bunch of functions, one for each value $a$ of $x$: $f_a(y) = a^2-y^2$. Intuitively, the first coordinate tells you which function to use, and the second coordinate tells you where to evaluate it. So instead of thinking of this function as a function with domain $\mathbb{R}\times\mathbb{R}$ and range $\mathbb{R}$, we can think of it as a function with domain $\mathbb{R}$ and range $\mathbb{R}^{\mathbb{R}}$ (functions from $\mathbb{R}$ to $\mathbb{R}$). This is called currying. But you can also think of $f(x,y)=x^2-y^2$ as follows: the second coordinate tells you which of a family $g_b$ of functions to use, and then you evaluate $g_b(x)=x^2-b^2$ at the first coordinate.
By the same token, a function whose values are functions can be thought of instead as a function of several variables. So an element of $C(I,L(E,F))$ can be thought of in two ways:

As a continuous function with domain $I$ and image linear maps from $E$ to $F$; or
As a function with domain $I\times E$, continuous in the first component and linear in the second component, and values in $F$.

Likewise, $L(E,C(I,F))$ can be thought of in two ways:

As a function with domain $E$ and image continuous functions from $I$ to $F$; or
As a function with domain $E\times I$ and values in $F$, linear in the first coordinate and continuous in the second.

As you can see by looking at the second descriptions of each, the two sets, $C(I,L(E,F))$ and $L(E,C(I,F))$ "should" be the same thing. The only difference is that in the first case we are thinking of $I$ as giving the "index" of the function you will use, which you then evaluate at a point in $E$, while in the second we think of $E$ as giving the "index" of the function you will use, which you then evaluate at a point in $I$. The description given just formalizes the fact that the first description of each set actually gives the same set "morally".

As far as restrictions: you don't need to restrict in principle: if you let $Y^X$ be the set of all functions from $X$ to $Y$, then for any three sets $A$, $B$, and $C$, "currying" tells you that there are natural bijections
$$(C^B)^A \longleftrightarrow C^{B\times A} \longleftrightarrow (C^A)^B$$
(one reason for the notation). When dealing with Banach spaces, you are mostly interested in functions that "respect the structure" (linear, and continuous); when dealing with topological spaces, you are mostly interested in functions that preserve the structure (continuous). Here, we have a topological space, $I$, and two Banach spaces, $E$ and $F$. So we want to focus only on the functions that "respect the structure". So we want them to be linear when between $E$ and $F$; and continuous when the domain is $I$.
Added. "Natural bijections" above means:

Given any sets $A$, $X$, and $C$, a map $f\colon X\to A$ induces a map $C^A\to C^X$ by "precomposition with $f$": given any $h\in C^A$, the map $h\circ f$ is in $C^X$.
Given any sets $A$, $C$, and $Y$, a map $g\colon C\to Y$ induces a map $C^A\to Y^A$ by "postcomposition with $g$": given any $h\in C^A$, the map $g\circ h$ is in $Y^A$.

The bijections $(C^B)^A \leftrightarrow C^{A\times B} \leftrightarrow (C^A)^B$ are such that given maps $h\colon C\to Z$, $g\colon Y\to B$, and $f\colon X\to A$, the induced maps $(C^B)^A\to (Z^Y)^X$, $C^{A\times B}\to Z^{X\times Y}$, and $(C^A)^B\to (Z^X)^Y$ commute with the corresponding bijections:
$$\begin{array}{ccccc}
(C^B)^A & \longleftrightarrow & C^{A\times B} & \longleftrightarrow & (C^A)^B\\
\downarrow &&\downarrow &&\downarrow\\
(Z^Y)^X & \longleftrightarrow & Z^{X\times Y} & \longleftrightarrow & (Z^X)^Y
\end{array}$$
is a commutative diagram.
Comment: You should copyright your answers and put them together into a book of solved problems; I'd steal a scanned copy from the web...er, I mean, I'd buy it.
Comment: As a computer scientist and occasional functional programmer, I've always been amazed at how complex and opaque mathematicians' notation manages to make these things look. They are very simple plumbing operations with almost no content -- why the idiosyncratic, unsystematic notation, different each time you encounter it? I mean, capital F's in various fonts?? I'd kill (well, at least maim slightly) for a book that explained functional analysis or differential geometry in Haskell notation, but apparently there's a secret covenant among authors that it _has_ to look forbidding ...
Comment: @gary: There is no glory, perhaps, but good notation *is* important. As Leibniz said, find good notation and it can solve half your problem. Witness how much more flexible calculus is to use with Leibniz's notation than with Newton's. But it really is a matter of usefulness. We tend to use $f$, $g$, $h$ for functions, $x$, $y$ for points, $v$, $w$, for vectors, as a sort of "visual cue". This means that if we have lots of functions around, but want to keep those visual cues, we need to have ways of differentiating among several different $f$s and $g$s and $h$s$, $v$s and $w$s, etc.
Comment: Actually, now that I've calmed down a bit (sorry for hijacking the comment thread) I think what bothers me most is not how things are named, but that textbook authors almost never deign to spell out the _types_ of definitions explicitly. If the author here had only written "Define $\mathscr F : (I \to V \to W) \to V \to I \to W$ as ...", there would almost have been no need to look at the actual definition, except to verify that it did the natural thing. The arrows can be decorated to distinguish continuous and linear maps where appropriate.
Comment: Unlike in other fields; notably Electrical Engineering and the IEEE, there is no glory in setting up standards in math of notation or otherwise, so this is, unfortunately, not likely to change any time soon.
Comment: @Arturo: I agree with you completely; I was just agreeing with Henning on his lament on how difficult it can be to navigate through much of mathematical notation. It would be great if there were congresses every few years where mathematical notation would be standardized.
Comment: @Henning Makholm: Could you please tell me why you wrote $\mathscr{F}: \ (I \rightarrow V \rightarrow W) \rightarrow V \rightarrow I \rightarrow W$  instead of  $\mathscr{F}: \ (I \rightarrow (V \rightarrow W)) \rightarrow ((V \rightarrow I) \rightarrow W)$ ?  After reading through the article on currying on Wikipedia it seems that this notation is the precise one...
Comment: @Arturo: Thanks for the great answer. Though I still have two questions left. Concerning 1): It is now clear to me how the map $F$ works and that the equation $F(x)(f)=f(x)$ is correct in the sense that on both sides are scalars. But does this equation really define the map $F:V \rightarrow V^{∗∗}$, as it was said in the Wikipedia article ? Shouldn't the definition state precisely element which in $V^{∗∗}$ $F(x)$ should assume ?
Comment: (Continuation) Concerning 2): You said that there are natural bijections between $(C^B)^A \ldots$ . Did you mean by "natural" "obvious/easy to construct once you have your fixed $C,B,A$s" or did you mean something with a category-theory flavour ? If you meant the latter, could you mabye describe it in laymen terms ? (Please bare with me, that I just read somewhere that in category theory the word "natural" has a precise meaning - but I personally don't have even the vaguest idea what categories are and how the word "natural" is given a precise meaning there.)
Comment: @resu, in the programming language tradition that I'm advocating for here, the $\to$ associates to the right, so $(T\to V\to W)\to V\to I\to W$ is the same as $(T\to(V\to W))\to(V \to(I\to W))$. The reason for this convention is that it minimizes the number of parentheses needed for a curried function. Your reading is _almost_ right, except that $((V\to I)\to W)$ would be something that took a _function_ in $V\to I$ as argument and gave you a $W$, whereas you actually have a something that takes a $V$ and gives you a function form $I$ to $W$.
Comment: @resu: (1) You need to prove that the image is in fact a linear function (i.e., that $F(x)$ is linear on $V^*$), and that the map itself is linear (linear on $V$); I will add that. (2) Natural in the sense of "natural transformations". E.g., given a map $g\colon C\to Z$, you automatically get a map $C^A\to Z^A$ by "post-composing with $g$". If you have a map $f\colon X\to A$, you get a map $C^A\to C^X$ by "pre-composing with $g$". (cont)
Comment: @resu: If you have bijections $X\to A$, $Y\to B$, and $C\to Z$, they induce a bijection $(C^B)^A\to (Z^Y)^X$, $(C^A)^B\to (Z^X)^Y$, and $C^{A\times B}\to Z^{X\times Y}$; these bijections commute with the bijections we define in general between the sets $(R^S)^T$, $(R^T)^S$, and $R^{S\times T}$.
Comment: @resu: Re-reading your comment: The map does in fact explicitly tell you what element $F(x)$ is! Remember, elements of $V^{**}$ are themselves maps with domain $V^*$ and codomain $K$. In order to specify what function $F(X)$ is, I just need to tell you what it does to any element of $V^*$. That is **precisely** what the equation $[F(x)](\mathbf{f}) = \mathbf{f}(x)$ is saying: it's telling you what scalar $\mathbf{f}$ is sent to under the map $F(x)$.
Comment: @resu: Nonentheless, the bijections between $(C^B)^A$, $C^{A\times B}$, and $(C^A)^B$ are pretty easy: they are precisely the ones I described above via "currying": given a function $f\colon A\times B\to C$, think of it as a function from $A$ to the functions from $B$ to $C$ by sending $a$ to $f(a,-)$; or from $B$ to the functions from $A$ to $C$ by sending $b$ to $f(-,b)$.
Comment: @Henning: I'm sure there are those who say Haskell notation is forbidding too. Actually, even $\lambda$ notation is not well-known in mathematics, and there are probably mathematicians who balk at the idea of writing functions as $f = \lambda x . \, x^2$. As for spelling out the type signature of a function, I think this is a matter of style. Most of the texts I've read recently have type declarations for functions. The trouble, perhaps, is that not everything is a function...
Comment: Great answer Arturo, thanks.


How does Spring and Hibernate interpret @Entity and other annotations

Question: I am confused as to how Spring and Hibernate interact through the annotation configuration.
If for example, I have the @Entity or any other '@___' annotation, how does Hibernate know to interpret this differently from Spring or any other annotations that are shared between Spring and Hibernate.
From my understanding, at least for the @Entity annotation, Spring will interpret this as a Java bean, while Hibernate understands this to mean that it will be creating a table for the database.
If then, what does @Id become interpreted for Spring? I know @Id indicates to Hibernate that the variable should be a primary key for the database?
A thorough explanation of the annotation configuration for Spring would be helpful as a precursor to your explanation of how these annotations work.
Thank you.
Comment: *From my understanding, at least for the @Entity annotation, Spring will interpret this as a Java bean,* Nope, how did you came up with that?
Comment: *what does @Id become interpreted for Spring?* nothing, just ignored.
Comment: A tutorial I was reading informed me of that... So then, the annotations or java based configuration, is inherent for Java Spring and with this Spring provides a framework that Hibernate is able to use for its own ORM?
Comment: Spring works on top of JPA, and Hibernate is standard JPA provided that comes with Spring.
Answer: Both frameworks are reading annotations that they are interested in. Its as simple as that.
Comment: These annotations are provided by the Spring framework?
Comment: You are asking me or telling me?
Comment: Also, how do you know what needs to be scanned? I was reviewing the LocalSessionFactoryBean and saw that you have to include the Model class to be scanned. Is this because SessionFactory is respective to Hibernate and therefore Hibernate needs to know where to look for the @Entity annotation?
Comment: `@Entity` `@Id` and all persistent stuff is a part of JPA not Spring.
Comment: You may have multiple persistance units ("databases") that does not reflect eachother, so you have a way of "grouping" entitities in them,
Comment: Okay so then Entity and Id are a part of JPA, that Spring is able to interpret because it is built on top of the JPA. the '@__' is Spring, but the Entity and Id are a part of JPA which Spring is able to utilize.


Meaning of the $\Sigma^{-1/2}$ operator

Question: In the 2010 article "Closed-form Algorithms in Mobile Positioning: Myths and Misconceptions" by Niilo Sirola I came across the following expression:
$$
x=(\Sigma^{-\frac{1}{2}}M)\ \setminus\ (\Sigma^{-\frac{1}{2}}v)
$$
where $M$ is a $n$ by $3$ matrix and $v$ is a vector of length $n$.
In the article the author explains that the backslash operator \ is used as a shorthand for solving a linear system in least squares sense, i.e. $x = A\setminus b$ means "let x be the least-squares solution to $Ax = b$".
But he never explains the meaning of $\Sigma^{-1/2}$ and I wasn't able to google it or derive the meaning from context. By the looks of it, it should be a matrix operator that preserves the dimensions, since we need matrix on the left side of \ , and a column vector on the right side of it.
Is this a well-known notation? What does it mean?
Answer: This is the square root of a matrix, which given a symmetric, positive semi-definite matrix $A$ is the unique symmetric positive semi-definite matrix $B=A^{1/2}$ such that
$$A = B^⊤ B =  B^2 $$
If you have an eigendecomposition $A=UDU^⊤$ with $D=\operatorname{diag}(λ_1, λ_2, …, λ_n)$, then $A^{1/2}$ is simply $UD^{1/2}U^⊤$ with $D^{1/2}=\operatorname{diag}(\sqrt{λ_1}, \sqrt{λ_2}, …, \sqrt{λ_n})$.
Taking the square root commutes with taking the inverse, so
$$ A^{-1/2} = (A^{1/2})^{-1} = (A^{-1})^{1/2}$$
Comment: Ahh, so $\Sigma^{-1/2}$ is not an operator, it's a matrix that's being raised to -1/2. Thanks!

Meaning of the $\Sigma^{-1/2}$ operator


What is my unit of observation for this time series data set?

Question: My professor gave us a data set to run a regression and I have a data set which lists years from 1959-2007, gross private investments (in billions of dollars), and gross private savings (in billions of dollars). Each year has a value in billions of dollars. My professor says to not include the unit of observation in our regression.
What is my unit of observation? Is it years? Billions of dollars? Something else?
Comment: I would say it is dollars. You're approximating the change in savings with respect to a change in investment using a line; its slope m tells you, for every dollar saved, m dollars in investments.
Comment: I thought that at first also, but the professor gave us guidelines, one which says "All datasets should have a column denoting unit of observation. Do not include your unit of observation in the regression."
Answer: In econometrics you typically have three different types of data sets.
(1.) Time Series
(2.) Cross-Section
(3.) Panel
What differentiates these three types is the unit of observation used.
Time series, as the name implies, major unit is time (years, months, or even seconds if you are using financial data).
Cross-section, is the objects observed. In microeconomics this may be people or businesses, in macroeconomics countries.
Panel, uses both time series and cross-section. Combining the two together gives a pair, (individual, time) that acts as the unit of observation.

You are using time series data, thus a unit of time would be your unit of observation. In this case you got years.

Sometimes you may desire to include your unit of observation. This may be due to reasons such as trends or individual differences. In the end it depends on the underlying model and specification strategy.


Cocontinuous product-preserving functors between Grothendieck toposes

Question: What is an example of a functor $$F : \mathcal{C} \to \mathcal{D}$$ between two Grothendieck toposes which preserves colimits and finite products, but is not left exact (i.e., does not preserve pullbacks)?
I just assume that there is such an example, since otherwise the notion of an algebraic morphism between toposes would probably not include left-exactness. But I am not experienced enough in topos theory to see such an example.
My question is essentially why the forgetful strict $2$-functor from (toposes with algebraic morphisms) to (cocomplete symmetric monoidal categories with cocontinuous symmetric monoidal functors) is not fully faithful.
Answer: It has been answered on mathoverflow.
[I want to mark answered questions...]


Python: Splitting a txt file and keep header

Question: Say I have an input file like this (splitfile.txt):
INPUT
HEADER
OF A TXT FILE
line 1
line 2
line 3
line 4
line 5
line 6

I want to split these files and keep the three header lines like this:
INPUT
HEADER
OF A TXT FILE
line 1
line 2

INPUT
HEADER
OF A TXT FILE
line 3
line 4

INPUT
HEADER
OF A TXT FILE
line 5
line 6

My Python code so far is just only splitting up this textfile:
lines_per_file = 2
s = None
with open('splitfile.txt') as split:
    for lineno, line in enumerate(split):
        if lineno % lines_per_file == 0:
            if s:
                s.close()
            sfilename = 'step_{}.txt'.format(lineno + lines_per_file)
            s = open(sfilename, "w")
        s.write(line)
    if s:
        s.close()

How can I do this?
Answer: you can read the header and save it in a variable to write to each new file you create.
lines_per_file = 2
s = None
with open('a.txt') as f:
    lines = f.readlines()
    headers, lines = lines[:3], lines[3:]
    for lineno, line in enumerate(lines):
        if lineno % lines_per_file == 0:
            if s:
                s.close()
            sfilename = f'step_{lineno + lines_per_file}.txt'
            s = open(sfilename, "w")
            s.writelines(headers)
        s.write(line)
    if s:
        s.close()

A cleaner answer:
LINES_PER_FILE = 2

def writer_to_file(name, headers, lines):
    with open(name, "w") as f:
        print(headers + lines)
        f.writelines(headers + lines)

with open('a.txt') as f:
    lines = f.readlines()
    headers, lines = lines[:3], lines[3:]
    [writer_to_file(f'step_{i + LINES_PER_FILE}.txt', headers, lines[i: i+LINES_PER_FILE]) for i in range(0, len(lines), LINES_PER_FILE)]

I prefer this one because there is no global variable s and by using with statement there is no need to worry about closing file.
Also it's better to have UPPER_CASE constant variables.
Comment: the problem with this solution is, that it only keeps the first line of the header and does not keep line 2 and 3.


CoreData unexpectedly found nil while unwrapping an Optional value

Question: Having an app where I type username, pick a picture from imagePickerController and all that data is saved to core data and retrieved to tableview cell and is working fine, however if I don't choose picture app is crashing with log "unexpectedly found nil while unwrapping an Optional value", I forgot how to do that, can't remember in what project I solved that.
let imageData = NSData(data: UIImageJPEGRepresentation(photoImageView.image!, 1.0)!)
newUser.setValue(imageData, forKey: "image")

Something like if image data != nil {
} ??
Comment: You *have* to read this: [Swift Optionals](https://developer.apple.com/library/prerelease/ios/documentation/Swift/Conceptual/Swift_Programming_Language/TheBasics.html#//apple_ref/doc/uid/TP40014097-CH5-ID330) (Apple documentation).
Answer: Try if let image = photoImageView.image {//use the image}. This will unwrap the optional in a safe way.
Be careful when you use the ! operator, you are basically guaranteeing the thing will never be nil. Use the if let statement or ? operator unless there is absolutely no way that the variable in question could be nil.
Answer: solved with this
if photoImageView == nil {
        let imageData = NSData(data: UIImageJPEGRepresentation(photoImageView.image!, 1.0)!)
        newUser.setValue(imageData, forKey: "image")
    }

Comment: `if photoImageView == nil` then `photoImageView.image!` **will** crash, the same way it already does. You should read the answers and links you get to your questions, if you really want to learn. And in any case you should check that the code you post as an answer actually works.
Comment: My answer is "read the link I've provided in the comments to your question", it contains your answers.
Comment: While this code may answer the question, providing additional context regarding why and/or how this code answers the question improves its long-term value.
Comment: that code is working (almost), now i can proceed to tableview with typing only in textfield, no crashing now, and when i go back to choose a picture and click save i again got no picture. Somehow i need to use if statement to say if photo is not picked, proceed only with textfield, but i can't do that because i dont know yet, last 7 days i'm into core data, and i'm a little overwhelmed to remember how i manage to even save textfield and image and load into tableview with core data. PLs help with that if statement, so if photo is not picked proceed to tableview when save is clicked, thank you.


What is the name of this validation procedure?

Question: I have a set of classified data. In order to test the precision of several algorithms- I split the data into train and test sets. For the test set I choose at random 30% of the data and the rest is for the train set. I do $N$ iterations in which I split the data this way and take the average precision of the $N$ iterations.
Does this validation procedure has a specific name?
Comment: that looks more like repeated hold out with a 70/30 split, if you are sampling without replacement, bagging if you are sampling with replacement.
Comment: CrossValidation? :)
Comment: @CloseToC I need to distinguish this validation from the k-fold cross validation, so the name "CrossValidation" is not what I am searching for.
Comment: @image_doctor "Repeated holdout" -> this is what I was looking for. You can add the comment as an answer.
Answer: That looks like repeated hold trials out with a 70/30 split.


delete message for one user but not for the other

Question: hello I have a database with 3 tables.
USERS('user_id','name','surname')
MESSAGE_GROUP('user_one','user_two', 'hash')
MESSAGES('from_id','group_hash', 'messages')

My php code enables me to send messages between users. My question is how to enable a user to delete a message from its mailbox but the other user still watching the message. The messages must be full deleted only if both users delete the message. I am not interesting about the code, I am interesting only in finding the logic behind this. Any proposals that includes mysql code are welcome. thanks
Comment: Go through my answer. http://stackoverflow.com/questions/17085067/delete-message-for-one-user-but-not-for-the-other/17085238#17085238
Answer: I think you should follow this. :)

You can keep an extra field in message_group table something like 'deleted_from' which will be initially 0
If user one deletes it make the value of 'deleted_from'=1, if user two deletes it, make the value of 'deleted_from' = 2.
When you go to delete the message for a user, and you find the value 'deleted_from' other than 0, delete the message completely, else mark the value of 'deleted_from' as '1' or '2'.
Answer: You will need to either:

Make a new table that specifies the mailboxes the message resides in, so that you can connect it to both users mailboxes
Or duplicate the message so that each user has their own copy that can be deleted
Add flags to the message table (not recommended) indicating whether the sender or recipient has deleted it. This I would avoid as it will not scale well if you have (or intend to add) group messaging.
Answer: add to MESSAGE_GROUP a status field with values :
0 no owner and should be deleted
1 only the sender owns the message
2 only the reciever owns the message
3 both sender and recievers own it
Answer: I would change the fields from the table in between like this (example that every user only can send a message to one person at a time):
    USERS('user_id','name','surname')
    MESSAGE_GROUP('user_id','message_id')
    MESSAGES('from_id', 'to_id', 'messages')

So every user that has a message, will have a row in the MESSAGE_GROUP. When one user deletes the post, delete that row in MESSAGE_GROUP


Adding new nested div to document

Question: I want to add a div with a child div to the document but append() seems to be the wrong option for this, what should I use instead?
  //Create Statusbar
  var status = $('#status');
  for (var i = 0; i < resources.length; i++) {
    var resource = document.createElement('div');
    var resourceCounter = document.createElement('div');
    resourceCounter.id = "r" + (i + 1);
    //resource.className = "resource";
    resource.innerHTML = resources[i];
    resourceCounter.innerHTML = saveData.resources ? saveData.resources[i] : 0;
    resource.style.background = "url('images/resources/" + resource.id + ".jpg') 0 26px / 100% auto no-repeat";
    resource.onclick = function () {
      alert(this.id);
    };
    resource.append(resourceCounter);
    status.append(resource);
  }

resource.append(resourceCounter); --> Uncaught TypeError: Object #HTMLDivElement has no method 'append'
Answer: here resource is a dom element, append is a method provided by jQuery, it is not present in the dom element
you can use appendChild instead
resource.appendChild(resourceCounter);

or wrap resouces with jQuery and use append
$(resource).append(resourceCounter);

Adding new nested div to document


How to sum up a measure based on different levels in Power BI using DAX

Question: I have the following table structure:
| Name 1 | Name 2 | Month  | Count 1 | Count 2 | SumCount |
|--------|--------|--------|---------|---------|----------|
| A      | E      | 1      | 5       | 3       | 8        |
| A      | E      | 2      | 1       | 6       | 7        |
| A      | F      | 3      | 3       | 4       | 7        |

Now I calculate the following with a DAX measure.
Measure = (sum(Table[Count 2] - sum(Table[Count 1])) * sum(Table[SumCount])

I can't use a column because then the formula is applied before excluding a layer (eg. month). Added to my table structure and excluded month it would look like that:
| Name 1 | Name 2 | Count 1 | Count 2 | SumCount | Measure |
|--------|--------|---------|---------|----------|---------|
| A      | E      | 6       | 9       | 15       | 45      |
| A      | F      | 3       | 4       | 7        | 7       |

I added a table to the view which only displays Name 1in which case the measure of course will sum up Count 1, Count 2 and SumCount and applies the measure which leads to the following result:
| Name 1 | Measure |
|--------|---------|
| A      | 88      |

But the desired result should be
| Name 1 | Measure |
|--------|---------|
| A      | 52      |

which is the sum of Measure.
So basically I want to have the calculation on my base level Measure = (sum(Table[Count 1] - sum(Table[Count 2])) * sum(Table[SumCount]) but when drilling up and grouping those names it should only perform a sum.
Answer: You don't want measure in this case, rather you need new column,
Same formula but new column will give your desired result.
Column = ('Table (2)'[Count1]-'Table (2)'[Count2])*'Table (2)'[SumCount]

Comment: Thanks for the answer. I need to stick with a measure because the calculation should not be done on the initial table. It should be done when I filter out the `Month` column. And all upcoming filters should only sum up that `Measure` column.
Please check my corrected answer.
Answer: An iterator function like SUMX is what you want here since you are trying to sum row by row rather than aggregating first.
Measure = SUMX ( Table, ( Table[Count 2] - Table[Count 1] ) * Table[SumCount] )

Any filters you have will be applied to the first argument, Table, and it will only sum the corresponding rows.

Edit:
If I'm understanding correctly, you want to aggregate over Month before taking the difference and product. One way to do this is by summarizing (excluding Month) before using SUMX like this:
Measure =
VAR Summary =
    SUMMARIZE (
        Table,
        Table[Name 1],
        Table[Name 2],
        "Count1Sum", SUM ( Table[Count 1] ),
        "Count2Sum", SUM ( Table[Count 2] ),
        "SumCountSum", SUM ( Table[SumCount] )
    )
RETURN
    SUMX ( Summary, ( [Count2Sum] - [Count1Sum] ) * [SumCountSum] )

Comment: That is pretty close to my requirement. Thanks! Is it possible for SUMX to somewhow "change the starting level". Because now the calculation is made on the base table (with month) but it should be made on the second level where month is excluded. So there should be a distinct filter or something for `Name 1` and `Name 2` in `SUMX()`. Do you know what I mean?


Parâmetro de método com classe abstrata

Question: Olá. Acredito que minha dúvida seja muito simples, mas realmente não encontrei nenhuma solução, após muita pesquisa.
No Java, Tenho duas classes concretas que estendem uma classe abstrata, conforme o exemplo:
public class Animal {}
public class Dog extends Animal {
    public void bark() {}
    public void eat() {}
}
public class Cat extends Animal {
    public void eat() {}
}

Somente a classe Dog possui o método bark().
Gostaria que um parâmetro de um método aceitasse qualquer objeto Animal, então fiz isso:
public void doSomething(Animal animal) {}

Mas eu preciso que esse método faça o animal latir, caso seja um cachorro, então tentei isso, sem sucesso:
public void doSomething(Animal animal) {
    animal.bark();
}

Há alguma forma de fazer o que eu quero?
Obrigado!
Comment: Creio que criar um método equal() para seu animal, e se Animal.equal(Dog) == verdadeiro, latir irá funcionar.
Comment: http://javafree.uol.com.br/topic-3932-Comparar-variaveis--ou-equals.html
Answer: Pensando apenas em Orientaçao a Objetos, isso não é possível.
O método doSomething recebe a classe Animal, logo, esse método tem que saber o que fazer com esse tipo de classe, o método bark(), pertence apenas ao Dog e não ao Cat, logo, se seu metodo que recebe Animal, receber a classe Cat, porém tentar chamar o método bark(), ele ficaria "perdido", sendo assim, não é possível fazer o que você quer de forma correta.
O que você poderia ter, é um método "Acao" por exemplo, onde na classe Dog vc implementaria o código que faz latir e na classe Cat você implementaria o código que faz miar por exemplo.
Ai seu metodo chamaria animal.Acao() e cada implementação de Animal faz o que tem que fazer...
Se seu método já sabe que vai chamar o "bark()", então, em teoria, vc já sabe que vai receber um cachorro e não um Animal e ai não faz sentido o método receber um Animal como parametro.
Você consegue fazer isso que você quer conforme foi dito nos comentários, utilizando equals e convertendo o objeto, porém, você quebraria o Open/Closed principle, pois a cada nova implementação de Animal, vc teria que adicionar um "if" novo no seu método doSomething para tratar somente esta implementação nova.
Answer: Amigos, muitíssimo obrigado pelas contribuições, que foram muito úteis. Ainda sou novato no Java e estou patinando em alguns conceitos.
Acabei pensando mais um pouquinho no conceito que estava tentando empregar e vi que não fazia muito sentido (no sentido de orientação a objetos).
Resolvi da seguinte forma:
Criei uma interface "barkable", implementei-a no Dog e defini a restrição no método doSomething() para a interface, ao invés da classe abstrata.
Em código, seria isso:
interface Barkable() {
    public void bark();
}
public class Dog extends Animal implements Barkable {
    //...
}
public void doSomething(Barkable animal) {
    animal.bark();
}

[]'s!
Answer: O Java por si só não faz compara atributos ou objetos, mas existe um método na classe Object que pode ser reescrito para criarmos esse critério de comparação. Esse método é o equals().
O equals() recebe um Object como argumento e deve verificar se ele mesmo é igual ao Object recebido para retornar um boolean. Se você não reescrever esse método, o comportamento herdado é fazer um == com o objeto recebido como argumento.
Um exemplo clássico do uso do equals() é para datas. Se você criar duas datas, isto é, dois objetos diferentes, contendo 31/10/1979, ao comparar com o == receberá false, pois são referências para objetos diferentes. Seria correto, então, reescrever este método, fazendo as comparações dos atributos, e o usuário passaria a invocar equals() em vez de comparar com ==.
Você poderia criar um método com outro nome em vez de reescrever equals que recebe Object, mas ele é importante pois muitas bibliotecas o chamam através do polimorfismo.
Exemplo:
public class Pessoa extends Object implements Serializable {
   private int idade;
   private String nome;

   public Pessoa() {
      setIdade(0);
      setNome("Alguem");
   }

   public boolean equals(Pessoa p) {
      boolean compare = false;

      if ((this.nome.equals(p.getNome())) && (this.idade == p.getIdade())   )
         compare = true;

      return compare;
   }
}

Até agora é a única maneira que me vem a cabeça.
Fonte 1
Fonte 2
Fonte 3

Parâmetro de método com classe abstrata