How long can a pianist maintain a repeated, fast figure in one hand?

Question: I'm working on an atmospheric/minimalist piece for Pierrot ensemble.  I have the following figure in the piano part.  I know that while the tempo is rather brisk (it's in 6/8 time, dotted quarter note = 80) it is not terribly difficult.  But I'm wondering how many bars of this a professional-level pianist can maintain without physical discomfort.  Obviously because I can play it as a very-not-professional it should be fine once, but can I expect a pro to be able to go for 8 bars?  16 bars?  Will they kill me if I ask for a solid minute?
If this is not good for a pianist, what could help?  Any good workarounds?
Comment: Ask the snare drum player in *Bolero*...
Comment: Here's (an almost continuous) 4 minutes of repetition of an ostinato in 8ths at 170 bpm: https://musescore.com/user/24361621/scores/6794860 So yeah, I think a 1 minute repetition of 16ths at 120 should be fine. It _is_ faster though.
Comment: Should be absolutely possible.
Comment: Before (or as well as ) asking here, I'd be asking the player who has to play it. After all - it's for that person, and the legitimacy of it will be purely down to him, surely. And with nothing for the l.h. to do, if it was me, I might use both hands to play it..!
Comment: Good thought @Tim2 but there isn't a specific player in mind yet
Comment: A reasonably good pianist could play this for ever. (Well, for as long as they could play anything else.)
Answer: At 80 dotted quarter notes per minute I run into two problems:

Fingers getting twisted up and
Losing count of the repetitions.

As a skilled amateur, after a few minutes of practice, I played 12 bars before getting tangled up. After a fingering adjustment, I played 20-or-so bars before losing count. I don't see fatigue as being an issue, even in an extended passage. One minute would be 40 bars, which seems fine.
A professional pianist should have little or no trouble with the passage (though I can't rule out an eye roll).
Answer: As a sideways answer, consider one of my favorite pieces: the seventh movement of Ligeti's Musica ricercata. (See a faster performance here.)
In this movement, the left hand has a rather rapid ostinato that lasts for over three minutes (!). It's a little slower than yours: Ligeti's is basically septuplets at 88 beats a minute, yours is a twelve-let at 80 beats a minute.
Eight bars of yours is definitely doable, as is sixteen. And a minute sounds doable, albeit it with some training. Pianists might not be happy with you, but they're not always happy with Ligeti either :-)
Comment: How does that compare with the opening 3:30 of Tubular Bells? [Serious question, I don't read well so have trouble abstracting from score to 'feel']
Comment: It doesn't last as long, but this question made me think of the repeated note in the right hand of the second movement.
Answer: You should be perfectly fine. Most professional pianists have probably played other really hard pieces with very fast trills / runs, for example Chopin & Liszt Etudes and Rachmaninoff's Etudes-Tableaux / Concertos (Rachmaninoff Piano Concerto 3, 3rd movement) so they should be able to play this perfectly.
Only problem could be the pianist losing count, but that's unlikely
Answer: Like Aaron, I'm a skilled amateur, and had little difficulty picking up your passage. After experimenting with a couple of different fingerings, I could keep it up pretty much indefinitely at 80 with little to no fatigue. But hopefully, this is accompanying something a bit less repetitious, because otherwise, like Aaron, I would easily lose count of the repetitions.
To give you a bit of perspective, Schubert's Erlkönig has what is considered one of the most difficult piano accompaniments in the repertoire, with relentless repeated notes throughout. Schubert was quite happy to write it anyway, nobody "killed" him for doing it (in fact, Liszt wrote a piano-only arrangement of it, because of course he did), and it gets performed quite often. Here are two versions, one with the music and one by the great Dietrich Fischer-Dieskau that is one of my favorites musically (and, interestingly enough, transposed down a step from G minor to F minor).


Field extensions and algebraic elements

Question:
Can somebody explain why taking beta gives $K(\beta)$ as a subspace of $K(\alpha)$?
Comment: It's a subset, and it's a field, so it's a subfield, so it's a subspace.
Comment: Nothing in the question suggested that the fields would be finite, so I removed that tag. use43290, if that is a mistake, please add the fact that $L$ and $K$ are finite (sets/fields) to the listed assumptions.
Comment: Because it is generated by elements $\beta^i$ which are themselves polynomials in $\alpha$ ?
Answer: It follows from the definition of $K(\beta)$ being the smallest field that contains $\beta$:

If $\beta \in K(\alpha)$, then
$$K(\beta)= \bigcap_{\substack{F \ \ \text{field} \\ \beta\in F}}F\subseteq K(\alpha)$$

Field extensions and algebraic elements


How to find all broken links in a website?

Question: I can use external tools to scan for broken links in my website (Screaming Frog for instant check and even Google Search Console for non instant check come to mind) but I wish to use Drupal.
Is there a page that lists all the broken links?
Comment: Drupal reports all 403/404 when being accessed. To find all broken links you need to run an external tool that crawls your whole page. This is nothing specific to Drupal.
Comment: @leymannx there could be such a tool in Drupal just as there is one in MediaWiki. Yes, the programs are very different but I don't know much about upgrades in Drupal 9.
Comment: MediaWiki is no content management framework like Drupal where links could exist in literally a million places that are hard to guess.
Answer: Actual broken internal links (when a user clicks a link that is broken) are reported on the status report at <code>/admin/reports/dblog</code>.  You can filter by "page not found" to show all the bad links.
If you want to find external links and also identify broken internal links before your users do, you can use the Link Checker module.
That said, I think a better approach is to use one of the sitemap modules (Simple XML Sitemap or XML Sitemap) to list all the public content on your site and then point an external tool at the sitemap.
Comment: `/admin/reports/dblog` will present the link only if the link was clicked? Darn, I aspired for something that doesn't require that...
Comment: @drupalerdeepinside You can use the Link Checker module then.


Testing if two distributions have the same mean by using a sample distribution

Question: ORIGINAL Phrasing: Let's say I have a population distribution, A. For example, this could be the distribution of the maximum height (not the current height) that someone will reach during their life. Let's say A only contains data for people born in 1900 and 1901.
Now imagine a subpopulation of this distribution, B. This distribution may or may not have the same statistics as A - I don't know. For example, B might be all of the Males in A, which might have a higher mean. Or it might be all of the people born in 1900, which I would not expect to be very different from A in terms of statistics.
Now let's say I draw a random sample from A such that it contains only members of B. So for example either all males, or all people born in 1900. It's assumed that I don't know everyone's gender or birth year in A, so I can't actually construct B and look at it.
How can I use my sample to determine if B's statistics (e.g. mean) differ from A? It seems like standard C.I. / p-value approaches can work. But at the same time I'm not sure, because the random sample of B may by chance not be representative of B, especially if the sample is small. In other words, B itself may have its own (different) variance and mean than the sample. So for example, if I did have B, I could compute C.I. with its statistics to determine if its mean is the same as A, but the C.I. computed from the sample will be different than those computed from B itself. And then there's the fact that B itself is also a subset of A.
Am I overthinking things? Are standard C.I. / p-value approaches sufficient?
EDITED Phrasing:  Let's say I have a population distribution, A with size NA = 1000000.  For example, this could be the distribution of the maximum height (not the current height) that someone will reach during their life.  Let's say A only contains data for people born in 1890 through 1910.  We don't actually have the birth year for any person, but we know it is in that range.
In theory, there is therefore a subpopulation, B, with size NB, of all people in A born in 1890, but this population cannot be observed, and NB is not known.
Now someone comes along and labels NC = 1000 people as being born in 1890.  We'll consider these people sample C. He says that they were randomly selected from the NB people in B to be labeled (somehow, this person knew everyone who belonged in B, even though I don't).  So far these are the ONLY people whose birth years I know.
Can we determine if B's heights are different from the total population A using the sample C, and put confidence on that?  As a secondary followup, what if I can't assume that C is a random sample from B?  What if someone just gives me a bunch of 1890 labels but I'm not sure whether that was randomly done or not?
My concerns are that there's two types of error going on.  The error in B itself when comparing its mean to that of A, which I can't really observe, and the error in the fact that C, even if randomly sampled, can be a bad sample of B.
The other issue is that C (and B also) is part of A, so in comparing C to A, there's duplicate observations that belong to both distributions.
Am I overthinking things?  Are standard C.I. / p-value approaches sufficient?
Answer: Problem: Consider a population of $N$ people, with each person $i$ having a characteristic vector $(X_i, S_i)$ which consists of a characteristic of interest $X_i$ and a binary covariate $S_i \in \{ 0,1 \}$.  Let $N_* \equiv \sum_{i=1}^N S_i$ be the size of the subpopulation of people with $S_i=1$, and suppose that we are interested in comparing the population to the subpopulation.  Without loss of generality, we can order the indices so that the first $N_*$ people are in the subpopulation, and the remaining $N-N_*$ people are not in the subpopulation.$^\dagger$
The goal of the analysis is to compare the characteristics of the population and the subpopulation.  For definiteness, we will assume that we wish to compare the means of these groups.  The means of the population, subpopulation, and non-subpopulation, are denoted respectively by:
$$\bar{X}_N = \frac{1}{N} \sum_{i=1}^N X_i \quad \quad \quad \bar{X}_{*} =\frac{1}{N_*} \sum_{i=1}^{N_*} X_i \quad \quad \quad \bar{X}_{**} =\frac{1}{N-N_*} \sum_{i=N_*+1}^{N} X_i.$$
Now, if I understand your description of your sampling correctly, you are sampling from the whole population via simple-random-sampling without replacement (SRSWOR), so you will get people from the subpopulation and the non-subpopulation at random.  (This is a little ambiguous - at one point you say you sample from the population so that you only get values from the subpopulation, but at another point you say you can't identify the covariate until after sampling, which would mean that it is impossible to sample only from the subpopulation.)  You want to use this data to test whether the mean of the population differs from the mean of the subpopulation.  That is, you want to form a confidence interval or hypothesis test on the (unknown) value $\bar{X}_N - \bar{X}_{*} $.

Solution: If that is a correct interpretation of your sampling scheme then this problem can be handled using ordinary methods for comparison of two groups (the subpopulation and the non-subpopulation).  To see this, it we note that with a bit of algebra it have be shown that:
$$\bar{X}_N - \bar{X}_{*} = \frac{N-N_*}{N} \cdot (\bar{X}_{**} - \bar{X}_{*}).$$
Thus, comparing the mean of the population and subpopulation is equivalent to comparing of the mean of the subpopulation with the mean of the non-subpopulation.  (For a test of equality these are exactly the same, and for a confidence interval for the difference, you apply the scalar multiplier in the formula shown.)  Comparison of these two groups is the famous Behrens-Fisher problem which is a well-known problem in statistics.  The comparison is usually undertaken using a confidence interval or hypothesis test for the mean difference via Welch's approximation.

UPDATE: From the updated description of your sampling mechanism, it now appears that you are saying that you only have sample data from the subpopulation of interest, but you do not have any sample data from the remaining part of the population.  If that is the case then you can estimate the mean of the subpopulation, but there is no basis to estimate the difference in means between the subpopulation and the rest of the population.  If you would like to accomplish the latter then you will need some data (or some known summary statistics) from the remaining part of the population.

$^\dagger$ That is, we take $S_1 = ... = S_{N_*} = 1$ and $S_{N_*+1} = ... = S_{N} = 0$.
Comment: As an example of the sampling:
Imagine one main population of `all people`.  There's a theoretically well-defined subpopulation of `all males` that can't be empirically observed.  I know the gender of only a small fraction of `all males`, the `all known males` sample, and not of any of the other males or females.
Comment: So then the question is, in determining if `all males` mean is different than `all people`, should my confidence interval capture the fact that there's a chance that `all known males` could be a bad sample of `all males`, even if randomly sampled from `all males`? What's the best I can do to determine if `all males` is different than `all people`, given the sample of `all known males`?

My confusion is basically that there's two levels of sampling going on:  All to All Males (biased sample), and All Males to All Known Males (random sample).  And I'm not sure how to combine the error.
Comment: Sampling error under SRSWOR is already taken into account by standard confidence intervals for this problem, so the possibility of a bad sample is already built in.  Since you are sampling at random without knowing sex in advance, you should obtain a random sample of males and a random sample of females.  You can compare these using standard methods for comparison of means in two populations.
Comment: In this scenario, I don't know who is a female.  The only group I know is the comparison group.  E.g. the `all known males` group is made up of only males.  I don't know the group of any of the rest of the population.  It's more of a One vs. All comparison.  `all males` vs. `all people` comparison, using `all known males` as a proxy, to compare the mean of `all known males` to `all people`, in an attempt to conclude something about `all males`.
Comment: Okay, I'm really confused now.  I'm afraid I really can't follow your description of your sampling method; there are several things you are saying that seem contradictory to me.  Perhaps you could give an update on your question with a detailed account of exactly how you are getting your sample.
Comment: Will do, stay tuned
Comment: I updated the question
Comment: Let me know if this answer is correct with the clarified sampling method.  Would like to make sure the bounty goes somewhere :)
Comment: So with the update, it's still possible to compute statistics of the FULL population. So I can compute mean, STDV of the full population of all people, and then of the population of people who are labeled (the sample), but not of the subpopulation (all people who would fit the desired label).  I'll accept this since it seems to be closest to what I'm looking for, but I'm not sure it's entirely right either
Comment: If you already know the mean of the full population then that means there is no inference to be made there.  In that case, you just need to do a hypothesis test to compare the (unknown) mean of the subpopulation to the (known) mean of the population.  That would be a one-sample hypothesis test.
Comment: Right, that's what I'm trying to determine.  As I said, to me it seems like there's two potential issues:
1.  Error introduced due to the probability that the sample is not representative of the subpopulation
2.  The fact that most of the subpopulation is already part of the full population and cannot be separated out.

It sounds like you're saying these are not issues, and standard methods still work.  Ok!  I mostly thought this too, just wanted to make sure I wasn't missing something.
Comment: The second part should not be an issue.  If the population mean is known then it is known.  If you are testing for a difference between the mean of the subpopulation and the mean of the full population, and the latter is known, then it doesn't matter that the subpopulation forms part of the full population.
Answer: Well, here is one way. Males are approximately 4 inches taller than females. Male heights and female heights tend to be normally distributed. For your "A" population, fit the data with a mixture model consisting of two normal distributions. That will yield (1) a fraction of individuals with normally distributed shorter heights, which are likely to be female, and, (2) an another normal distribution with taller heights. Now the taller peoples' normal distribution with have a mean and standard deviation, that mean and standard deviation can then, for sufficiently large populations, be tested by Fisher's z test, against your B population of males. A finding of no significant difference should be associated with a confidence interval for the mean difference, which will give you some idea of how much uncertainty (e.g., how powerful) that negative result is.
BTW, to maintain independence, the "A" population should not have any sex related information per se.
For the second problem, 1900-1901 versus 1900 one needs the two data lists or it is intractable. If yes, we can eliminate the 1900 data from the 1900-1901 list to make two independent lists; one for 1900 and the other for 1901. Once the correction of the data is completed, if there is no subject overlap, the analysis itself could be done with either a CDF empirical distribution comparison or an explicit double mixture distribution independent male/female double comparison. However, the interpretation of the results depends on what the data is. When were the heights measured? At birth, now, at age 60?
Response to edit: One would use a find distribution routine for mixture models (similar to hidden Markov model) using a criterion like BIC for model selection. That will identify the number of distinct populations. The parameters of the selected model are then refined using a find distribution parameters routine. It does not matter how many "genders" one imagines to be present, the modelling will only return contributory results, which then have to be identified as particular sub-populations by other means. Mixture models will identify how many individuals are in each sub-population. It will not identify which data entries belong to which population in the region where the populations overlap, but it will yield how many there are in each population, and what the distribution parameters are for each sub-population.
Depending on what you are modelling, a normalization step, such as data ranking may be needed prior to other analyses. Also, two-step cluster analysis with k-means may be the starting point applicable depending on the exact circumstances of your modelling, and that may identify which data entries belong to which cluster.
Comment: @CHP It is not clear what you are asking. I can only answer what you do ask or the answer is "see all statistical methods," which would not be helpful. If you would add an edit to your question and ask a more precise question, including data types, distributions expected if known, and so forth, that would be more likely to elicit an answer that is more relevant to your needs. Not for naught the old expression is, "garbage in---garbage out."
Comment: The problem in the question is a toy problem that mirrors a more complicated real life problem, so the specifics (e.g. height measurement dates) don't come into play.  This solution is an interesting approach for the simplified version of the problem in the question, but in practice there may be like 20 groups or some other unknown number.  At any given point I want to do a 'One vs. All' type of comparison.  E.g. is Group 17 > All groups?  Then later, is Group 3948 > All groups?
Comment: I'll try to re-word the question.  But to give a closer example to the actual problem, I have a population of P = 100000 people.  Someone says "These K = 1000 people are all males, and there's M total males in the population of P, where M is >> K, but is unknown.  Tell me if the mean height of all of the M is > than the mean height of the population of P people as a whole, using the K people you know are males.  You don't know the gender of the P - K group, and there could be an unknown number of genders, not just two."
Comment: @CHP That does not change the form of my answer. One would use a find distribution routine for mixture models (similar to hidden [Markov model](https://en.wikipedia.org/wiki/Hidden_Markov_model)) using a criterion like BIC for model selection. That will identify the number of distinct populations. It does not matter how many "genders" one imagines to be present, the modelling will only return contributory results, which then have to be identified as particular sub-populations by other means. Mixture models will identify how many individuals are in each sub-population.
Comment: @CHP con't. It will not identify which data entries belong to which population in the region where the populations overlap, but it will yield how many there are in each population, and what the distribution parameters are for each sub-population.
Comment: @CHP Also, [two-step cluster analysis](http://www.statisticssolutions.com/cluster-analysis-2/) with k-means may be the starting point applicable depending on the exact circumstances of your modelling, and that may identify which data entries belong to which cluster. Depending on what you are modelling, a normalization step, such as data ranking may be needed prior to other analyses.
Answer: I think the way you're thinking about this is a bit confused, let's analyse a few of the things you say:

Now let's say I draw a random sample from A such that it contains only members of B

If you draw a truly random sample from A, then given you know how A is distributed, you know the true distribution of the mean of the random sample. If you draw a sample from A such that it contains only members of B, and B turns out to be a biased sample of A, then it's usually pretty unlikely you were genuinely sampling at random from A.
This is a classic problem in, for example, election polling. Pollsters try to sample from the voting population (voting distribution A), but reach a biased subset (e.g. digitally and politically engaged urban voters, or B). They make inferences about the general population based on their sample assuming random error, but their sampling method means they were not randomly sampling in the first place so they get their inference wrong.
I think what you might be trying to ask, or at least a (albeit perhaps not the only) sensible question to ask, might be something like the following:

I draw a sample from A, and I want to know whether my sampling method is truly random, or whether I've obtained a biased subset.

(note that if B is sampled from A, but you find that "B's mean differs from A", this is equivalent to saying that the sample was not truly random in the first place).
One way of doing this is by calculating the p-value of obtaining the sample mean you obtained. You do need to know the details of the population distribution A you were sampling from, it isn't clear to me either from your question whether this is known or not.
Likewise, the Bayesian method would be to assume an underlying distribution for B. If you know that A is normally distributed with known mean and variance, you could assume that B is also normally distributed with the same variance, assume a prior which is centred around the mean of A, and do an inference on the mean of B using Bayes Theorem. Then you could calculate the confidence you have that the mean of B is within $\pm x$ of the mean of A. If your confidence is high that the mean of B within $\pm $ of the mean of A, where x is pretty small, you can conclude that your sampling method is likely to be "not very biased". If your sample sizes are small, then this method becomes very sensitive to your prior, something some people find dis-satisfactory about Bayesian methods.
Comment: That's close.  What I'm trying to ask is, is B truly different than A. I KNOW the sampling method is biased relative to A, but we can assume it's random relative to B.  So the members I sample from B are randomly sampled from B, but B itself may or may not turn out to be like A.

In my example, what I would be trying to determine is:

"Are the population of M males actually taller than the N person population as a whole, assuming I ONLY know that these K people are males, where K << M, and I don't know any of the other N - K people's gender (some of whom will be male, some female)"
Comment: Even though the sampling from B is random, there's some chance that the statistics won't reflect B.  And B itself has some uncertainty when comparing it to A even if you had all members of B.  So in essence I'm trying to understand how these combine - if they do, to determine my confidence.


Sum of the equation in combinatorics

Question: Do help to solve these 2 questions
$\
{n \choose 0}^2 + {n \choose 1}^2 +{n \choose 2}^2 +...+{n \choose (n-1)/2  }^2= ? $ when n is odd
$\
{n \choose 0}^2 + {n \choose 1}^2 +{n \choose 2}^2 +...+{n \choose n }^2= ? $
$\
{n \choose 0}^2 + {n \choose 1}^2 +{n \choose 2}^2 +...+{n \choose n/2  }^2= ? $ when n is odd
Comment: use ${n \choose i}^2 = {n \choose i}{n \choose n-i}$
Comment: For the second one, you could consider the coefficient of $x^n$ on the LHS and RHS of $(1 + x)^{2n} = (1 + x)^n(1 + x)^n$ separately.
Answer: For the second one, note that consider the binomial expansion of $(1 + x)^n$, then we can collect the terms of $x^n$ in $(1 + x)^n(1+x)^n$ to get:
$$\sum_{i = 0}^n \binom{n}{i}\cdot\binom{n}{n - i} = \sum_{i = 1}^n \binom{n}{i}\binom{n}{i}=\sum_{i = 0}^n\binom{n}{i}^2$$
which is exactly what we need. On the other hand, $(1 + x)^n(1 + x)^n = (1 + x)^{2n}$ so the coefficient of $x^n$ on both sides should be equal. That is to say,
$$\sum_{i = 1}^n\binom{n}{i}^2 = \binom{2n}{n}$$
For the first one, you could use the same trick, just collect a different term instead of $x^n$.

Sum of the equation in combinatorics


mysql++ problem with linking

Question: I have problem, I use mysql++ 3.1.0:
after building mysql++.sln in debug and release mode for vc2008, and using instal.hta then
adding dirrectoires in vc++ directories to lib and include folders,
and adding in project properties-> Linker-> General Additional Library Directories directory to lib folder which have mysqlpp.dll, mydqlpp.lib, mysqlpp_d.dll, mydqlpp_d.lib.
These code:
<code>#include <iostream>
#include <mysql++.h>
using namespace std;
using namespace mysqlpp;

int main()
{
    Connection con;

    return 0;
}
</code>
gets these errors:
<code>error LNK2019: unresolved external symbol "__declspec(dllimport) public: virtual __thiscall mysqlpp::Connection::~Connection(void)" (__imp_??1Connection@mysqlpp@@UAE@XZ) referenced in function _main
error LNK2019: unresolved external symbol "__declspec(dllimport) public: __thiscall mysqlpp::Connection::Connection(bool)" (__imp_??0Connection@mysqlpp@@QAE@_N@Z) referenced in function _main
</code>
in visual studio 2008. How can I fix this?
Answer: I think you missed the last step here, which adds several libraries to your project: the MySQL++ DLL import library, the MySQL C API library, and the Winsock DLL import library.
Comment: Excuse me, my English isn't very good, so I don't understand you well.If you are speaking about adding mysql server libraries to my project  I did it. Or I must add something else to my project?
Comment: That's just one of the three you need. You need `libmysql.lib`, `wsock32.lib`, *and* `mysqlpp.lib`. You have probably left out the last one at least.
Comment: Yes, you do. It's part of the Windows SDK, which comes with Visual C++. Just add it to the library list.

mysql++ problem with linking


Video Call in Ionic and Overlay Screen In Ionic Application

Question: How can I integrate Mesibo Video APIs in Ionic? Gone through the docs but couldn't see anything IONIC specific.
Is there any IONIC Demo Application ?
Is there any other way to use Mesibo Android sdk in Ionic application?
I want to implement video call overlay screen concept where 'while one to one video call in place and user can work on all the application pages in parallel'. Please let know if this feature can be implemented in Ionic application using Mesibo APIs.
Answer: mesibo provides native APIs for Android (Java & Kotlin), iOS (obj-c, Swift), C++, Javascript, and Python. You need to write a plugin if you like to using mesibo (or any native android library) from ionic
https://cordova.apache.org/docs/en/latest/guide/platforms/android/plugin.html
Answer: you can try cordova-call, it works background and foreground quite well


How to find out who is sending email

Question: I have a server which doesn't have much activity. Today, suddenly server load went higher than 12 which compared to its normal value(0.5) is way too high. I checked and found out that postfix is being used by a program named smpt  to send emails. Its full arguments:
 smtp -t unix -u

I use CentOS release 6.7 , I update my server on regular basis. netstat  shows me something like this:

I also checked /var/log/maillog  which told me in last 10 hours hundreds of thousands of email has been sent.
How can I find out who is behind this? How can I find out if it's internal or external?
P.S: postfix is down.
Comment: Possible duplicate of [Fighting Spam - What can I do as an: Email Administrator, Domain Owner, or User?](http://serverfault.com/questions/419407/fighting-spam-what-can-i-do-as-an-email-administrator-domain-owner-or-user)
Comment: Likely the malware which broke into your server, most likely via a security issue in a web application, is sending it. This is but one of many reasons you should not disable SELinux. For now, you'll have to find and clean the malware, and most likely blow the server away and reinstall it.
Comment: At least post some logs.
Answer: To find out how this is happening, you need to look at postfix logs. Either your postfix is misconfigured or you have some authorized host that is compromised (usually a webserver). If the postfix logs say this is coming from your local host, an lsof -i -n can yield useful information.
Answer: Most probably you have open relay enabled and some bot/scanner has picked you up. See your postfix config file (/etc/postfix/main.cf) and look for something like:

mynetworks = 0.0.0.0/0

If it is the case here, you will have to decide which networks are allowed to relay mail, or even disable it altogether leaving just the localhost.

How to find out who is sending email


Proving a Mapping Preserves in Set Unions

Question: I need to prove that $f(A_0 \cup A_1) = f(A_0) \cup f(A_1)$ where $f: A \to B$ is a mapping of sets and $A_0,A_1 \subseteq A$.
So, I saw this question and figured out the first direction ($f(A_0 \cup A_1) \subseteq f(A_0) \cup f(A_1)$). I got a little stuck in the other direction and I wanted to know if I worded my proof logically:
Let $y \in f(A_0 \cup A_1)$ be arbitrary. Then $y \in f(A_0 \cup A_1)
= \{f(a) | a \in A_0 \cup A_1\}$ means that $y = f(x)$ for some $x \in A_0 \cup A_1$. By the definition of set union $x \in A_0 \cup A_1$ means $x \in A_0$ or $x \in A_1$. Recall $y = f(x)$ and suppose $x\in A_0$, it follows that $y \in \{f(a) | a \in A_0\} = f(A_0)$. Now, suppose instead that $x \in A_1$ and $y = f(x)$, it follows that $y \in \{f(a) | a\in A_1\} = f(A_1)$. Thus, in either case, $y \in f(A_0) \cup f(A_1)$. Since $y \in f(A_0 \cup A_1)$ and we have just shown that $y \in f(A_0) \cup f(A_1)$ then $f(A_0 \cup A_1) \subseteq f(A_0) \cup f(A_1)$.
Let $y \in f(A_0) \cup f(A_1)$ be arbitrary. Then $y \in f(A_0) = \{f(a)|a \in A_0\}$ or $y \in f(A_1) = \{f(a)|a \in A_1\}$. So $y \in f(A_0)$ means $y = f(x)$ for some $x \in A_0$ or $y = f(x)$ for some $x \in A_1$. By the definition of set union $x \in A_0 \cup A_1$, thus $y \in f(A_0 \cup A_1)$. Therefore, $f(A_0 \cup A_1) = f(A_0) \cup f(A_1)$.
I think I'm missing the connection between defining the set union and getting to the desired result.
Answer: This looks fine to me except for one point in your proof for the other inclusion, namely:

So $y \in f(A_0)$ means $y = f(x)$ for some $x \in A_0$ or $y = f(x)$ for some $x \in A_1$.

$y \in f(A_0)$ means $y = f(x)$ for some $x \in A_0$, but not that $y = f(x)$ for some $x \in A_1$ (unless $A_0 \subseteq A_1$). What you want to say is something like:

$y \in f(A_0)$ means $y = f(x)$ for some $x \in A_0$, and $y \in f(A_1)$ means $y = f(x)$ for some $x \in A_1$. In either case, since $y \in f(A_0) \cup f(A_1)$ we have that $y = f(x)$ for some $x \in A_0$ or for some $x \in A_1$, so by definition of set union $x \in A_0 \cup A_1$.


Reshaping Wide to Long

Question: My data is currently organized in Stata as follows:
iso_code    indicator                                                    value
ALB         Share of employees in agriculture (% of total employees)    2.52084
ARG         Employment in agriculture                                    11.153
ARG         Share of employees in agriculture (% of total employees)    .104509
AUS         Employment in agriculture                                    329.66
AUS         Share of employees in agriculture (% of total employees)     1.47018
AUT         Employment in agriculture                                    159.549
AUT         Employment-to-population ratio, rural areas                  60.4239
AUT         Labour force participation rate, rural areas                 61.9622
AUT         Share of employees in agriculture (% of total employees)    .838354
AZE         Employment in agriculture                                    1769.3

And I would like each country to be in one row, and then the variable names and their values as separate columns.
iso_code      Employment in agriculture   Share of employees in agriculture (% of total employees)
AUT             159.549                        .838354

I have tried the following code, but I keep getting this error: "indicator.1720000058412552 invalid variable name"
reshape wide indicator value, i(iso_code)  j(value)

Answer: Solved the problem with the following code:
// Converting the "indicator" variable into numeric to reshape the dataset
encode indicator, gen(indicator_numeric)

// Reshape long to wide
reshape wide indicator value, i(iso_code)  j(indicator_numeric)

Reshaping Wide to Long


The proper right-hand picking technique for playing the electric guitar

Question: When I was taking electric guitar lessons for a year I was told, by my former teacher, that the pick should be parallel to strings at all times and so he insisted on it.
The other thing he was very strict about was the "amount" of the pick crossing the strings. Only the very tip of the pick should be hitting the strings (only one point ideally). I am finding this task very hard if not impossible.
After a year of doing nothing but practicing right hand (without significant progress) I felt so discouraged that I stopped attending the lessons. Here I would like to turn to you, experienced guitarists, how much of a emphasis should be put on the subjects above. Are these requirements even right?
Thank you very much for your input. I am loving playing guitar, but I would rather spend my time practicing songs, scales, etc. than this.
Comment: I hope the lessons were free!
Comment: This all feels a bit… Victorian to me; like putting a really rough plank above a piano keyboard to "aid technique".
Comment: So your teacher told you to only practice picking without playing any songs or anything for a **year**?? That's kinda criminal.
Comment: Playing guitar using a pick is one way. No-one here has considered mentioning that it's entirely possible with NO pick. Many superb guitarists - in many different genres - play that way.
Comment: @Tim Unfortunately weren't. And since I was a student I worked hard to be able to afford them.
Comment: @ToddWilcox I did some basic scales, but I think there were considered just "picking patterns".
Comment: I don't think we're getting the whole story.  What book if any were you learning from?  How do you know you were not making progress?
Comment: The distance from the bridge is something else to consider when picking. The feel and control of the string, not to mention the sound, changes dramatically when moving away from the bridge. What works well close to the bridge may not work well for you away from the bridge.
Answer: Could I be bold enough to state there is no one 'proper right hand technique ' to playing guitar. If there was, everyone would use it. There are many right hand techniques, and good players will swap between them as they play.
Sorry to say, your teacher believed one technique would do for all - and he was sadly wrong. That's it.
Answer: I would agree with your instructor to some degree.  Only the very tip should be allowed to touch the strings, you don't need a lot.  However the term "parallel" is ambiguous without a diagram.  A lot of shredders will talk ad nauseam about the 45 deg angle, blah blah blah.  This is kind of BS too.  I hold the pick as close to "parallel as I can" and by that I mean that the plane of the pick is "parallel" to the string I am picking, perpendicular to some degree to the plane made by the strings.  I find that I can achieve incredible speed, accuracy, and cleanliness with this.  However I caveat: (1) The natural movement of the hand + wrist + arm will necessarily cause the pick to attack the string at a very slight angle, and (2) many picks (specifically the ones I use) are sculpted or taper down.  This means that you CANNOT possibly attack the string dead on with the flat part of the pick if you tried.
I find the shredder mechanics total crap and the traditional method of picking quite superior when mastered.  From the point of view of learning it is typically a better investment to work at the correct technique and posture as a beginner rather than develop bad habits that lead to crappy playing later that are hard to reverse.  It is possible that your instructor was just trying to get you on the right path and train your body to adapt to correct posture.  Guitar is a hard instrument to master and short cuts don't lead to mastery.
Answer: Using as little of the pick as possible is very important but you should expect to take years developing your pick technique.
Being able to strum with the pick exactly parallel to the strings is also important, because there are situations where that is the way to get the best tone. Again, it took me several years to develop my strumming to this point. So for both of these goals, keep them in mind, but be patient with them.
That said, it is important to learn how to pick and strum as above, but there are many situations where one or both of those points is not appropriate. The thing is, picking and strumming parallel and with as little of the pick as possible is both pretty difficult and also should be sort of the default way that you pick. When you do it differently, it should be a choice. So over time it will give you the best tone and timing to try to make the picking technique automatic and natural. I think that's what your teacher was going for and maybe wasn't clear about it and perhaps imparted some unreasonable expectations.
It can get very frustrating learning an instrument. It sounds like to me your teacher was too strict. These techniques are best seen as goals and not requirements. There are plenty of famous musicians who play guitar in front of thousands who have terrible pick technique. But none that I can think of who are famous because of their guitar playing, more often their songwriting and/or singing. I can assure you most pro guitarists have mastery over fractions of a millimeter of positioning of their picks at all times. If you want to play fast, or even if you want to strum folk chords with excellent tone, if you want to develop ergonomics that will keep your hands and arms healthy for a lifetime of playing guitar, I suggest you keep technique in mind, but don't feel ashamed or stressed about your technique not yet being where you or anyone else thinks it should be. It's a journey, not a destination.
Comment: @PepaZdepa You're gonna develop some bad habits and have to work to correct them. That's ok because that's what happens to every single guitarist ever. Hopefully you're playing guitar because it's fun. Don't let anyone else make it not fun, and be on the look out for yourself making it not fun! Just do your best to learn it right the first time but nobody can learn it perfectly so you'll have some setbacks. We all do.
Comment: Thank you very much for encouraging words. But how shouldn't I be stressed about my technique, when there is a chance of developing bad habits, which are generally very hard to break once embraced?
Comment: This is what I needed to hear. Thank you very much again! After a while I feel motivated.
Comment: Interesting thoughts. Never really given it a lot of consideration, but playing today, looking at what happens, I find that my pick has a variety of different positions - from tight in to halfway out when strumming. It appears it's on the move quite a lot. Wouldn't consider it a bad habit, or a terrible pick technique though.
Comment: It depends on what you're going for. I've choked up tightly with lots of thumb for false harmonica. I've hit with the tip and played fast. I've set the end against them pickguard and tried to strum without lifting.
Comment: @DaveJacoby - that's my point. There is no one 'correct' way.
Comment: @Tim If my answer gives the impression that I'm saying there is one way to pick then that is not what I'm trying to say and maybe I should edit it. What I strongly believe is that different pick techniques work best in different situations, and the picking technique the OP's teacher recommended is a valid one that is difficult to learn but should definitely be learned so it is available when it suits the music. Other techniques for other situations are easier, in my experience. And just because there are multiple good techniques doesn't mean there are no bad ones. There is bad picking.
Comment: @Tim I think it's interesting that it sounds like you are sort of seeing how you pick instead of deciding how you pick.  I'm recommending that guitarists learn to decide exactly how they pick for each stroke and when they've made that decision they know how to pick that way because they've already spent the time to learn how to pick that way. My experience is that learning to pick as the OP's teacher suggested opens up a wide variety of *intentional* pick techniques that can be chosen and implemented effectively. I do object with the teacher's methods while agreeing with the goal.
Comment: @ToddWilcox - what I seem to have found out is that there are various different scenarios, and I use the pick differently for different situations - changing on the fly, so not actually aware of how I do hold it. I guess after 60 oddd yrs of playing all sorts, it's developed a life of its own. But at some stage if something wasn't productive, would I have continued that way?
Answer: There's Troy Grady, who created Cracking the Code to determine the way fast players in many genres pick, so that all players can learn to play as fast as their music requires. He goes way in depth with details I cannot remember, so search him out.
Being said, I should add content. I generally angle the pick forward about 20°, in part for comfort, in part because the pick slides over the string more smoothly and in part to avoid pick noise. There's a higher-end slap when you hit squarely that displeases me. Or perhaps there was something else I noticed 30 years ago that I've forgotten. I've seen players who angle similarly but back, and their thumbs look weird to me, like hands should not bend like that, but it works for them.
There are mistakes in playing that can become ingrained, where they damage your body but are hard to unlearn, but to me, that's about keeping your hand loose, not tense. Even if picking square with the string was good technique, which I don't think it is, spending hours to perfect it before getting to the parts that make guitar fun to play is madness.
So, play. Be attentive to tension in the wrist and fingers; if they hurt or go numb over time, you're doing it wrong. Listen and find a picking angle that works for you and sounds good to you. Watch Troy Grady videos for further ideas. Playing scales and learning songs doubles as picking practice, so you can move forward with art as you develop craft.
And shun your former teacher.
Answer: Please note that the the technique affects not only speed and efficiency, but also the sound. For instance Devin Townsend mentions angling the pick to get more scratching attack sound which he finds appropriate in some situations. So, while your teacher might have a reason to suggest you parallel alignment, I wouldn't stick to it zealously.
Answer: Agree with everybody above, and probably beating a dead horse, but that's some of the WORST advice!  Whatever feels comfortable for you!    It just slowly comes in time…some players caress the guitar, others attack it, so find YOU.  I learned early on, just how important the right and left hand are independently, and as a unit.  Left hand did all the heavy lifting, and the right hand was just having fun!  Righty's your rhythm, your drum, muter, silencer and so much more.  Slightly mute all of the strings with your left hand around the 5th fret, grab a pick, and just practice natural up and down motions. Don't worry about playing any notes, just get a good, natural groove going.  You'll probably drop the pick a few hundred times, buts that ok.  I STILL drop mine, after 30+ of playing!  You'll slowly feel the amount of pressure to apply, holding the pick, and striking the strings.  Good fundamentals are important, but when you're playing, you have to be relaxed.  As far as what your teacher told you, just disregard all of that silliness-


Convergence of this Doob Martingale Sequence

Question: Let $U$ be a Uniform$[0,1]$ variate and conditioned on $U$, let $\{X_{n}\}_{n\geq 1}$ be iid $\operatorname{Bern}(U)$ variates. Then  show that $E(U|\sigma(X_{1},...,X_{n}))\xrightarrow{a.s.} U$
My attempt(s): Firstly, I can immediately notice that $Y_{n}=E(U|\sigma(X_{1},...,X_{n}))$ is a Doob Martingale sequence that is uniformly bounded (by $1$) in $L^{\infty}$ . So, by Martingale convergence theorem $Y_{n}\xrightarrow{a.s\, ,\, L^{p},\,p<\infty} X $ for some $X\in L^{p}$ for all $p<\infty$. But I don't know how to show that $X$ is equal to $U$.
I also tried to use the method of moments as $U$ is compactly supported. I tried to show that $E(U^{m})=E(E(U^{m}|X_{1},...,X_{n}))\to E(X^{m})$ but the issue is that I need to consider $\bigg(E(U|X_{1},...,X_{n})\bigg)^{m}$ instead of $E(U^{m})$ and that we only have   $\bigg(E(U|X_{1},...,X_{n})\bigg)^{m}\xrightarrow{L^{m}} X^{m}$. This does not give us that $X^{m}$ has the same moments as $U^{m}$.
I also know from the Convergence Theory that $Y_{n}=E(X|X_{1},...,X_{n})$ which would mean that $E(X|X_{1},...,X_{n})=E(U|X_{1},...,X_{n})$. From this can we conclude that $X=U$ as we have that $\int_{A} (X-U)\,dP=0$ for all $A\in \sigma(X_{1},...,X_{n})$. I am very unsure about this as I am not using any property of  $X_{1},...,X_{n}$ let alone the fact that conditioned on $U$, they are iid Bernoulli$(U)$ variates.
Since $X_{n}$ are iid, I tried to think of using Kolmogorov's Zero-One law but I couldn't see how that could help.
Can anyone provide a hint or help me with this?
Comment: $\int_{A} (X-U)\,dP=0$ for all $A\in \sigma(X_{1},...,X_{n})$ implies $X=U$ a.s. only when $X-U$ is measurable w.r.t. $ \sigma(X_{1},...,X_{n})$. We certainly don't have this.
Comment: @geetha290krm You are correct. I forgot about the measurability issue.
Comment: If you're familiar with estimation theory: $\mathbb{E}[U|\sigma(X_1^n)]$ is the optimiser of $\min \mathbb{E}[ (U - \tau)^2]$ over $\sigma(X_1^n)$ measurable $\tau$ (i.e. estimates of $U$). But its well known that the mean-square optimal Bernoulli parameter estimate is just $\sum X_i/n$, which converges to $U$ by the SLLN. Some $\omega$ chasing is probably necessary to make this precise (you need the SLLN over only $\omega : U(\omega) = u$).
Comment: @stochasticboy321 Sorry I have not had a course in statistics. But I appreciate your comment and I think it will be helpful to other users who will come across this post.
Answer: The $\sigma$-algebra generated by $X_1,\dots,X_n$ is generated by a partition
hence in order to compute $\mathbb E\left[U\mid\sigma(X_1,\dots,X_n)\right]$, it suffices to determine for each $\delta_1,\dots,\delta_n\in\{0,1\}$, the quantities
$$\tag{(*)}
\mathbb E\left[U\mathbb{1}_{(X_1,\dots,X_n)=(\delta_1,\dots,\delta_n)}\right]\mbox{ and }\mathbb P\left((X_1,\dots,X_n)=(\delta_1,\dots,\delta_n)\right).
$$
To do so, we use conditioning with respect to $U$ and the assumption, the result will be expressed in terms of $\ell(\delta)=\sum_{i=1}^n\delta_i$ and $n$. By assumption
$$
\mathbb E\left[\mathbb{1}_{(X_1,\dots,X_n)=(\delta_1,\dots,\delta_n)}\mid U\right]=U^{\ell(\delta)}(1-U)^{n-\ell(\delta)}.
$$
Integrate this to get the second part of (*). For the first one,
multiply by $U$, put $U$ inside the conditional expectation and take the expectation.
You will get that
$$
\mathbb E\left[U\mid\sigma(X_1,\dots,X_n)\right]=\sum_{(\delta_i)_{i=1}^n\in\{0,1\}^n} c_{n,(\delta_i)_{i=1}^n}\mathbf{1}_{(X_i)_{i=1}^n=(\delta_i)_{i=1}^n}.
$$
In order to show that $\mathbb E\left[U\mid\sigma(X_1,\dots,X_n)\right]$ converges to $U$ almost surely, we use the fact that  if $\mathcal G$ is a $\sigma$-algebra and $U$ and $\mathbb E[U\mid\mathcal G]$ has the same distribution, then $U=\mathbb E[U\mid\mathcal G]$ almost surely.
Once the distribution of $\mathbb E\left[U\mid\sigma(X_1,\dots,X_n)\right]$ is determined, it suffices to check that it converges in distribution to a uniform random variable.
Comment: Thanks for your answer (+1) .Can you please elaborate on the second paragraph in your answer "To do so, we use conditioning with respect to $U$ and the assumption, the result will be expressed in terms of $\ell(\delta)=\sum_{i=1}^n\delta_i$ and $n$. " . I could not understand it. Also, I tried to prove convergence in distribution in my attempt using method of moments. So it is not clear to me how the terms $\ell(\delta)$ will help me in figuring out the distribution.
Comment: Additionally, can you also tell me why equality of distribution would imply equality almost surely? It is clear that $U$ and $E(U|\mathcal{G})$ are defined in the same probability space. If I am not wrong, $U$ will have to be $\mathcal{G}$ measurable or atleast measurable wrt the completion of $\mathcal{G}$ (i.e. G with the null sets). In that case, I can see that $U$ must equal $E(U|\mathcal{G})$. I hope I am making sense here.
Comment: @Dovahkiin I have added a bit more details and also a link.
Comment: Thanks. Much clearer now.
Comment: Can you explain to me how to show that $\displaystyle\sum_{\delta\in \{0,1\}^{n}}f(c_{n,\delta})\dfrac{\Gamma(\ell(\delta)+1)\Gamma(n-\ell(\delta)+1)}{\Gamma(n+2)}\to \int_{0}^{1}f(x)\,dx$ where $c_{n,\delta}=\dfrac{\Gamma(\ell(\delta)+2)\Gamma(n-\ell(\delta)+1)}{\Gamma(n+3)}$ keeping with your notation ? Is this a real analytic result? I think this result must be true. (f is a continuous bounded function)
Comment: You can group the $\delta$ which have the same $\ell(\delta)$, which will give a sum over a single index, say $j$. The binomial coefficient which will appear multiplied by the quotient of gamma's will give something simpler. At some point, I believed that we would get something like a Riemann sum but it does seem so simple. Maybe an alternative would be to look at the cumulative distribution function.
Comment: I thought that looking at the CDF would be more difficult that proving the convergence of a Riemann like sum. That's why I went forward with proving convergence of expectations for continuous bounded functions. I'll have to think more in terms of the cdf. Please update if you find a way out for the limit. I'll also update you in the comments. Anyways, a huge thanks for the help so far. I know that you are the go to expert in this site when it comes to Probability Theory.
Comment: An idea could be to use the fact that $(c_{n,\ell(\delta)})_{\ell(\delta)\leqslant\lfloor (n-1)/2\rfloor}$ is increasing and we can approximate $f(c_{n,\ell(\delta)})$ by something times the integral of $f$ over $[c_{n,\ell(\delta)},c_{n,\ell(\delta)}+1]$. For the $\delta$ such that $\ell(\delta)>\lfloor (n-1)/2\rfloor$, we use this time decreasingness.


Drush not selecting database

Question: I'm using a Vagrant environment generated from https://puphpet.com/ for a Drupal project & it is all going well except for Drush which gives me an error any time I try running "drush cc -all", see the error below.

Additional uncaught exception thrown while handling exception.

Original

PDOException: SQLSTATE[3D000]: Invalid catalog name: 1046 No database selected: SELECT 1 FROM {blocked_ips} WHERE ip = :ip; Array
(
[:ip] =&gt; 127.0.0.1
)
in drupal_is_denied() (line 1933 of /var/www/includes/bootstrap.inc).

Additional

PDOException: SQLSTATE[3D000]: Invalid catalog name: 1046 No database selected: SELECT * FROM {system} WHERE type = &#039;theme&#039; OR (type = &#039;module&#039; AND status = 1) ORDER BY weight ASC, name ASC; Array
(
)
in system_list() (line 165 of /var/www/includes/module.inc).

Drush command terminated abnormally due to an unrecoverable error.                                                                                [error]

From what I gather the problem is that it isn't selecting a database. Does anyone know where the details for this would be stored? The Drupal site itself is connecting & using the database fine.
I'm not sure if this would cause and issue but the database details for the Drupal site are stored as an Apache environmental variable.
Comment: From which folder are you running the drush command? Try running it with `drush -r /var/www/html status` (or `-r /var/www` in your case) and see if that gives you anything meaning ful.
Comment: Another thought, who owns the files? That is what user owns the files. Drush needs to be able to read settings.php
Comment: I'm running the command from /var/www & the output I get from that command is the same as I when I run Drush cc -all.
Comment: When I run ls -l in /var/www/sites/default the line for settings.php reads -rwxrwxr-- 1 www-data www-data 26633 Jun  9 19:43 settings.php*
Answer: What you are seeing above is a bug in Drush where Drush cannot bootstrap the database of a Drupal site if the database credentials are correct, but the database itself is empty.  To fix this problem:

If your database is empty, run "drush site-install" to create a database.
If your database is not empty, check the settings of your database prefix, and make sure that it is correct.

This was recently fixed in https://github.com/drush-ops/drush/pull/1442.
Comment: Thanks for the answer but the database isn't empty, unless you meant a specific table? When I visit the site in the browser displays all the content from the database correctly so the connection details must be correct. settings.php is pulling the connection details from an Apache environmental variable, could that cause any problems? The site works fine when accessing it through the browser, it's just Drush that doesn't.
Comment: Is the output the same if you run drush from var/www/sites/default ? B/c drush can't bootstrap correctly from just var/www b/c it does not know anything about the site yet.
Comment: @burnsjeremy I didn't realise that, although I've just tried it & it is still giving me the same error. I've also tried running it with sudo but it didn't have any affect.

Drush not selecting database


State space model estimation

Question: I would like to estimate this system with state space modeling in order to perform some initial tests as an empirical analysis.
$
\begin{cases}
x_t = \mu_t + \beta_1x_t + \varepsilon \\
\mu_t = c_{t-1} + \gamma \mu_{t-1} + \epsilon_{1,t}\\
c_{t} =  \rho c_{t-1} + \epsilon_{2,t}
\end{cases}
$
Do you know if there is a standard toolbox (R or Matlab) that allows me to express the two states equation with the autoregressive component?
I've already checked some toolboxes but I wasn't able to express the states equation in this way.
Comment: There are important details missing. What variables are observed and which are not? Are you interested in estimating the parameters or the latent states (i.e. are the parameters known)? Are the error terms correlated?
Comment: Hi Hejseb, I'm sorry you are right. There is also a typo on the dependent (it's obviously y and not x). The observed are y and x, the other are latent while the error terms are supposed to be uncorrelated.
Answer: Have you considered Matlab's Systems identification toolbox? If the standard state space modeling functions don't cut it, perhaps the grey box modeling functions will.

State space model estimation


For basic maths calculations on very large csv files how can I do this faster when I have mixed datatypes in my csv - with python

Question: I have some very large CSV files (+15Gb) that contain 4 initial rows of meta data / header info and then the data. The first 3 columns are 3D Cartesian coordinates and are the values I need to change with basic maths operations. e.g. Add, subtract, multiple, divide. I need to do this on mass to each of the coordinate columns. The first 3 columns are float type values
The rest of the columns in the CSV could be of any type, e.g. string, int, etc....
I currently use a script where I can read in each row of the csv and make the modification, then write to a new file and it seems to work fine. But the problem is it takes days on a large file. The machine I'm running on has plenty of memory (120Gb), but mu current method doesn't utilise that.
I know I can update a column on mass using a numpy 2D array if I skip the 4 metadata rows.
e.g
arr = np.genfromtxt(input_file_path, delimiter=',', skip_header=4)
arr[:,0]=np.add(arr[:,0],300)

this will update the first column by adding 300 to each value. But the issue I have with trying to use numpy is

Numpy arrays don't support mixed data types for the rest of the columns that will be imported (I don't know what the other columns will hold so I can't use structured arrays - or rather i want it to be a universal tool so I don't have to know what they will hold)
I can export the numpy array to csv (providing it's not mixed types) and just using regular text functions I can create a separate CSV for the 4 rows of metadata, but then I need to somehow concatenate them and I don't want to have read through all the lines of the data csv just to append it to the bottom of the metadata csv.

I know if I can make this work with Numpy it will greatly increase the speed by utilizing the machine's large amount of memory, by holding the entire csv in memory while I do operations. I've never used pandas but would also consider using it for a solution. I've had a bit of a look into pandas thinking I maybe able to do it with dataframes but I still need to figure out how to have 4 rows as my column header instead of one and additionally I haven't seen a way to apply a mass update to the whole column (like I can with numpy) without using a python loop - not sure if that would make it slow or not if it's already in memory.

The metadata can be empty for rows 2,3,4 but in most cases row 4 will have the data type recorded. There could be up to 200 data columns in addition to the initial 3 coordinate columns.
My current (slow) code looks like this:
import os
import subprocess
import csv
import numpy as np

def move_txt_coords_to(move_by_coords, input_file_path, output_file_path):

    # create new empty output file
    open(output_file_path, 'a').close()

    with open(input_file_path, newline='') as f:
        reader = csv.reader(f)
        for idx, row in enumerate(reader):
            if idx < 4:
                append_row(output_file_path, row)
            else:
                new_x = round(float(row[0]) + move_by_coords['x'], 3)
                new_y = round(float(row[1]) + move_by_coords['y'], 3)
                new_z = round(float(row[2]) + move_by_coords['z'], 3)
                row[0] = new_x
                row[1] = new_y
                row[2] = new_z
                append_row(output_file_path, row)

def append_row(output_file, row):
    f = open(output_file, 'a', newline='')
    writer = csv.writer(f, delimiter=',')
    writer.writerow(row)
    f.close()

if __name__ == '__main__':
    move_by_coords = {
        'x': -338802.5,
        'y': -1714752.5,
        'z': 0
    }

    input_file_path = r'D:\incoming_data\large_data_set1.csv'
    output_file_path = r'D:\outgoing_data\large_data_set_relocated.csv'
    move_txt_coords_to(move_by_coords, input_file_path, output_file_path)

Comment: Not sure why you ask a new question - all what you need can be found on SO: [skip-rows-during-csv-import-pandas](https://stackoverflow.com/questions/20637439/skip-rows-during-csv-import-pandas)   and [how-to-read-a-6-gb-csv-file-with-pandas](https://stackoverflow.com/questions/25962114/how-to-read-a-6-gb-csv-file-with-pandas)  and `df[x] += delta_x` for a basic "add-offset-value" to one column - else you would need to use iloc etc.
Comment: Have you considered using [pandas](https://pandas.pydata.org/) dataframes to read/load the csv and then apply the operations only on the desired columns as if they were numpy arrays?
Remark that pandas won't be efficient if you load all the data together, thus use batchsize! Otherwise you can check out the [dask](https://dask.org/) library
Comment: @Patrick I don't want to skip any rows. I was just doing that so I could get it into numpy. Regardless of that I've now discovered how to do it with Pandas while retaining my 4 header rows. But now I'm trying to work around the issue of "Unnamed" appearing in cells that were blank. I still want those cells to be blank.
`df = pd.read_csv(input_file_path, header=[0,1,2,3])` did the trick for the 4 header rows.
and I was able to just do a simple `df.centroid_x += move_by_coords['x'] ` to update specific columns. @FBruzzesi I will need to look into batchsize thanks for that bit of advice
Answer: Okay so I've got an almost complete answer and it was so much easier than trying to use numpy.
import pandas pd

    input_file_path = r'D:\input\large_data.csv'
    output_file_path = r'D:\output\large_data_relocated.csv'

    move_by_coords = {
            'x': -338802.5,
            'y': -1714752.5,
            'z': 0
        }

    df = pd.read_csv(input_file_path, header=[0,1,2,3])
    df.centroid_x += move_by_coords['x']
    df.centroid_y += move_by_coords['y']
    df.centroid_z += move_by_coords['z']

    df.to_csv(output_file_path,sep=',')

But I have one remaining issue (possibly 2). The blanks cells in my header are being populated with Unnamed. I somehow need it to sub in a blank string for those in the header row.

Also @FBruzzesi has warned me I made need to use batchsize to make it more efficient which i'll need to check out.
---------------------Update-------------
Okay I resolved the multiline header issue. I just use the regular csv reader module to read the first 4 rows into a list of rows, then I transpose this to be a list of columns where I convert the column list to tuples at the same time. Once I have a list of column header tuples (where the tuples consist of each of the rows within that column header), I can use the list to name the header. I there fore skip the header rows on reading the csv to the data frame, and then update each column by it's index. I also drop the index column on export back to csv once done.
It seems work very well.
import csv
import itertools
import pandas as pd

def make_first_4rows_list_of_tuples(input_csv_file_path):
    f = open(input_csv_file_path, newline='')
    reader = csv.reader(f)
    header_rows = []
    for row in itertools.islice(reader, 0, 4):
        header_rows.append(row)

    header_col_tuples = list(map(tuple, zip(*header_rows)))
    print("Header columns: \n", header_col_tuples)
    return header_col_tuples

if __name__ == '__main__':
    move_by_coords = {
        'x': 1695381.5,
        'y': 5376792.5,
        'z': 100
    }

    input_file_path = r'D:\temp\mydata.csv'
    output_file_path = r'D:\temp\my_updated_data.csv'

    column_headers = make_first_4rows_list_of_tuples(input_file_path)
    df = pd.read_csv(input_file_path, skiprows=4, names=column_headers)
    df.iloc[:, 0] += move_by_coords['x']
    df.iloc[:, 1] += move_by_coords['y']
    df.iloc[:, 2] += move_by_coords['z']
    df.to_csv(output_file_path, sep=',', index=False)

Comment: try adding `, index_col=0` to your `read_csv` statement - not sure if it helps - see [https://pandas-docs.github.io/pandas-docs-travis/reference/api/pandas.read_csv.html#pandas.read_csv](https://pandas-docs.github.io/pandas-docs-travis/reference/api/pandas.read_csv.html#pandas.read_csv)


Caught AttributeError while rendering: 'str' object has no attribute '_meta'

Question: def broadcast_display_and_form(request):

    if request.method == 'POST' :
        form = PostForm(request.POST)
        if form.is_valid():
            post = form.cleaned_data['post']
            obj = form.save(commit=False)
            obj.person = request.user
            obj.post = post
            obj.save()

            readers = User.objects.all()
            for x in readers:
                read_obj = BroadcastReader(person = x)
                read_obj.post = obj
                read_obj.save()
            return HttpResponseRedirect('/broadcast')
    else :
        form = PostForm()
    posts = BroadcastReader.objects.filter(person = request.user)
    return render_to_response('broadcast/index.html', { 'form' : form , 'posts' : posts ,} )

My template
{% extends "base.html" %}
{% load comments %}
{% block content %}
<form action='.' method='POST'>

    {{ form.as_p }}

  <input type="submit" value ="send it" /></input>

</form>

{% get_comment_count for posts.post as comment_count %}
{% render_comment_list for posts.post %}

{% for x in posts %}

       {{ x.post.person }} - {{ x.post.post }}

{% endfor %}

{% endblock %}

Comment: Could you post the traceback for the error as well?
Comment: anything to help myself out of the situation
Comment: As `sunn0` wrote, you should really post your traceback also.  We have no idea where the error is occurring, so it's hard to even begin.  Also, what is the definition for PostForm and BroadcastReader?
Answer: What is posts.post supposed to be? posts is a BroadcastReader QuerySet, and probably doesn't have such an attribute? I'm guessing the comment rendering tags are causing your error? Try removing them, or using posts.0 or similar to debug.
Which "post" do you actually want to display comments for? Your view logic is unclear.
Comment: Hey sorry to bother you guys , i had just put the url for comment framework in my application's urls.py instead of project's.. Thanks for you help
Comment: But now i want to hide the name , email and url fields for authenticated users.. How do i go about doing that ??
Comment: It would be best to open a new question, since this is almost completely unrelated.


How to modify contact form

Question: I'm trying to modify the contact form to add a privacy clause before the "Send" button, I have to add a text and two mandatory check boxes. But I just started with and I have not much idea.
Thanks.
Comment: What version of Drupal/Webdform are you using?
Comment: Im using Drupal 7.63 actually
Answer: You can go to /admin/structure/webform and then click Build on the form you are using as a contact form.
Then you can click Add element and add your checkboxes or any other field you need.
The option to make the checkboxes mandatory will be under Form validation.
Comment: I'm using Contact form actually.  Should i download and install Webform? thanks
Comment: There is no option you _should_ go for. The answer is for webform since your question is tagged. You can of course use any other suitable module.

How to modify contact form


Configuring Apache 2.4 on Fedora 23

Question: I have got problems with apache 2.4 on Fedora. When on Debian I do

service apache2 start

I can access this machine from another computer just by
typing it's IP address on browser., but on Fedora I can't.
And second problem is I can't access on Fedora itself directory localhost/cgi-bin/ It says

Forbidden
You don't have permission to access /cgi-bin/ on this server.

I know that problem with httpd.conf file (mine is on default state), I read few answers on Stack
but didn't understand them. Please help!
Comment: Are you on Fedora Workstation, Fedora Server, or something else? It makes a difference because the firewall config is different.
Comment: Also, where is the directory `localhost/cgi-bin` on your filesystem?
Answer: On Fedora 23 start Apache like this (as root)
<code>$ systemctl start httpd
</code>
check if started
<code>$ systemctl status httpd
</code>
to make it start by default:
<code>$ systemctl enable httpd.service
</code>

Configuring Apache 2.4 on Fedora 23


Simultaneous Fouls

Question: In a football match, if two players, each from a different team, commits fouls simultaneously, how should the free kick/penalty be awarded?
It could be that at a corner kick, one defender A1 pushes one of his opponent B1 in the defender's own penalty area while another opponent B2 pushes A2, a teammate of A1; and the two fouls are simultaneous.
Comment: Dropball situation I think.
Answer: The referee would stop play and restart with a dropped ball to be taken from where the ball was located when play was stopped (unless the ball was within the goal area - it would then be brought to the nearest point on the goal area line parallel to the goal line).
On p. 74 of the 2014/15 FIFA Laws of The Game (relevant passage in italics):

More than one offence occurring at the same time

Offences committed by two players from the same team:

the referee must punish the most serious offence when players commit
  more than one offence at the same time
play must be restarted according to the most serious offence committed

Offences committed by players from different teams:

the referee must stop play and restart it with a dropped ball from
  the position of the ball at the time of the stoppage, unless play was
  stopped inside the goal area, in which case the referee drops the ball on
  the goal area line parallel to the goal line at the point nearest to where
  the ball was located when play was stopped

In practice this rarely happens, as one foul will usually occur before the other, even if it is by fractions of a second.
The only time I can imagine simulataneous fouls occurring is if two players from opposing teams dive in to win the ball and collect each other instead. Even in this situation, they'd probably have to be facing each other, as a side-to-side collision where both players are using similar amounts of force is unlikely to be seen as a foul.
In the situation you described, most pushing and holding occurs at a corner kick before the ball is put into play. If this is the case, the restart would be a retake of the corner kick.

Simultaneous Fouls


Can we *ever* use certain log/exp identities in the complex case?

Question: This article on Wikipedia points out that certain identities for the log and exponential functions which are familiar from the real case require care when used in the complex case. Failures in the following identities are pointed out
$$\log{z^w} \equiv w \log {z}\\(zw)^{\omega}\equiv z^{\omega}w^{\omega}\\e^{zw} \equiv (e^z)^w$$
The article alludes to the fact that the identities do not always work, even if we consider them as an assertion about equalities of sets (and consider the functions as multivalued).
Now, how can we work with these identities without leading ourselves into error? Is the answer simply never to employ them in the presence of a complex base or exponent? Occasionally I see uses of these identities; here is one example:
Here is a scanned page from Schaum's outline on complex variables which shows how to evaluate $\int_{0}^{\infty}\frac{x^{p-1}}{1+x}dx$ with a keyhole contour (for $0<p<1$). As usual, on the return path, we have to parameterize the path as $z = e^{2\pi i}x$, $x\in \mathbb{R}$, rather than simply $z=x$. Now when the author exponentiates this term, he uses $(x e^{2\pi i})^{p-1}=x^{p-1}e^{2\pi i(p-1)}$, employing the second identity above. What justifies this, given that the identity does not hold in general?
Comment: Good question. No, $p$ is a real number between $0$ and $1$. I made an edit to clarify this.
Comment: Is $p$ an integer?
Answer: By definition $z^w = e^{w \log(z)}$ (for whichever branch of $\log(z)$ you are using), so
$$ \log(z^w) = w \log(z) + 2 \pi i n$$
for some integer $n$.   If, for example, you are using the branch of $\log$ that has imaginary part in $(\alpha, \alpha + 2 \pi]$, you choose $n$ so that the imaginary part is in that interval.
Similarly, $(zw)^\omega = e^{\omega \log(zw)}$, and $\log(zw) = \log(z) + \log(w) + 2 \pi i n$ for suitable integer $n$, so
$$(zw)^\omega = e^{\omega(\log(z) + \log(w) + 2 \pi i n)} = e^{2 \pi i n \omega} z^\omega w^\omega $$
And
$$(e^z)^w  = e^{w \log(e^z)} = e^{w (z + 2 \pi i n)} = e^{2 \pi i n w} e^{wz}$$
EDIT: In Schaum's example, it is misleading to write $z = e^{2 \pi i} x$ (which would be the same as $x$), it's really $e^{(2 \pi - \epsilon) i} x$ where $\epsilon > 0$ is arbitrarily small.
Then $\log(z) = \log(x) + (2 \pi - \epsilon) i$ (for the branch of log with imaginary part in $[0, 2 \pi)$), and $$z^{p-1} = e^{(p-1)\log(z)} = e^{(p-1)\log(x) + (p-1)(2\pi - \epsilon) i} = x^{p-1} e^{(p-1)(2\pi - \epsilon) i}$$
Comment: This is helpful, thank you. Would you say that the manipulation $(e^{2 \pi i} x)^{p-1} = e^{2 \pi i (p-1)}x^{p-1}$ is unjustified, unless we include a factor of $e^{2 \pi i n (p-1)}$?
Answer: I should have thought a bit before asking the silly question in my comment.  I also should have looked at your link.    Clearly if $p$ were an integer we wouldn't be doing the problem this way in the first place.
Keep in mind that the map $z\mapsto z^p$ is multivalued.  When we write $z^p=r^pe^{i\theta p}$ (with $\theta$ assumed to lie in $(0,2\pi)$ as suggested by our contour), it may look like we're applying the identity $(xy)^p=x^py^p$ to $z=re^{i\theta},$ but what we're really doing is making the choice as to which value of $z^p$ we're going to take.  The actual values of $z^p$ are $r^pe^{i\theta p}e^{2\pi inp}.$  What we are doing is to choose $n=0.$
Why do we make this particular choice?  Well, we want the values of $z^p$ to agree with the values of the real function $x^p$ when we are on the piece of the contour that lies just above the real axis, which it does if $n=0$ since $\theta$ is close to $0.$  Of course we have to use the same choice throughout the entire contour since we don't want our function to have any discontinuities.  If you think about it, that is why we have to remove the positive real axis from our contour: once you go $360^\circ$ or more around the origin, you will necessarily encounter the multivaluedness of $z^p.$
Comment: That makes a lot of sense, thank you.
Answer:
Can we ever use certain log/exp identities in the complex case?

Two answers:

Can we ever use them?  Yes.
Can we blindly use them?  No.

For example: That computation from Schaum's can be done along those lines.  By applying these identities carefully.  Not by applying them blindly.
Comment: The user has to think about what he/she is doing.  Which (as we know) most people want to avoid.
Comment: What distinguishes their careful use from their blind use?


Nerd Fonts folder icon not showing up

Question: I have installed Nerd Fonts and selected one of the nerd fonts for my terminal.
For some reason, the default folder icon set in Nvchad config in Nvim Tree just shows up as an Asian character.
I tried different terminals, Gnome terminal and Guake. I also tried to open the nvimtree.lua config file from Nvchad in both Nvim and Vim, and only the default folder icon is showing up this way.
I have updated my Nvchad and NerdFonts to the latest release. I also tried to download the lua config file directly from Github, but still the same issue.
Answer: You need to have the newer version of your font as nerd-fonts have been updated recently. For example, you have to download the new version of 'MesloLGL Nerd Font' and install it in your fonts.
Also, you must make sure your code (nvimtree.lua) has [fragment of nvimtree.lua code] 1 the correct icons!
Answer: You're experiencing issues with certain tools like Nvchad due to outdated versions of Nerd Fonts, I recommend obtaining your preferred font directly from the GitHub repository. Navigate to the "patched-fonts" folder within the repository at the following link:
GitHub Repository - Nerd Fonts
Downloading the font directly from this source ensures you're using the latest patches and fixes, which should help alleviate any compatibility problems with tools like Nvchad.

Nerd Fonts folder icon not showing up


Function templates and Private Inheritance

Question: I recently ran into a problem when using a private inheritance scheme in which the base class defined a template method and the (privately) derived class made that method public via a using declaration under the public access specifier. The template was designed to take the address of a function and invoke that function pointer. However, upon attempting to pass the name of the function to the derived class template method, I receive an error message that states that the base-class method cannot access a private member declared in the derived class. Here is a code segment that models the issue:
class A
{
public:
     template<class T> void Funct(T pFunct) { }
};

class B : private A
{
public:
    using A::Funct;
};

void Show(void) { }

int main(void)
{
    B b;
    b.Funct(Show);
}

The exact resulting error message is: 'A::Funct' : cannot access private member declared in class 'B'
I am able to resolve the issue simply by:
1)Preceding the function argument name with the address-of operator:
b.Funct(&Show);

2)Explicitly qualifying the template type argument:
b.Funct<void(*)(void)>(Show)

If the Show() function were a template as well, I would need to explicitly qualify the template using the proper template type arguments used to instantiate Show.
My question is not how to solve the problem but why the error message is being generated. Why does instantiating Funct() with Show versus with &Show cause the compiler to do two different things. And why does instantiating Funct() with Show cause the Funct() method to attempt to access the private data in class B (which I'm assuming is the A subobject in class B)?
Comment: What compiler? You example compiles fine with g++ 4.2.
Comment: Interesting. I'm using Microsoft Visual C++ 2010
Comment: Is this visual studio?  Some of the older versions had problems with member function pointers in template parameters.
Comment: Yeah, it is VC. Might be some sort of bug then?
Comment: Using Intellisense, if you don't have `&Show` or specify `void(*)(void)` as the type explicitly, VC does not know the correct type for the template.  If you have `&Show`, Intellisense will correctly display `Funct` as the fully qualified function when you hover over it.
Answer: Compiles fine with Comeau Online. Ergo, compiler bug. Report it.
Cheers & hth.,
Answer: bcc32 gives error: error bccE2247: 'Funct<void (*)()>(void (*)())' is not accessible in function main()
I believe this is a compiler bug.


Set variable inside if statement not working

Question: I have file1.txt and test.bat in the same folder.
Inside test.bat:
This works:
SET test=file1.txt
COPY %test% file2.txt

This does not work:
IF ""=="" (
SET test=file1.txt
COPY %test% file2.txt
)

Why?
The only difference is "if" statement.
Whats more interesting:
IF ""=="" (
COPY file1.txt file2.txt
)

works
Answer: This works:
setlocal enabledelayedexpansion

IF ""=="" (
SET test=file1.txt
COPY !test! file2.txt
)

More about "enabledelayedexpansion":
https://ss64.com/nt/delayedexpansion.html


Wpf making a growing transparent window

Question: I would like to make my window be transparent but only a part of the window and not all of it.
My goal will be a window that his left side is fully seen and ,slowly, to right side will be seen less and less until the right side is fully transparent.
How do I do it?
Answer: You can easily perform this by setting AllowTransparency and adding an opacity mask on your window. Like this:
<code><Window x:Class="Testing1.MainWindow"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="MainWindow" Height="350" Width="525" AllowsTransparency="True" WindowStyle="None">

       <Window.OpacityMask>
    <LinearGradientBrush EndPoint="1,0.5" StartPoint="0,0.5">
        <GradientStop Color="Black" Offset="0"/>
        <GradientStop Offset="1"/>
    </LinearGradientBrush>
    </Window.OpacityMask>

    <Grid>

    </Grid>

</Window>
</code>
Comment: I tryed to change the black color to anything else but it still showing a white color only. even the black u wrote does not effect a thing
Comment: It's normal. It is an opacity mask only alfa value of the brush counts. Use background property to play with the color ;)

Wpf making a growing transparent window


Goodness of fit of a SUR model

Question: I know that McElroy R^2 is a measure of goodness of fit for Seemingly Unrelated Regressions (SUR models), but how can one judge that the estimated equations are well fit by using the McElroy R^2?
Answer: This kind of $R^2$ is a monotonic transformation of the F statistic from the the test that all the coefficients in the system are zero. It is bounded on $[0,1]$, so it has the same kind of interpretation as the ordinary $R^2$:
$$ \text{McElroy }R^2 = 1-\frac{U'W*U}{Y'W*Y},$$ where

U is the stacked vector of residuals
Y is the stacked vector of dependent variables
W is the variance-covariance matrix of residuals

If your model is the cat's pajamas, the vector of residuals will be close to zero, so $R^2 \approx 1$ since the second term drops out. If the model has no explanatory power, then $U\approx Y$, so $R^2 \approx 1-1=0$.
The full citation is
McElroy, Marjorie B. (1977) "Goodness of Fit for Seemingly Unrelated Regressions", Journal of Econometrics, 6(3), November; 381-387. I can't seem to find an ungated link.

Goodness of fit of a SUR model


Getting the 307 Redirection Response instead of the target-location page content from an enpoint in Spring Boot

Question: So I have a simple endpoint in my Spring Boot App, which just redirects to another website:
@Controller
public class FeedbackController {
<code>@GetMapping(path = "/test")
public Mono<ResponseEntity<Void>> method() {
    String redirectUrl = "https://www.google.com";
    return Mono
            .just(ResponseEntity.status(HttpStatus.TEMPORARY_REDIRECT).location(URI.create(redirectUrl)).build());
}
</code>
Making a GET request to this endpoint (e.g. in browser or with postman) gives me the page content of google as a response. However for testing purposes I want to make sure that the response is a TEMPORARY_REDIRECT with www.google.com as the Location Header.
How can I make a request to this endpoint so that the response is the 307 TEMPORARY_REDIRECT instead of the 200 with page content from the target website?
Answer: First
to test if its working, you could use simple tools like curl :

We add the -L flag to tell curl that we want to know if we are getting redirected

<code>curl -L http://www.....
</code>
Go further
Then, you could simply use tools like MockMvc to automate this test
See this SO post


JDBC: multicolumn IN query

Question: I have a following query:
SELECT
  date, userId, value
FROM
  tbl_table
WHERE
  date = to_date(:date, 'YYYY-MM-DD')
    AND
  userId = :userId

It allows to request for a single value like this:
MapSqlParameterSource args = new MapSqlParameterSource();

args.addValue("date", date, Types.VARCHAR);
args.addValue("userId", userId, Types.VARCHAR);

SqlRowSet rowSet = jdbcTemplate.queryForRowSet(SQL_SELECT, args);

jdbcTemplate.queryForRowSet(SQL_SELECT_MARKET_VALUE, args);

This is totally ok, but extremelly slow in case you have to query value for many date/userId pairs.
I would like to optimize it using multicolumn IN clause, but how do I handle multicolumn list via JDBC (or better question: is it possible using JDBC)?
Comment: `where (date, userid) in ( (?,?), (?,?), (?,?) )` will work in plain JDBC. I don't know if it will work with JdbcTemplate and named parameters though
Comment: @a_horse_with_no_name That's a big question: JDBC template expands list parameters into wildcards itself.
Comment: I am solving this with a dynamically created query. Fast enough for me.
Answer: Oracle supports multiple columns in "in" predicate:
SELECT
  date, userId, value
FROM
  tbl_table
WHERE
  (date, userId) IN ((to_date(:date1, 'YYYY-MM-DD'), :userId1), (to_date(:date2, 'YYYY-MM-DD'), :userId2))

However JDBC doesn't provide a decent support of in-statement parameters - you will have to build the query using StringBuilder or use some of workarounds described here
Comment: Unrelated, but: the `to_date()` seems unnecessary if you are passing real dates (which you should)
Comment: Yes, it does. Question here is more related to JDBC-ORA bridge and the case, when number of parameters is unknown.
Comment: @a_horse_with_no_name not only unnecessary but probably harmful, as it would trigger an implicit date-to-string conversion using the default date format
Answer: If what you want is to pass JDBC a list of date/userId pairs, or a list of dates and a list of userIds, I think it will not work.
A possible workaround in Oracle would be using a global temporary table with ON COMMIT DELETE ROWS. Your would have:
  -- DDL for the workaround
  CREATE GLOBAL TEMPORARY TABLE admin_work_area
    (d DATE,
     userId VARCHAR2(10))
  ON COMMIT DELETE ROWS;
...
-- Start of query method pseudo-code
...
-- You should be able to JDBC-batch these for better performance
INSERT INTO temp_multicolumn_filter (d, userId) VALUES (date1, userId1);
INSERT INTO temp_multicolumn_filter (d, userId) VALUES (date2, userId2);
...
-- Query using temp_multicolumn_filter
SELECT date, userId, value
FROM tbl_table
WHERE
(date, userId) in (select d, userId from temp_multicolumn_filter);
...
-- End of query method pseudo-code

As the temporary table has the ON COMMIT DELETE ROWS, each transaction will only see its own date/userId pairs. Just remember that if you use the temporary table more than once in the same transaction, you might need to clear it before using it.
UPDATE:
Another option would be using a PIPELINED PL/SQL function to "build" your table inside the query:
-- DDL for this workaround
CREATE TYPE date_userid_pair AS OBJECT (
    d DATE,
    userId VARCHAR2(10));

CREATE TYPE date_userid_dataset IS TABLE OF date_userid_pair;

CREATE FUNCTION decode_date_userid_pairs(dates in varchar2, userIds in varchar2)
RETURN date_userid_dataset PIPELINED IS
    result_row date_userid_pair;
BEGIN
    WHILE there are more "rows" in the parameters LOOP
        result_row.d := -- Decode next date from dates
        result_row.userId := -- Decode next userId from userIds

        PIPE ROW(result_row);
    END LOOP;
END;

// Start of query method pseudo-code
...
// This is Java code: encodeList encodes a List of elements into a String.
encodedDates = encodeList(listOfDates);
encodedUserIds = encodeList(listOfUserIds);
...
// Query using temp_multicolumn_filter
SELECT date, userId, value
FROM tbl_table
WHERE
(date, userId) in (
    select date, userId
    from TABLE(decode_date_userid_pair(:encodedDates, :encodedUserIds));
...
// End of query method pseudo-code

But this is more hacky, and if you don't have privileges to create a temporary table, then you probably won't have CREATE TYPE either (you might not even have CREATE FUNCTION privilege).
Comment: Great solution, but creating temp tables is above my privileges and due to bureaucracy reasons I have only option to come up with workaround.
Answer: It depends of details. If user/date filter is quite persistent (should be user more than once) temporary table will be the best decision. You can fill it once, you can edit it, and you can use it several times without reloading.
If you need of quite large number of pairs, I'd recommend you to use a table type. It would be something like this:
create type DateUserPair as object (dt date, userid integer);
create type DateUserPairs as table of DateUserPair;
....
SELECT
  date, userId, value
FROM
  tbl_table src,
  table(cast :filter as DateUserPairs) flt
WHERE
  src.date = flt.dt and
  src.userId = flt.userId;

If filter would be small, filtering by (date, userId) in ((?,?), (?,?), ...) would be simple and clever.
Btw, your approach
  date = to_date(:date, 'YYYY-MM-DD')

isn't good practise. Such conversions should be done by client, not by server. Use
  date = :date

and assign it as date instead.
Comment: Date conversion is done on server on purpose. Otherwise there are so many places, where timezone mess could happen.

JDBC: multicolumn IN query


How to access elements of a String array from main()

Question: I am having difficulty retrieving a value from a string array.
I need to have it output the name of the day of the week.. i.e. Sunday for the number 0. I have the code perfect for outputting the numbers, but I can't get it to pull from my array of day names.
import java.util.Scanner;

public class FutureDateJava {
    String[] dayStrings = { "Sunday", "Monday", "Tuesday",
                            "Wednesday", "Thursday", "Friday",
                            "Saturday" };

    public static void main (String[] args) {
        Scanner input = new Scanner(System.in);

        System.out.println("Enter today's day: ");
        int today = input.nextInt();

        System.out.println("Enter a day in the future ");
        int future = input.nextInt();

        int futureDay = (today + future) % 7;

        System.out.print("Today is " + today +
            " and the future day is " + futureDay);
    }
}

Answer: Actually access the array:
System.out.print("Today is " + dayStrings [today % 7] + " and the future day is " + dayStrings [futureDay]);

The %7 is to make sure that a day (eg 10) doesn't cause an out of bounds exception. However, this isn't exactly optimal, it assumes that Monday is day 0, but the idea is there.
Comment: Oh, sorry about that, make dayStrings `static`. If the array isn't gonna change, so also consider making it `final`,
Comment: No need to use `futureDay % 7` again.
Comment: I'm glad to know I was sort of close.  I did try the code and when I use the textpad compiler it says error a non-static variable dayStrings cannot be referenced from a static context and then shows the System.out.print


How to change cell type of a databound column

Question: I have a datagridview that is bound to a dataset via a table adaptor which in turn linked to a SQL table.

I want to change the cell type of the status and priority columns to comboboxes.
I have tried this:
 private void dgvFechas_ColumnAdded(object sender, DataGridViewColumnEventArgs e)
    {
        try
        {
            if (e.Column.ValueType == typeof(TextBox) && (e.Column.Index == 5) || (e.Column.Index == 6))
            {
                e.Column.CellTemplate = new DataGridViewComboBoxCell();
            }
        }
        catch (Exception ex) { }
    }

but it doesnt seem like the event handler fires when the program starts up.
So im not too sure how i can get it to work.
Comment: @JohnG, I guess really the columns are never added becuase the columns are synced to the datatable witch is synced to the SQL database. There should be no vaules "Priority" and "Status" columns until a user set one via a combobox.
Answer: I followed this guide which allowed me change the cell type:
https://social.msdn.microsoft.com/Forums/windows/en-US/c6d60712-135e-4fd3-a6a0-51dbb4be0dca/how-to-add-combobox-to-winform-datagridview-bound-to-datatable?forum=winforms
After doing this i got a data_error which i fixed by ignoring the data error event. I know that isnt really the correct way of doing things, but it works for me.
 private void dataGridView1_DataError(object sender, DataGridViewDataErrorEventArgs e)
    {
        //ignore
    }


How To change Text to Speech Mac Sierra

Question: There was a setting OSX Yosemite where you could change your dication voice and speed. Does anyone know how to find those settings on macs latest macOS Sierra?
I have looked under keyboard where there are dictation settings but that seems to be you speaking to your computer and it typing it out.
Also looked at accessibility and Voice over settings but not what I am looking for.
The Default voice was Alex and there is a speak back speed that you can change.
Thanks for your help.
Answer: Accessibility -> Speech contains the settings you mention.
Comment: You are the winner I have been looking all over for this  Thank you.
Answer: Accessibitity > Speech
Here you can set speed and choose the voise just like in Yosemite

How To change Text to Speech Mac Sierra


FreeBSD-12.0 PF firewall default rule is blocking when more specific rule allowing is ignored

Question: Our PF firewall contains this:
. . .
scrub in all fragment reassemble no-df max-mss 1440
### em1 ipv4 = 22.214.171.124
nat               log   on $ext_if \
                  from  $net_nat \
                  to    any -> ($ext_if)
. . .
antispoof         log   for $ext_if
block return  out log   all
block drop    in  log   all
. . .

Followed somewhat later by this:
pass          in  log   quick \
                  from  188.8.131.52 \
                  to    any

pass          out log   quick \
                  from  any \
                  to    22.214.171.124

However, TCPDUMP shows this happening:
 00:00:00.116888 rule 3/0(match): block in on em1: 184.108.40.206.2148 > 188.8.131.52.59865: Flags [R.], seq 1, ack 1, win 5707, length 0

 00:00:00.115632 rule 3/0(match): block in on em1: 188.8.131.52.2148 > 220.127.116.11.62733: Flags [R.], seq 1, ack 1, win 159, length 0

 00:00:00.011031 rule 2/0(match): block out on em1: 22.214.171.124.64105 > 126.96.36.199.2148: Flags [P.], seq 2111901423:2111901475, ack 316150303, win 258, length 52

 00:00:00.074555 rule 3/0(match): block in on em1: 188.8.131.52.2148 > 220.127.116.11.58208: Flags [.], ack 1, win 159, length 0

 00:00:00.065409 rule 3/0(match): block in on em1: 188.8.131.52.2148 > 220.127.116.11.56489: Flags [.], ack 1, win 159, length 0

 00:00:00.077103 rule 3/0(match): block in on em1: 126.96.36.199.2148 > 188.8.131.52.62245: Flags [P.], seq 0:36, ack 1, win 136, length 36

 00:00:00.040241 rule 3/0(match): block in on em1: 126.96.36.199.2148 > 18.104.22.168.58208: Flags [.], ack 1, win 159, length 0

 00:00:00.026616 rule 3/0(match): block in on em1: 22.214.171.124.2148 > 126.96.36.199.56489: Flags [R.], seq 1, ack 1, win 159, length 0

My question is: Why?  What is causing the later 'quick' rule to not match and instead letting the default rules take effect?
Comment: Rule 2 is: `block return  out log   all`
Rule 3 is: `block drop    in  log   all`
Comment: Run "pfctl -sr -vv", identify and post the the blocking rules.
Answer: If I'm reading this right, you have a NAT router, and a client behind it that you wish to allow out to the internet. I'm noticing that almost all of my rules state the interface they apply to, and don't bother stating the direction. Yours are the other way around, heavy on the direction but don't specify the interface.
Looking at your only two 'pass' rules,
pass in  quick from 126.96.36.199 to any
 pass out quick from any to 220.127.116.11
These only make sense when applied to your router's LAN interface. The first permits that particular IP to talk to the router and whatever it's connected to, the other lets things talk to that IP via the router. That's an okay first step. However, I believe the packet is not allowed to proceed any further, beause the corresponding rules for the WAN interface are missing. Though I don't remember if 'nat' requires this (and if the IP used in the rule needs to be pre-nat or post-nat). A quick thing to try would be to remove the 'in' and 'out' keywords from those two rules.
In my home setup, I dodge most of the work by using these kinds of sloppy shortcuts:
set skip on { lo0 $lan_if $vpn_if } # trusted interfaces
 nat on $wan_if inet from !$wan_if to any -> ($wan_if:0)
 pass out quick all                  # outbound default to allow
 block return in quick all no state  # inbound default to deny, return icmp(unreachable)
Answer: It looks more like there's some kind of state mal-synchronisation other than rules logic's issue.
You could try to loosen scrub partially or removing it completely. For e. g., no-df is being cut-copied blindly, but removing DF would interfere with Path MTU discovery.