Postfix: For a certain address, only allow emails from a certain domain

Question: I've set up my server so that, in order to track down the source of spam (and block it), the email address that I use to sign up for stuff is email@example.com (e.g. for Facebook, it would be email@example.com).
Now, suppose that I get spam emails to email@example.com - (from, e.g., firstname.lastname@example.com). I'd like to block them, but continue to receive emails from the facebook.com domain - i.e. I would like to restrict email to a certain address to come from a certain domain.
Is this possible with Postfix? Or will I need something additional? (I'm already running Dovecot, although that doesn't seem like the right tool).
I'm not using a conventional spam filter because the above seems more elegant - however, if there is a spam filter that can implement the above, I'd be happy to use it.
Answer: Haraka can handle this case easily, and it's very easy to setup in front of Postfix so that your current Postfix setup continues to work as-is (or if you don't want to do that, it can deliver direct to Dovecot via LMTP).
You'd setup a Haraka plugin like the following:
<code>var net_utils = require('./net_utils');

exports.hook_rcpt = function (next, connection, params) {
    var recip = params[0];
    if (connection.relaying) {
        // Already relaying (i.e. outbound) - skip.
        return next();
    }

    // Get who this is coming from and change a.b.example.com into just example.com
    var check_domain = net_utils.get_organizational_domain(connection.transaction.mail_from.host);

    // Turn 'example.com' into just 'example'
    check_domain = check_domain.replace(/\..*$/, '');

    if (check_domain.toLowerCase() != recip.user.toLowerCase()) {
        return next(DENY, "You dirty dirty spammer");
    }

    next();
}
</code>
Add that to your plugins directory, and add a line to config/plugins specifying what you called the file so that it is loaded into your configuration.


Does the unbalance of the pos and neg samples lead to the low variance of the neural network's outputs?

Question: I train a neural network and the batch size is 1000. I check the outputs from my model every time it fed with 1000 sample, and I find that the outputs, which shall be the predictions for the binary classification task. What confused me is that the the outputs in every batch has very low variance, the max prediction score is 0.25 and the min is above 0.1.  There are 25% positive samples in my dataset whose labels are 1.
So I want to know why the variance is so low. It seems that the model has no ability in the classification task, after several epochs training.
Comment: What's your architecture, loss function, activation functions, etc? How many samples do you have?
Comment: There are 500 millions samples and the loss function is log_loss in tensorflow.   There are two layers of MLP.
Comment: So you have two outputs using softmax activation as final layer before the loss function? I don't use tensorflow, but I believe `sigmoid_cross_entropy` with a single output is the conventional choice for binary classification (though equivalent to 2-output softmax+logloss).
Comment: I just use one output as logit and it will be functioned by the sigmoid. The output from sigmoid which ranges from 0 to 1 acts as the final prediction (probability).
Answer: It seems like you could answer this question experimentally: artificially rebalance your classes and check if the outputs are likewise low-variance.
Moreover, examining the raw scores is not a great way to assess how well the model is doing. The real question is whether the scores do a "good job" of matching the true labels. There are a number of ways to measure this:

cross entropy is lower when the predictions are closer to the true labels. Is the cross-entropy of your model lower than it would be by chance? Random scores have the value $\log(k)$ in expectation, for $k$ the number of classes.
ROC AUC, a.k.a. the $c$-statistic, measures how well the model predictions separate the two classes. A value greater than 0.5 indicates that your model is better than chance.


is there a notation to designate the induced homomorphism including base point?

Question: Let $X,Y$ be a topological spaces.
Let $f:X\rightarrow Y$ be a continuous function.
Fix $x_0\in X$.
Define $f_*([r])=[f\circ r]$ for every loop $r$ at $x_0$.
Then, $f_*:\pi_1(X,x_0) \rightarrow \pi_1(X,f(x_0))$ is called the induced homomorphism.
As you can see the notation $f_*$ itself shows no information about which base point is chosen.
Is there a typical notation for resolve this problem?
For example, when one wants to emphasize on where a limit is taken, one usually denotes like $\lim_{x\to p, x\in A}$.
I want to know whether there is a corresponding notation for $f_*$.
Answer: To my knowledge there is no common notation for this. This is perhaps because the conjugacy class of a fundamental group of a connected manifold is actually independent of the base point:
Given two base points $x_0, y_0 \in X$ and any path $\alpha: [0, 1] \to X$ such that $\alpha(0) = x_0, \alpha(1) = y_0$, we get a map
$$\Phi: \pi_1(X, x_0) \to \pi_1(X, y_0)$$ defined by
$$\Phi([\gamma]) = [\alpha \cdot \gamma \cdot \alpha^{-1}].$$
Here, $\cdot$ denotes path concatenation and $\alpha^{-1}$ is the curve $\alpha$ run backward, and checking shows that it is well-defined.
So, while the fundamental group itself depends on a choice of base point, the conjugacy class of the fundamental group does not, i.e., it is an invariant of the (connected) manifold. In this sense, the base point of a fundamental group is inessential in a way that, e.g., the "base point" of a limit is not.
Answer: There is another viewpoint different from the answer of @Travis, in which there is no ambiguity up to conjugacy.
Namely, it is common to put the base point into the notation for the map $f$ itself, like this:
$$f : (X,x_0) \to (Y,y_0)
$$
In this notation, one regards $f$ not just a morphism in the category of topological spaces and continuous maps, but as a morphism in the category of base-pointed topological spaces and base-point preserving continuous maps. In this manner the base points for induced homomorphism $f_* : \pi_1(X,x_0) \to \pi_1(Y,y_0)$ are indeed unambiguous.


Searching Unique Id in Multiple Tables

Question: I have a Scenario where i am required to search unique id in Mutiple tables and assign a level based on where it exist.
Table Hierarchy

        TableA
                TableB
                         TableC

Table B holds primary key of table A and Table C holds the Primary of TableB
I need to have a function that takes Id as parameter and Searches that Id in this Table Hierarchy and Return Level i-e If it exist in TableA Level should be One. If it exists in TableB Level should be 2 and if it's in TableC then level should be 3
Comment: What if it exists in more than one table?
Answer: Here's one way:
SELECT CASE WHEN EXISTS(SELECT 1 FROM TableC WHERE id = @id) THEN '3'
            WHEN EXISTS(SELECT 1 FROM TableB WHERE id = @id) THEN '2'
            WHEN EXISTS(SELECT 1 FROM TableA WHERE id = @id) THEN 'One'
       END AS "Level"

Comment: How this can be transformed into a function that takes Id as parameter and Returns Level for each row. Basically i want to use it in Query
Answer: You need a query that looks like this:
SELECT 1 FROM TableA WHERE id = {0}
UNION ALL
SELECT 2 FROM TableB WHERE id = {0}
UNION ALL
SELECT 3 FROM TableC WHERE id = {0}

The advantage to this over "WHEN" statements is that a) I don't like WHEN statements, and b) if a key exists in multiple tables, this returns a record for each, giving you the opportunity to build program logic around handling that case.
Answer: Assuming you want to return the highest level, 3 being the highest for TableC, then something like this should work using the CASE statement:
SELECT
   CASE
     WHEN C.Id IS NOT NULL THEN 3
     WHEN B.Id IS NOT NULL THEN 2
     WHEN A.Id IS NOT NULL THEN 1
   END as Level
FROM (SELECT @Id as Id) H
   LEFT JOIN TableA A ON H.Id = A.Id
   LEFT JOIN TableB B ON H.Id = B.Id
   LEFT JOIN TableC C ON H.Id = C.Id

Where @Id is the Id of the value you are searching for, for example SELECT 1 as Id.
Good luck.
Comment: Sorry, there is no table as Hierarchy. There are three tables i-e TaBLEA, TABLEB and TABLEC

Searching Unique Id in Multiple Tables


How to read/write this app.config setting?

Question: Having the following app.config:
<configuration>
  <configSections>
    <sectionGroup name="businessObjects">
      <sectionGroup name="crystalReports">

      </sectionGroup>
    </sectionGroup>
  </configSections>

  <businessObjects>
    <crystalReports>
      <crystalReportViewer>
        <add key="maxNumberListOfValues" value="5000"/>
      </crystalReportViewer>
    </crystalReports>
  </businessObjects>

How would you read and how would you set the setting maxNumberListOfValues at runtime? As extra, what if the app.config didnt exist? How would you set the setting dinamically at runtime then?
Comment: Its a variable that uses the Crystal Reports Runtime not our app directly, the problem is that they doesnt show any other way of setting its value...thats why i would like to know how to change this variable value at runtime...
Answer: You can use following code to get the app.config file to edit..
Configuration configFile = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);

Also to check whether it is empty or not can use something like
            //Check
            if (ConfigurationManager.AppSettings.AllKeys.Contains(str someKey))
            {
                if (configFile.AppSettings.Settings[str someKey].Value == string.Empty)
                    return true;
            }
            else
            {
                // If app config does not contain needed Keys
                //Handle issue
            }

But, case is app.config file is meant to read only values. If you want to edit it on runtime use a user config file. In visual studio there is a file called settings.Settings in properties folder of your project.Use this.
Comment: Its a variable that uses the Crystal Reports Runtime not our app directly, the problem is that they doesnt show any other way of setting its value...thats why i would like to know how to change this variable value at runtime...
Comment: @ase69s Use the settings.Settings file. You can define the variable as a `user scope` property and change it or read it in runtime.
Answer: The whole point of settings within the app.config is that they're static, constant variables if you will. They're not meant to be changed.
Answers such as this explain reading variables from App Config: Reading settings from App.Config
If you didn't have the app.config, you'd simply set maxNumberListOfValues as a global variable in your solution, in, say, a base file which all your forms/pages inherit from.
Comment: Not entirely true. Application settings can be modified and saved in the ConfigurationSections. Using globals for this is not good practice.
Comment: Its a variable that uses the Crystal Reports Runtime not our app directly, the problem is that they doesnt show any other way of setting its value...thats why i would like to know how to change this variable value at runtime...
Comment: Agreed @Sani Huttunen, but I'd stay away from modifying ConfigurationSections on the fly.


How to deal with NSNull in my data?

Question: I am getting an [NSNull rangeOfCharacterFromSet:]: exception when attempting to display a label.  The error reads

'NSInvalidArgumentException', reason: '-[NSNull length]:

Below is my code.
UILabel* spousename = [[UILabel alloc] initWithFrame:CGRectMake(10,line, 300,20)];

[spousename setText:[_thisburial objectForKey:@"Spouse"]];

I tried to test for a null character with the following code but that didn't catch the error. It blew up when trying to execute the addSuview instruction.
if(![spousename isEqual:nil])
{
    [self.view addSubview:spousename];
}

So how do I test for NSNull in a label so I can prevent from getting this [NSNull rangeOfCharacterFromSet:]: error message?
Answer: Your data doesn't actually have a real value for "spouse" and this has been indicated by storing an instance of NSNull as the value. You need to check for this.
This line:
[spousename setText:[_thisburial objectForKey:@"Spouse"]];

Should be updated to check the value. Something like this:
id spouse = _thisburial[@"Spouse"];
if ([spouse isKindOfClass:[NSString class]]) {
    spousename.text = spouse;
} else {
    // The "spouse" value wasn't set or it's not a string
    spousename.text = @""; // Set to whatever you want here
}

Answer: [_thisburial objectForKey:@"Spouse"] is returning an NSNull. There needs to be special handing for this.
NSString *spouseString = [_thisburial objectForKey:@"Spouse"];
if (spouseString == (id)[NSNull null]) {
    spouseString = @"";
}

[spousename setText:spouseString];

Comment: Thanks for the suggestion.  I had to use if ([spouseString isEqual: [NSNull null]]) and that worked.
Comment: Sorry!  The spouseString == [NSNull null] didn't work!   It didn't recognize the null.
Comment: I tested your suggestion again using f ([spouseString.text isEqual: [NSNull null]]) and that worked also.  The key problem was that I wasn't using the text suffix on the variable.
Comment: @Dave that odd `spouseString.text` should crash. Neither `NSString` nor `NSNull` has a property `text`.
Comment: It really doesn't matter. @rmaddy has a better answer.

How to deal with NSNull in my data?


How to skip unsafe usages when trying to delete unused declarations in Android Studio?

Question: I am aware that Androd Studio can find unused declarations in your project. You just run the "Unused declarations" inspection, as described here.
However, sometimes when you try to delete multiple unused declarations all together, you get warnings such as these:
<code>constructor whatever() has 1 usage that is not safe to delete.
field whatever has N usages that are not safe to delete.
method whatever() implements whatever
</code>
I understand that I need to look at those warnings individually to make sure I am deleting without breaking code.
However, if I try to delete many unused declarations, even if one of them has a single unsafe usage, Android Studio won't delete anything at all. For example, suppose Android Studio found 50 unused declarations in my project. I multiselect all of them (using the Shift Key) and then click the "Safe Delete" button. If one of those 50 unused declarations has at least one unsafe usage, the other 49 won't be deleted, even if they were ready to be deleted without problem.
What I would like to know is if Android Studio allows to delete multiple unused declarations, and if it finds one unsafe usage, skip that unused declaration BUT keep going with the other ones.
In my project I have tons of unused declarations, but I can't delete many of them because there are some that trigger the warnings above. But most of them don't trigger any warning: I can browse my code files and find greyed out methods and I can safe delete them individually.
I have the latest version of Android Studio, 4.0.1
Answer: IntelliJ has several bugs regarding unsafe usages.
What I normally do is, after running the unused declarations inspections, I group the results by packages. Then, I try to delete all the unused usages of a particular package. I go through every package, one by one. That way, I minimize the number of unsafe usages found and can resolve them there at the spot.
Answer: This is a bug that has already been reported to JetBrains.
You can track it here and upvote it.
https://youtrack.jetbrains.com/issue/IDEA-247964


How to add prev page, and next page of posts on Wordpress?

Question: I want to add previous page and next page links on my footer, so I can limit each page to only 5 posts. I DON'T want next post, and previous post. I tried putting this in, and out of the WP loop.. but still not working!  I'm brand new to this WordPress stuff.  I already implemented next post and previous post for single.php.  I want to have previous post, and next post to be used on the index.php and other pages like /categories/, but NOT single.php.

            <?php posts_nav_link('','<','>'); ?>

You can find more articles at the archive.

Projects

iOS Betas

Recent Posts

                                <?php
                                    $args = array( 'numberposts' => '5' );
                                    $recent_posts = wp_get_recent_posts( $args );
                                    foreach( $recent_posts as $recent ){

                                        echo '
' .   $recent["post_title"].'
 ';

                                    }
                                ?>

What i'm working on

© 2013 Matthew Kosloski

Comment: What exactly are "previous page" and "next page" supposed to link to?
Comment: To the second page of the posts.  I want to limit the first page to have 5 WP posts, and the continue the rest on a second page.
Answer: in your admin settings > Reading (wp-admin/options-reading.php) change the posts per page count to 5
and for the links you're writing it wrong as per the codex
try this
<?php posts_nav_link('','<','>'); ?>

UPDATE:
You can add this if/else statement to show different on single or index
<?php if(is_single()) { // single-view navigation ?>

    <?php $posts = query_posts($query_string); if (have_posts()) : while (have_posts()) : the_post(); ?>

        <?php previous_post_link('%link', '<'); ?>  | <?php next_post_link('%link', '>'); ?>

    <?php endwhile; endif; ?>

<?php } else { // archive view navigation ?>

        <?php posts_nav_link('','<','>'); ?>

<?php } ?>

see more help here
Comment: Does it have to be in a loop?  Because It's not showing up.  Maybe because it's in the footer-home.php?  I have different footers for different pages, I don't know if that's effecting it.  I posted my footer-home.php code above.
Comment: So on line 129 I would change the value of 1 to 5?

 <?php _e( 'posts' ); ?
Comment:  |  outputs the names of the posts.  I just need to do something like this :: http://codex.wordpress.org/Next_and_Previous_Links and I need to put it inside the code above as footer-home.php
Comment: @MatthewKosloski updated again. you have to run a string in the function in order to get customized text different than the default


Unmarshalling objects with same name for multiple elements

Question: Sample xml file to unmarshalled
  <category>
    <id>abcat0010000</id>
    <name>Gift Center</name>
    <path>
      <category>
        <id>cat00000</id>
        <name>Best Buy</name>
      </category>
      <category>
        <id>abcat0010000</id>
        <name>Gift Center</name>
      </category>
    </path>
    <subCategories>
      <category>
        <id>pcmcat140000050035</id>
        <name>Capturing Photos & Videos</name>
      </category>
      <category>
        <id>pcmcat140000050036</id>
        <name>Listening to Digital Music</name>
      </category>
      <category>
        <id>pcmcat140000050037</id>
        <name>Computing Made Easy</name>
      </category>
      <category>
        <id>pcmcat140000050039</id>
        <name>Simple GPS Navigation</name>
      </category>
      <category>
        <id>pcmcat140000050040</id>
        <name>Playing Video Games</name>
      </category>
      <category>
        <id>pcmcat140000050041</id>
        <name>Watching HDTV</name>
      </category>
      <category>
        <id>pcmcat140000050042</id>
        <name>Enjoying Favorite Movies</name>
      </category>
      <category>
        <id>abcat0012000</id>
        <name>Him</name>
      </category>
      <category>
        <id>abcat0013000</id>
        <name>Teens</name>
      </category>
      <category>
        <id>abcat0014000</id>
        <name>Kids</name>
      </category>
    </subCategories>
  </category>

My java objects
 @XmlRootElement(name="category")
    public class Category {
        String id;
        String name;

        public String getId() {
            return id;
        }

        @XmlElement
        public void setId(String id) {
            this.id = id;
        }

        public String getName() {
            return name;
        }

        @XmlElement
        public void setName(String name) {
            this.name = name;
        }
    }

Relationship between both types of 'category' is little confusing
@XmlRootElement("category")
public class ExtendedCategory extends Category {
    /*String id;
    String name;*/
    List<Category> path;
    List<Category> subCategories;

    /*public String getId() {
        return id;
    }

    @XmlElement
    public void setId(String id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    @XmlElement
    public void setName(String name) {
        this.name = name;
    }
*/
    public List<Category> getPath() {
        return path;
    }

    @XmlElementWrapper(name="path")
    @XmlElement(name="category", type=Category.class)
    public void setPath(List<Category> path) {
        this.path = path;
    }

    public List<Category> getSubCategories() {
        return subCategories;
    }

    @XmlElementWrapper(name="subCategories")
    @XmlElement(name="category", type=Category.class)
    public void setSubCategories(List<Category> subCategories) {
        this.subCategories = subCategories;
    }
}

I'm getting exception that Category can not be cast to ExtendedCategory
Things work fine if I change top level xml element to something else (+ corresponding change in Extended category)
How to Unmarshalling objects with same name for multiple elements ?
Answer: @XmlRootElement(name = "category")
public class Category {

    String id;
    String name;
    List<Category> path;
    List<Category> subCategories;

    public String getId() {
        return id;
    }

    @XmlElement
    public void setId(String id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    @XmlElement
    public void setName(String name) {
        this.name = name;
    }

    public List<Category> getPath() {
        return path;
    }

    @XmlElementWrapper(name = "path")
    @XmlElement(name = "category", type = Category.class)
    public void setPath(List<Category> path) {
        this.path = path;
    }

    public List<Category> getSubCategories() {
        return subCategories;
    }

    @XmlElementWrapper(name = "subCategories")
    @XmlElement(name = "category", type = Category.class)
    public void setSubCategories(List<Category> subCategories) {
        this.subCategories = subCategories;
    }
}

I am trying
   Category c = JAXB.unmarshal(new File("yourXml"), Category.class);

and I get category with 2 pathes and 10 subcategories, like you need.
next:
JAXB.marshal(c, System.out);

and I get same XML like in your example.
This class works well. For child categories set field path to null and field subCategories to null


De Finetti's Coherence Principle and Frequentist interpretation

Question: So, without proof or citation, I often see that the Coherence Principle by de Finetti does not hold with Frequentist statistics.  It is pretty easy to create examples of this fact.  The exception would be where the Frequentist and the Bayesian results map to the same values everywhere.
Can someone cite a proof for me, or provide one as the general case?  I have looked and failed to find one in the literature.
EDIT Proof that if I were to gamble that $\mu\ge{5}$ and construct the odds using a Frequentist method, then there would exist cases where a Dutch Book could be constructed.  That would except the case where the Bayesian and Frequentist solution map to the same answer.
As to how do you apply coherence to Frequentist statistics, you don't.  However, econometrics does.  Finance is almost purely a Frequentist discussion.  Less than one percent of finance articles are Bayesian and they usually cover side cases.  Models like Black-Scholes, aside from being problem-ridden, are also Frequentist in construction.
I can show that in my specific case, there will arise cases where a Dutch Book can be constructed.  What I was hoping for was a general proof.
I had assumed the fact that the Dutch Book Theorem does not hold in finance was probably not a problem in real-world activity, but I am pretty certain that I was wrong.
Comment: A proof of what exactly?
Comment: "I often see..." Where does this statement come from? How do you apply coherence in frequentist statistics?
Comment: *Proof that if I were to gamble that μ≥5 and construct the odds using a Frequentist method, then there would exist cases where a Dutch Book could be constructed.* Frequentist and Bayesian methods make conditional probability statements (like percentage coverage intervals) based on different conditions. Both of them can be Dutch booked depending on how you set up the gamble and determine performance.
Comment: Some months ago I raised a technical objection to your view on the extreme fat tailedness of the distribution of stock returns. I explained (by referring to the trading rules at major stock exchanges and perhaps other issues -- this was so long ago I cannot remember) why stock returns cannot have such fat tails. But I never got a notification that you responded. I wonder if I just missed it, or did you not respond? If the latter, I am curious to hear your opinion on my criticism of your view.
Comment: @RichardHardy I remember you objecting.  What I do not remember is if I edited the answer or not.  I had a period where I was quite sick and, simultaneously, busy.  I will go backward and try and find it.
Answer: Savage has a result about any procedure for making decisions under uncertainty that satisfied a set of reasonable-looking conditions including coherence. He showed it implied the existence of finitely-additive beliefs over events and linear utilities over states of nature.  There's a description here
If you have well-defined priors over everything in the world I don't think it's controversial that you would update them using Bayes' Theorem, so this gets basically at the same point that non-Bayesian inference must fail to be coherent.
The 'Dutch Book' argument is related but slightly different: it says that if you identify degrees of belief in uncertain statements with willingness to take certain bets, and if the degrees of belief aren't Bayesian probabilities, someone can find a set of bets you will accept that has negative value with probability 1 and take all your money.
The problem with the Dutch Book argument is that it assumes everyone should always be willing to take one or other side of every offered bet. Since they clearly aren't, the claim that they rationally must be is a hard sell,  and one can argue that the whole problem is an artifact of identifying degrees of belief with willingness to bet. That's combined with the lack of any real evidence that frequentists are subject to Dutch Book tricks to siphon away all their money.
If you regard the betting part of the argument as just a metaphor and not really about betting or money you are basically back at Savage's argument.
Savage's argument is, I think, regarded as more persuasive, at least in an ideal sense. The reason it doesn't convince people that Bayesian statistics is the only way in practice, is that it assumes you are already able to make coherent decisions over all possible events (in order to have coherent priors). In that case you wouldn't need Bayesian statistics, you'd naturally just update your beliefs properly. Perfectly rational beings would automatically be Bayesian, but it doesn't necessarily follow that the best strategy for imperfect humans is to try to do formal Bayesian inference. It's a plausible supporting argument, but the proof isn't as airtight as people sometimes argue.
Answer: Frequentist inference can be (intra-experimentally) coherent without having the same results as a Bayesian. For a modern example, see:
https://arxiv.org/abs/2111.10715
Comment: Thanks, I guess I know what my weekend reading will be.


Explicit formula for zeta function

Question: The explicit formula for $\zeta(s)$:
$$
\psi(x)=x-\sum_{|\operatorname{Im}\rho|<T}\frac{x^\rho}{\rho}-\log(2\pi)-\log\left(1-\frac{1}{x^2}\right)+O\left(\frac{x\log^2T}{T}\right),
$$
where $\psi(x)=\sum_{p^k<x}\log p$, $x>1$, $\rho$ is a nontrivial zero of $\zeta(s)$, and the sum over $\rho$ is taken with multiplicities.
$\textbf{My question}:$ Is there any literature on the behavior of the equality when we take formal derivatives on both sides? I am looking for asymptotic results after taking the derivative on both sides.
Comment: I'm not quite sure what a formal derivative of the left hand side even means. The left hand side is discrete and locally constant, except at integers. What do you want the derivative of the left hand side to mean? I suppose one possibility would be to differentiate the Mellin representation of the left hand side, which would exist. But is that what you want?
Comment: @mixedmath: I don't have any specific representation in mind. Any sort of interpretation for the derivatives will work for me. Can you please point me to some references regarding the differentiation of Mellin representation that you have stated?
Comment: The derivative of $\psi(x)$ is a distribution, a sum of Dirac deltas, and the derivative of the RHS converges to this distribution **in the sense of distributions**. You should work on $\sum_{k=-\infty}^\infty \delta(x-k) = \sum_{n=-\infty}^\infty e^{2i \pi n x}$ which is the fundamental theorem of Fourier series encoded in term of distributions.
Answer: (Expanding on my comment)
The classical way of recognizing this explicit formula is to perform a careful analysis of the integral
$$
\psi(X) = \frac{1}{2\pi i} \int_{5 - i\infty}^{5 + i\infty} \left( - \frac{\zeta'(s)}{\zeta(s)} \right) X^s \frac{ds}{s}.
$$
The convergence of this integral is a bit delicate, but it may be possible to define $\psi'(X)$ by defining it to be the value of the $X$-derivative of the integral.
A simpler problem would be to try to understand the integral
$$
\frac{d}{dX} \frac{1}{2\pi i} \int_{5 - i\infty}^{5 + i\infty} X^s \frac{ds}{s}.
$$
This behaves very well away from integer values of $X$, but at integer values this requires some interpretation.

Explicit formula for zeta function


Sending form to e-mail

Question: I'm trying to send a form to a e-mail using AJAX.
Problem : I'm not reciving the e-mail.
Obs.: The form is in portguese just ignore the text part ;)
index.html
    ...

<form id="ajax_form" action="">

 Nome :

 <input type="text" required  name="nome" size="60">

 E-mail :

 <input type="text" required  name="email" size="60">

 Curso :

 <select name="cursos" style="width: 395px" class="auto-style1">
 <option>Sistemas de Informação</option>
 </select>

 Turno :

 <input type="radio" name="turno" value="1"></input>Manha
 <input type="radio" name="turno" value="2"></input>Noite

 Área de Interesse(max 2) :

 <input type="checkbox" name="interesse" /> CIO

 <input type="checkbox" name="interesse" /> Hardware

 <input type="checkbox" name="interesse" /> Sistemas Empresariais

            <input type="checkbox" name="interesse" /> Redes Sociais e Mobile

     <input id="Send" class="btn" type="submit" value="Enviar"></input>
     As Inscrições comecaram dia 26
     </form>
...

     ...

jqueryscript.min.js
...
$(document).ready(function() {
    jQuery('#ajax_form').submit(function(){
        var dados = jQuery( this ).serialize();

        jQuery.ajax({
            type: "POST",
            url: "sender.php",
            data: dados,
            success: function( data )
            {
                alert( data );
            }
        });

        return false;
    });
...

sender.php
<?php
    $nome = $_POST['nome'];
    $email = $_POST['email'];
    $curso = $_POST['cursos'];
    $turno = $_POST['turno'];
    $interesse = $_POST['interesse'];

    $mensagem = '';
    $mensagem .= '
Formulario :
';
    $mensagem .= '
';
    $mensagem .= '
Nome: 	' . $nome . '
';
    $mensagem .= '
Email: 	' . $email . '
';
    $mensagem .= '
Curso: 	' . $curso . '
';
    $mensagem .= '
Turno: 	' . $turno . '
';
    $mensagem .= '
Interesse: 	' . $interesse . '
';
    $mensagem .= '
';
    $mensagem .= '';

    $mail = new PHPMailer();
    $mail->SetFrom($email, $nome);
    $mail->Subject = 'Inscrição NUTI';
    $mail->MsgHTML($mensagem);
    $mail->AddAddress('firstname.lastname@example.com', 'Bruno');
    $mail->AddAddress($email, $nome);
?>

Thanks in advance :)
EXTRA : I want to alert the user that the form was sent. :)
Comment: You could use a tool like [Fiddler](http://fiddler2.com/) to monitor the requests and responses, that might help you to see where the problem is. With it you can check if the data actually gets sent or not.
Answer: If you don't $mail->Send() you will never get an email!
$mail = new PHPMailer();
$mail->SetFrom($email, $nome);
$mail->Subject = 'Inscrição NUTI';
$mail->MsgHTML($mensagem);

$mail->AddAddress($email, $nome);

if (!$mail->Send()) {
  // PHPMailer Error!
}
else {
  // Email sent!
}

BTW You may implement a command to echo the HTML-Email to the browser, so you can test and see how they will look like (without sending and waiting for emails).

Sending form to e-mail


Assemblies não encontrados para Visual Studio

Question: Olá, estou trabalhando com o Visual Studio a pouco tempo e é a primeira vez que me deparo com essa mensagem ao tentar depurar o programa:

"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(1126,5): error MSB3644: Os assemblies de referência para a estrutura ".NETFramework,Version=v4.5" não foram encontrados.
Para resolver isso, instale o SDK ou o Pacote de Direcionamento para essa versão da estrutura ou redirecione o aplicativo para uma versão da estrutura para a qual você tenha o SDK ou o Pacote de Direcionamento instalado. Observe que os assemblies serão resolvidos do GAC (Cache de Assembly Global) e serão usados no lugar dos assemblies de referência. Dessa forma, talvez seu assembly não seja direcionado corretamente para a estrutura pretendida."

Gostaria de saber o que eu precisaria fazer para poder compilar na versão que está o meu .NET Framework (se não me engano está na 4.7.0). Já tentei mudar o parâmetro de versão para 4.7.0 nos três arquivos .config que o programa possui, mas o compilador ainda parece referenciar a versão 4.5.
Desde já, agradeço pela ajuda.
Comment: Tentou referenciar automaticamente outro .NET Framework (como o v4.6) nas configurações do projeto?
Answer: Já tive o mesmo problema.
Navegando até a pasta bin, você pode checar a linha 1126 do arquivo `Microsoft.Common.CurrentVersion.targets. Logo vc se depara com:

    <GetReferenceAssemblyPaths
        Condition="'$(TargetFrameworkMoniker)' != '' and ('$(_TargetFrameworkDirectories)' == '' or '$(_FullFrameworkReferenceAssemblyPaths)' == '')"
        TargetFrameworkMoniker="$(TargetFrameworkMoniker)"
        RootPath="$(TargetFrameworkRootPath)"
        BypassFrameworkInstallChecks="$(BypassFrameworkInstallChecks)"
        >
      <Output TaskParameter="ReferenceAssemblyPaths" PropertyName="_TargetFrameworkDirectories"/>
      <Output TaskParameter="FullFrameworkReferenceAssemblyPaths" PropertyName="_FullFrameworkReferenceAssemblyPaths"/>
      <Output TaskParameter="TargetFrameworkMonikerDisplayName" PropertyName="TargetFrameworkMonikerDisplayName" Condition="'$(TargetFrameworkMonikerDisplayName)' == ''"/>
    </GetReferenceAssemblyPaths>

Provavelmente ao tentar compilar sua aplicação, o visual studio não
  esta encontrando as referencias para tal Assembler, Note o comentário
  antes da tag GetReferenceAssemblyPaths que diz q caso ele não
  encontre tal referencia, ele deve assumir os que estão dentro da pasta
  C:\Program Files\Reference Assemblies\Microsoft\Framework. se você
  está fazendo referencia a um Assembler que não existe logo a IDE vai
  gerar a Exceção.

Para mudar o Estrutura de Destino do projeto você pode:

Click com direito sobre o projeto e selecione o SubMenu    Propriedades.

Para usar a versão que não existe, basta efetuar o downlaod do Framework em questão?

Pode ser direto do site .NET SDKs for Visual Studio

Ou então dentro da própria IDE:


Difference in log-rank, tarone-ware, gehan-breslow, Peto-Peto etc

Question: Everything is already in the title.
I study survival analisys
Could you tell me what is the difference among log-rank, tarone-ware, gehan-breslow, Peto-Peto etc....
In which case more suitable log-rank or tarone-ware etc
Thanks
Answer: Klein and Moeschberger's text "Survival Analysis" (p 195-197) has a nice comparison of these tests and a few others. They differ in the weighting of events along the course of the study: The Gehan test puts high weights on early deaths. Peto-Peto still weights early deaths higher but not quite so severely. Tarone-Ware is intermediate between the Gehan and Peto-Peto. The log-rank test has equal weighting. You can also construct various Fleming Harrington tests with higher weights at later times.


Como redirecionar formulário a um código de verificação?

Question: Quando o usuário clicar em "Cadastre-se", o serviço pede que você digite as palavras de verificação de segurança. Se você não conseguir ler o que está escrito, pode clicar no link "Tente palavras diferentes", ou pode ouvir as palavras, clicando em "um captcha sonoro".
Answer: Captcha (por Wikipedia)

CAPTCHA é um acrônimo da expressão "Completely Automated Public Turing test to tell Computers and Humans Apart" (teste de Turing público completamente automatizado para diferenciação entre computadores e humanos): um teste de desafio cognitivo, utilizado como ferramenta anti-spam, desenvolvido pioneiramente na universidade de Carnegie-Mellon. Como o teste é administrado por um computador, em contraste ao teste de Turing padrão que é administrado por um ser humano, este teste é na realidade corretamente descrito como um teste de Turing reverso.

Um exemplo de captcha com reCAPTCHA

Link para requisição de chave para usar a API

Exemplo de formulário HTML com o trecho:
<form method="post" action="verify.php">
    <?php
        require_once('recaptchalib.php');
        $publickey = "your_public_key"; // cadastre-se no link informado acima
        echo recaptcha_get_html($publickey);
    ?>
    <input type="submit" />
</form>

Trecho para validar do lado do servidor:
<?php
  require_once('recaptchalib.php');
  $privatekey = "your_private_key"; // observe a imagem em anexo.
  $resp = recaptcha_check_answer ($privatekey, $_SERVER["REMOTE_ADDR"], $_POST["recaptcha_challenge_field"], $_POST["recaptcha_response_field"]);

  if (!$resp->is_valid) {
      // seu fluxo caso o usuário tenha errado o captcha
  } else {
      // seu fluxo caso o usuário tenha acertado o captcha
  }
?>

Informações adicionais depois que se cadastrar para uso da API:

Fonte e documentação adicional
Uma curiosidade sobre o reCAPTCHA (offtopic, por Wikipedia)

O serviço reCAPTCHA fornece, para os sites inscritos, imagens de
  palavras que o reconhecimento óptico de caractéres (OCR) software não
  foi capaz de identificar. Estes sites inscritos (os quais seus
  propósitos não são geralmente relacionados à ajuda do projeto de
  digitalização de livros) apresentam estas imagens para humanos
  decifrarem como palavras CAPTCHAs, como parte do seu procedimento
  normal de validação. Depois eles retornam os resultados para o serviço
  reCAPTCHA, que envia esses resultados para a digitalização de seus
  projetos.


Proving an inequality concerning arbitary complex numbers

Question: $\def\abs#1{\left|#1\right|}$If $y$ and $z$ are any complex numbers then prove that
\[
 2 \abs{y+z}\ge \bigl(\abs y + \abs z\bigr)
\abs{\frac y{\abs y} + \frac z{\abs z}} \]
Comment: Perhaps one of the pluses on the right-hand side should have been a multiplication?
Comment: @Henning Makholmes:- Yes , multiplication it is.
Answer: $\def\abs#1{\left|#1\right|}$If $y=r(\cos A+i\sin A), z=R(\cos B+i\sin B)$
$|y+z|=\sqrt{r^2+R^2+2Rr\cos(A-B)}$
$$ \abs{\frac y{\abs y} + \frac z{\abs z}}=\sqrt{2+2\cos(A-B)}$$
$2|y+z|$ will be $\ge (|y|+|z|)(\abs{\frac y{\abs y} + \frac z{\abs z}}) $
if $2\sqrt{r^2+R^2+2Rr\cos(A-B)} \ge (r+R)\sqrt{2+2\cos(A-B)}$
if $4(r^2+R^2+2Rr\cos(A-B)) \ge (r+R)^2(2+2\cos(A-B))$
if $2(1-\cos(A-B))(R-r)^2\ge 0$ which is true
Comment: You mean y=r(cosA+isinA) , z=R(cosB+isinB) ?
Comment: @Souvik, sorry for the missing parenthesis, rectified.
Answer: The inequality is not true in general. Let $y=1, z=-1$ we have
$$
2|y+z|=0<2=(|y|+|z|)+\left|\frac{y}{|y|}+\frac{z}{|z|}\right|
$$


Covariance between two random variables with different number of values?

Question: Let's say I have a RV $X$ with values $100$, $200$ and their associated probabilities, and some RV $Y$ with values $35$, $47$ and $862$ with associated probabilities.
What does it even mean to find the covariance of those Random Variables? Looking at the formula it looks like we're doing $$(x_1 - \mu_x) (y_1 - \mu_y) P(x_1 \cap y_1) + (x_2 - \mu_x) (y_2 - \mu_y) P(x_2 \cap y_2) + ... + (x_n - \mu_x) (y_n - \mu_y) P(x_n \cap y_n)$$.
How does that make sense if the random variables have different numbers of values?
Comment: Where did you get that formula?  Of course you need to know the joint probability of all combinations of values from X and Y in order to compute their covariance. So basically, you will have a double summation, and the number of terms (variable values) in each sum need not be the same.
Comment: @MarkL.Stone, sorry for the late reply and thanks for the clarification. The formula was in the prof's slides. That's why I was confused because there was only a single summation symbol and a single index, so it didn't really make any sense if we assume different data set sizes.
Answer: A more typical way to write out the covariance of two discrete random variables would be:
\begin{align}
Cov(X, Y) = E[(X - \mu_X)(Y - \mu_Y)] = \sum_x \sum_y (x - \mu_X)(y - \mu_Y)P(X=x, Y=y)
\end{align}
In your case you would get:
\begin{align}
(100 - \mu_x)(35 - \mu_y)P(X=100, Y=35) +
(100 - \mu_x)(47 - \mu_y)P(X=100, Y=47) +
\dots
+ (200 - \mu_x)(862 - \mu_y)P(X=200, Y=862)
\end{align}
So in short I think that what you are missing is that you should be summing over all possible pairs of outcomes, giving you $3\times 2 = 6$ things to sum over.
Comment: Thank you for explaining. In our slides there was only a single summation symbol and a single index, so I couldn't make sense of it in the context of data sets of different sizes. It makes sense now.


Uncaught TypeErrors have me baffled

Question: This is my first Drupal 8 site but I have been building Drupal sites since Drupal 6. I am getting lots of Uncaught TypeErrors in the JavaScript console that I can't figure out. I have spend hours searching and have not found an answer. What am I missing?

I am developing a custom theme. The site is on Mamp. Any insight would be helpful.
Comment: I think I am narrowing down the problem. I started a new theme from the start and am copying files from the current theme and when I copied the page.html.twig file the JavaScript errors appeared. I just have to find what caused the problem.
Answer: It seems like all js variables are missing, that are in:
<code><script type="application/json" data-drupal-selector="drupal-settings-json">{...}</script>
</code>
This json array is on every page drupal delivers. You can check this on a fresh drupal install.
There can be a problem with your theme, probably with the html template.
Comment: If I switch to a core theme, the errors go away. What should I look for? Should I be looking in html.html.twig or THEMENAME.libraries.yml?
Comment: I checked the page source and the js variables appear to be loaded. The custom theme prints the
``
Comment: First check if the json array is in the html, if not check the html.html.twig. Second check if drupalSettingsLoader.js is loaded, if not check the dependencies in *.libraries.yml, look for core/drupal or core/drupalSettings.
Answer: I think I found the answer. I believe the page.html.twig file I was using had a hidden character that was causing problems. I am using the same code in a new theme and everything is fine, but I had to type the code again instead of copying it. When I copied it, the errors returned. It helped to test each block of code at a time with code that I knew worked.

Uncaught TypeErrors have me baffled


nesting vs not nesting?

Question: I am following the tutorial from Bodo Winter.
Bodo created the following model on page 10:
politeness <- read.csv("http://www.bodowinter.com/uploads/1/2/9/3/129362560/politeness_data.csv")
lmer(frequency ~ attitude + gender + (1|subject) + (1|scenario), data=politeness)

However, it appears that scenario is nested within subject, so I would use (1|subject/scenario) instead. Am I misunderstanding the data/experiment?
Comment: I can't reproduce the example, because the dataset referenced in the doc gives a 404 error. http://www.bodowinter.com/tutorial/politeness_data.csv
Comment: @JeremyMiles I updated my question with the link to the data file.
Comment: I don't see this as a stats question, but don't know the rules for on/off topic here :)
Answer: According to:
> xtabs(~ subject + scenario, politeness)

       scenario
subject 1 2 3 4 5 6 7
     F1 2 2 2 2 2 2 2
     F2 2 2 2 2 2 2 2
     F3 2 2 2 2 2 2 2
     M3 2 2 2 2 2 2 2
     M4 2 2 2 2 2 2 2
     M7 2 2 2 2 2 2 2

we can see that every level of subject appears in every level of scenario and vice versa. Thus, with the way the data are coded, these are crossed random effects. So, the model:
lmer(frequency ~ attitude + gender + (1|subject) + (1|scenario), data=politeness)

would appear to be appropriate.
Comment: Sorry for the delay, yes it does answer my question perfectly, thank you very much!

nesting vs not nesting?


QTPLUGIN, what's the purpose of it?

Question: While studying the Plug & Paint Example I've noted there is no QPLUGIN in the plugandpaint.pro file.
In the Static Plugins section of the "How to Create Qt Plugins", I read "To link plugins statically, you need to add the required plugins to your build using QTPLUGIN."
So, what's going on here?
Experimentally, I commented Q_IMPORT_PLUGIN(BasicToolsPlugin) in the main.cpp and added QTPLUGIN += pnp_basictools to the .pro file, as a result no plugins where loaded.
Answer: The documentation seems to be a bit misleading. This is my understanding of the QTPLUGIN qmake parameter:
QTPLUGIN can be used to specify a list of names of Qt platform plugins that are to be linked with an application when linking against static build of Qt. Examples of such plugins are image formats like qgif, qjpeg, qsvg, database support like qsqlite, qsqloci, qsqlpsql, some language codecs.
The variable has no effect when linking against a dynamic build of Qt.
Comment: @KubaOber, do you agree?

QTPLUGIN, what's the purpose of it?


$X$ local compact metric space contains countable dense subset $\Rightarrow X=\cup_{j=1}^{\infty} K_j$, $K_1\subset K_2\subset…$ compact

Question: Let $X$ be locally compact metric space, that is every point is surrounded by a compact neighbourhood. Show that

If $X$ contains a countable dense subset $T$, then there are compact subsets $K_1\subset K_2\subset…$ so that $X=\cup_{j=1}^{\infty} K_j$

Let $t_i$ be elements of $T$, since $X$ is locally compact there is $\epsilon>0$ so that $\overline{B_{\epsilon}(t_i)}$ is compact and $X=\cup \{\overline{B_{\epsilon}(t_i)}|\epsilon\in \mathbb{Q}\}$. But how can I create an increasing sequences of compact sets?
Comment: Do you only consider metric spces?
Comment: @PaulFrost he must be otherwise it is false.
Comment: @PaulFrost The rational sequence topology (or Mrówka $\Psi$) are Tychonoff separable locally compact and not $\sigma$-compact.
Comment: There need not be one $\varepsilon$ that works for all $t_i$ at the same time..And if we have a different $r_i$ per $t_i$ the claim that the union is $X$ is possibly false.
Answer: If we're in a metric space, a countable dense set means there is a countable base.
This means there is also a countable base $\{B_n\}$ so that $\overline{B_n}$ are all compact (follows from this).
Then apply the fact that finite unions of compact sets are compact to easily modify this to an increasing sequence of compacta.
Comment: I take $K_j=\cup{n=0}^j \overline{ B_n}$, why is then $X=\cup_{j=0}^{\infty} K_j$?
Comment: @user30523 if $K_j = \bigcup_{i=0}^j \overline{B_i}$ then the union of the $K_j$ the same as those all those closures hence equal to $X$.
Comment: $\{B_n\}$ countable base of $X$ means that $X=\cup_n B_n$?
Comment: @user30523 so certainly the closures too.
Comment: why can I find a base so that the closures are compact?
Comment: @user30523 that's a standard fact.


Why would I ever want to use `async def` over `@asyncio.coroutine`?

Question: Python 3.5 greatly expanded support for asynchronous programming with a new function definition syntax. Whereas async functions were previously just "generators with benefits":
def generate_numbers():
    """
    Generator function that lazily returns 1 - 100
    """
    for i in range 100:
        yield i

generate_async = asyncio.coroutine(generate_numbers)
generate_async.__doc__ = """
    Coroutine that lazily returns 1 - 100
    This can be used interchangeably as a generator or a coroutine
    """

they now have their own special declaration syntax and special behavior by which they are no longer usable as usual generator functions:
aysnc def generate_async_native():
    """
    A coroutine that returns 1 - 100
    This CANNOT be used as a generator, and can ONLY be executed by running it from an event loop
    """
    for i in range(100):
        await i

This is not a question about the functional or practical differences between these types -- that is discussed in this StackOverflow answer.
My question is: why would I ever want to use async def? It seems like it provides no additional benefit over @asyncio.coroutine, but imposes an additional cost in that it

breaks backward-compatibility (Python 3.5 code with async def
won't even parse in older versions, although this is arguably a feature and not a bug) and
seems to provide less flexibility in how the function can be called.
Comment: I removed the `syntax` tag and added the `python` one. In general, you should add the python tag to all python questions since that is the tag that most people keep track of.
Answer: One possible answer is given by Martijn Pieters:

The advantages are that with native support, you can also introduce additional syntax to support asynchronous context managers and iterators. Entering and exiting a context manager, or looping over an iterator then can become more points in your co-routine that signal that other code can run instead because something is waiting again

This has in fact come to fruition with new async with and async for syntax, which cannot be as easily implemented with a "tack-on" solution like a decorated generator.


Logged into Ubuntu as domain user, asks for first local user password on admin tasks

Question: I have an odd issue that happens on all of the Ubuntu machines I join to my Samba4 domain.
When installing, I create a user named user and then use that user to get the domain going. Everything goes smoothly after domain join, and I can login with first.last and password to the machine.
However, when logged in, if I try an administrative action in GNOME, such as opening a VPN for all users, it asks for the password for user vs first.last (or even root). How can I adjust this?
Thanks!
Answer: Gnome uses PolKit for authorization checks. By default root and the group sudo is considered as admins.
You just need to add first.last to the sudo group:
adduser first.last sudo

Answer: In fact adding domain users in sudoers is not enought as it will ask for local admin account for administrative tasks such as installing packages throught the GUI. It's not convenient to do everything in sudo command line.
So you have to edit the etc/group file and add the domain user in the "adm" group just like that :
adm:x:4:syslog,somelocaladmin,firstname.lastname@example.com


Auto CC on outbound SMTP mail

Question: Is there a way to have IIS SMTP mail server automatically CC all mail it sends to a fixed email address?  Now, when Chilkat mail sends messages via SMTP, there is no record of the sent messages.  We would like to CC a common mailbox with all these messages to create an easy to use log of the messages that have been sent.  Any ideas would be appreciated.
Answer: Doesn't the IIS6 logfiles contain records of sent email? You may be able to enable extended logging or diagnostic logging in IIS6? I don't have a server handy, or I'd try it out.. :)

Auto CC on outbound SMTP mail


JavaFX BorderPane will not take on background colour

Question: I'm trying to teach myself basic JavaFX by following the tutorials provided by Oracle.
In the BorderPane tutorial (https://docs.oracle.com/javafx/2/layout/builtin_layouts.htm) it specifies a background colour.
This is a snippet of my code:
/**
 * This Method creates and defines a horizontal box with a button.
 */
public HBox addHorizontalBoxWithButton() {
    // set up horizontal box and button
    HBox hBox = new HBox();
    hBox.setPadding(new Insets(10, 10, 10, 10));
    hBox.setSpacing(10);
    hBox.setStyle("-fx-background-colour: #FFFFFF;");
    hBox.setAlignment(Pos.CENTER);
    Button startButton = new Button("CLICK ME");
    startButton.setPrefSize(100, 30);
    // set up a message
    Text message = new Text("Click the button to get started.");
    message.setId("message");

    hBox.getChildren().add(message);
    hBox.getChildren().add(startButton);

    return hBox;
}

I have tried various different background colours, none of which work. Am I missing something here?
Also, I am using a .css file but it only adds style to the 'message'.
Answer: The only problem with the original code is that you have a "typo" (anglification?) in your style setting. It should be
hBox.setStyle("-fx-background-color: #FFFFFF;");

not
hBox.setStyle("-fx-background-colour: #FFFFFF;");

Using an external style sheet with
#hbox {
    -fx-background-color: red ;
}

is a better solution than using inline styles.
Answer: Okay I just solved this.
I changed my code like so:
/**
 * This Method creates and defines a horizontal box with a button.
 */
public HBox addHorizontalBoxWithButton() {
    // set up horizontal box and button
    HBox hBox = new HBox();
    hBox.setId("hBox");
    hBox.setPadding(new Insets(10, 10, 10, 10));
    hBox.setSpacing(10);
    // hBox.setStyle("-fx-background-colour: #FFFFFF;");
    hBox.setAlignment(Pos.CENTER);
    Button startButton = new Button("CLICK ME");
    startButton.setPrefSize(100, 30);
    // set up a message
    Text message = new Text("Click the button to get started.");
    message.setId("message");

    hBox.getChildren().add(message);
    hBox.getChildren().add(startButton);

    return hBox;
}

And I added this to the .css file:
#hBox {
-fx-background-color: linear-gradient(#04B45F, #81F79F);
}


Accelerando/Ritardando duration?

Question: As I reach to the end of a piano piece, I see a ritardando right here,

My question is how long an accel. or rit. last?
Looking at one webpage, it says the ritardando can be extended by a dashed horizontal line. I'm assuming the accel. or rit. only lasts for the measure it's marked on, but I have found nothing online that says so also.
Comment: " I'm assuming the accel. or rit. only lasts for the measure it's marked on" IIRC some notation software apps used to interpret it that way (and maybe they still do) - but that's just what some nameless computer programmer thinks it means, and the answers below are better advice.
Answer: The abbreviation rit. means either ritardando (slowing down) or ritenuto (immediately slower). (This ambiguity is why i never use the abbreviation rit.)
In the former case, the musical context would suggest slowing down over the remaining four bars, but as dwilli mentions your interpretation of the music will inform you. Equally, your interpretation may lead you to interpret the rit. as a ritenuto — slowing the tempo immediately — in which case you would maintain the new tempo until the end.
A rule of thumb that you can use here is that any tempo change lasts until it is countermanded. A rit., rall., allargando, accel., stringendo, etc. will last until the end of the extension line or until the next tempo marking: a tempo/Tempo Primo/etc. or setting a new tempo.
Comment: @JohnCollins I use _rall./rallent./rallentando_ or _allarg./allargando_ for gradual slowing of tempo & _riten._ as the the abbreviation for _ritenuto_. Whether to use the abbreviations or not depends on the duration of the change & the available space in the score.
Comment: That is the rule of thumb, though note that classical pieces do not always follow it.  For example, _ritard._ can be used without explicit _a tempo_ to indicate a momentary slowing of tempo.  This occurs in Schumann's Blumenstück, for instance.
Comment: (Lest I cause confusion, I am commenting only on the "rule of thumb" itself.  I do agree this particular _rit._ is probably intended to extend to the end of the piece.)
Comment: That's why I qualified it as a "rule of thumb" (https://en.m.wikipedia.org/wiki/Rule_of_thumb)
Comment: What do you use instead of _`rit.`_? You mean the whole words "_ritardando_" or "_ritentuo_"? Or, some of the other options entirely? Personally I'd argue ritardando is simply too commonly used a word to be legitimately used taking up so much precious notation real estate (i.e., too long a word for such a frequently required presence - Rachmaninoff or esp, Scriabin, [piano] sheet music comes to mind)
Comment: Right, so if it's ever "rit", let's say the world of [piano] music notation were perfectly idealized (obviously not quite the case; though we've arrived at something which works well enough!), but assuming standardization and simplification were the goals at hand -- as you allude to, one can continue to use "rit." and know that it means ritardando because it would be "riten." otherwise. Whereas rallentando is just another word for the exact same thing and less brief than "rit." (which is, so nicely, brief). Then there's e.g. Debussy using French words no one who doesn't speak French knows etc.
Comment: @JohnCollins Fair enough, but I would never use _ritardando_ because I think it's an ugly word. _de gustibus non diputandum._ Typographically, "rit." & "rall." differ in size by about an en-width in most cases: a small price for greater clarity. That's why I make those choices in my scores. As for the vernacular, i tend to agree that it's overused. I did an arrangement recently where I replaced French instructions with more accessible Italian instructions.
Comment: @DeanRansevycz You know, I did look more into this (e.g. http://forum.pianoworld.com/ubbthreads.php/topics/1984748/1.html). I would have to concede there is _some_ difference to _rall._ (vs. _rit._) o.c. but I personally would like to see _rit._ used more often just for the sake of improving standardization. Ultimately there is some (intentionally un-noted) rubato (personal musicality) to every performance/interpretation & yes I agree completely (_ritardando_ should _**always**_ be rit.); the whiter the score the better, e.g. do we not ensure to minimize accidentals? etc. & yes clarity rules
Answer: It's this kind of situation where you have to become a musician and not just be a player. When the instructions are vague you are given license to interpret them. Slow down in a way that expresses your musical understanding of the piece. Rely on your gut to tell you just how. You might slow down all the way to the end or you might slow down for two measures, depending on what sounds best to you. You might vary the deceleration of your speed if it suits the music.
Answer: Coming to the end of the piece, I'd rit. gradually to the last note. Often rit. or accel. is followed by 'a tempo' when the original speed needs to be re-applied. That bar 67 sounds weird, and possibly is named wrongly!
Comment: Thanks for your feedback! For measure 67 I tried looking up the name of the chord and Cmaj7sus was the first thing a found. I decided to research more and I think it's a C Major 7th chord (no 3rd).
Comment: The C G B could be Cmaj7, but A isn't a sus note - D and F are those.
Comment: Sus means without the 3rd, so it's right, but it believe is should be followed by something. Often sus chords without 2nd or 4th are called sus5, and the A in the melody is a 6th. So the right name could be Cmaj7(13)sus5?  Is this in the right order?
Comment: @coconochao - sus means 3 is replaced by either 2 or 4 - the 3 is suspended. Never heard sus 5, although a 'chord' with only 1 and 5 is known as a 'power chord'. I really couldn't name it meaningfully.
Comment: Instead of *a tempo* also *tempo primo* or a completely new tempo indication would terminate the modification.
Comment: @Tim Yes, for me sus5 is a power chord. Maybe not in american notation?
Comment: @coconochao  - Never heard it called anything but '5' or 'power chord' , mainly by guitarists. American notation, don't know. Never 'sus5'.
Comment: @Tim you are probably right, I must have got confused.
Answer: The composer of this piece was fond of writing A7 chords with a Db instead of a C# - I don't think we can find any excuses for this not just being 'wrong'.  He was not a meticulous notator.  So I wouldn't agonise too much over his precise intention with the rit.  He MIGHT have known the distinction between ritardando (slowing down) and ritenuto (immediately slower), and that ritardando is meaningless without an extention line, or perhaps a 'rit. al fine' to indicate duration.  Do whichever you find most effective.
That chord isn't Cmaj7sus.  If you wanted to include the A in the harmonic analysis I suppose it COULD be called Cmaj7(no3rd)(add6).   But it's really just an incomplete Cmaj7 with an A in the melody.  You ignore the A in your analysis of the preceding chord, I think you can ignore it again.
(The density of that passage shifts between 7, 6, 5 and 4 note chords.  Do you find that effective, or a weakness?)

Accelerando/Ritardando duration?


Why the IPL teams don't have any English cricketers?

Question: I can't see any England cricketers playing in this IPL season(2014). The only England cricketer playing in this IPL is Kevin Pietersen who was axed by England Cricket Board.
What is the reason for the IPL teams not to buy English players?
I remember some of the English players like Luke Wright, Morgan played in some IPL seasons before.
Comment: How does "relegations" apply to this question?
Comment: I would also like to know the significance of `relegations` in regards to this question as there is an [ongoing meta discussion](http://meta.sports.stackexchange.com/questions/505/should-promotions-and-relegations-tags-be-combined) about combining the tag with `promotions`.
Answer: There is no particular reason for the exclusion of English players from the teams; if that were the case, they wouldn't have been up for auction which they clearly were.
Players like Alex Hales were up for auction and weren't bid on by the franchises.
There can be a number of reasons for this:

Balance of the side, the fact that only four foreign players are allowed in a game. (Shaun Marsh had been prolific for KXIP but he was benched for the whole season)
Unavailability of players. If you look at the teams, apart from big names like Malinga and Perera (big names in T20 atleast), there wasn't much Sri Lankan participation either. This was because of the recently concluded Sri Lanka-England ODI series.

In a total of 8 teams, with 4 foreign players each, you have 32 international playing spots. Countries vying for these spots are 8 (excluding India and Zimbabwe and assuming only players from the top 10 ICC rankings are taken in the auction; Ryan ten Doeschate is an exception).
8 countries, 32 spots. Australia have the Big Bash League which brings out T20 talent, and similarly West Indies does. There are numerous of examples of big names originating from these tournaments and making it big, but hardly any such examples from England.
The Indian franchises are more willing to spend their money on current International players and those whose performance they see in local T20 leagues.
As for KP, he is recently 'retired' and is a huge name with a reputation of clearing the boundary, hence, he was picked up.
Many other big names like Jayawardene and Sanga weren't picked up just like Hales.
I don't see any official or discriminatory reason of any sort if that was a part of your query.
Answer: I doubt there is one single reason although the overlap of the IPL and the English domestic season has been a major contributing factor.  Many higher profile (centrally contracted) English players would not be available for the whole tournament.  Its likely that other players such as Alex Hales and Samit Patel who would be available for the whole tournament were not viewed as good value for money as their county sides would only release them from the domestic contract if they achieved a certain price in the auction.


How to plot stacked bars using hist()?

Question: I  trying to plot a stacked bar chart as part of a plt.subplots.
I currently have:
bin_n=100
df_name['column_name'].hist(bins=bin_n, ax=axs[0,1])

and I would like to add a categorical (string) column in the game.
My data set is like:
string_column  column_name
'A'                4.3
'A'                4
'B'                4.8
'B'                5
'B'                6.8

I've read a lot of examples of stacked bars but all of them are using arrays.
So my questions are:
Do I need to transform my dataset?
Can I apply stacked bars using hist() as above? If yes, how exactly?
Comment: What's your expected outcome? Do you want to stack per histogram or per category?
Comment: I would like stack per category.
Answer: df_name.pivot(index='column1', columns='string_column', values='column_name') \
    .plot(kind='hist', stacked=True)

Comment: Your answer may be the right solution, but Please add some explanation for the current and future developers facing similar questions. Remember, stackoverflow is a community place for answers to questions with explanation.

How to plot stacked bars using hist()?


How to distinguish between systemd unit run manually or by timer?

Question: I want to distinguish between systemd unit run by systemd timer or invoked manually with systemctl. I.e. to check, if backup is triggered as periodical job by schedule or is done "on demand".
What I've found so far:

there is no place for setting environment variables in timer units ?)
environment passed to unit is the same, regardless if it is triggered by the timer or manually

Running both periodic and manual invocations through systemd seems to be more convenient than putting note in MOTD "For scheduled operations systemd timers are used, for manual jobs use scripts in /usr/local/bin" ;)
Do you have any ideas (other than unit instantiation and copying unit with different environment for manual use )?
Comment: `systemd.timer` triggers a `systemd.service` unit, so you can use the `Environment` directive in the service file. Or install the `systemd-cron` package, to emulate the good old crontab.
Comment: Yes, at the moment I've found two resolutions: 1: copy systemd unit and set environment variables (so one more unit to maintain/remember about and invoke different unit for manual and scheduled operation) or 2: use "good old cron", where I can modify environment or just use additional parameter in command line.

So "Great shining systemd" vs. "Good Old Unix KISS" once again - "at least I tried" ;)
Answer: but here is a simple way to pass argument from your timer to your unit, and that is use templates
example
create a service with @ i the name
/etc/systemd/system/me@.service
   [Unit]

   Description=test

   [Service]
   ExecStart=/bin/echo 'hello from %i'

then your timer can be something like
/etc/systemd/system/me.timer
[Unit]
Description=timer for me

[Timer]
OnUnitActiveSec=10s
OnBootSec=10s
firstname.lastname@example.com

[Install]
WantedBy=multi-user.target

you see that now whatever you pass after the @ will become %i in your service file. so now you just
[~] sudo systemctl daemon-reload
[~] sudo systemctl start me.timer
[~] sudo systemctl status email@example.com
● firstname.lastname@example.com - test
   Loaded: loaded (/etc/systemd/system/me@.service; static; vendor preset: enabled)
   Active: active (running) since Thu 2018-02-08 19:35:38 PST; 7ms ago
 Main PID: 8899 (echo)
    Tasks: 1 (limit: 4915)
   CGroup: /email@example.com

Feb 08 19:35:38 algx systemd[1]: Started test.
Feb 08 19:35:38 algx echo[8899]: hello from timer

if you want to start the unit from the cli, just give it another param after the @
[~] sudo systemctl start me@cli
[~] sudo systemctl status email@example.com
✘  ~  sudo systemctl status firstname.lastname@example.com
● firstname.lastname@example.com - test
   Loaded: loaded (/etc/systemd/system/me@.service; static; vendor preset: enabled)
   Active: inactive (dead)

Feb 08 19:32:48 algx systemd[1]: Started test.
Feb 08 19:32:48 algx echo[8490]: hello from cli

this way you have a clear distinction on what is timer and what is cli, but you can expand this to any type of trigger... you dont maintain 2 units files, but you have a clear separation because from the point of view of systemd this are 2 different services.
Comment: It is exactly what I mean writing about "unit instantiation" - and it is already used to distinguish between multiple instances of service (choosing right environment file).
Is it possible to parse %i inside unit file (i. ie. having units firstname.lastname@example.com and firstname.lastname@example.com pass to the further processing just "instance1")?
Comment: sadly no, that is up to you app, there is more info in https://www.freedesktop.org/software/systemd/man/systemd.unit.html but in a nutshell, the thing after the `@` is called the instance name, and its just one thing.


Questions on SVM - Coordinate Ascent and Support Vectors

Question: I was reading up on the dual formulation of SVMs and have some questions to ask. https://github.com/epfml/ML_course/blob/master/lectures/07/lecture07a_svm.pdf

1) Why is it easy to optimize via coordinate ascent ? Is it because $\alpha$ is bounded between [0,1] ? If it is not because of its range of [0,1], then what about logistic regression ? Why does no one talk about coordinate descent for logistic regression ? Can alpha only be 0 or 1 ? Can it take any other values in between ?

2) From the diagram and also point no. 2, $\alpha_2 \subset (0,1) $. When would it be zero and when would it be 1 ? If it lies exactly on the margin then wouldn't that make $\max_{\alpha}\alpha_2[1 - y_2f_2] = 0 $ since it is correctly classified ? i.e. $f_2 = 1$
Answer: I think the simplest way to answer this is that it is easy to optimize via coordinate ascent because you only have box constraints (this is because you are solving for the unbiased version of SVM which doesn't have the constraint $\sum_i \alpha_i y_i = 0$). Thus, you can optimize for each $\alpha_i$ separately and not worry about any constraint violation (i.e., optimize for $\alpha_i \in [0,1]$). If you had the biased version, then for example SMO by Platt could be used which chooses a pair for which to optimize at each stage (and maintains the bias constraint throughout). $\alpha$ can be any value between 0 and 1, in particular for vectors on the margins (technically, if $\alpha_i$ is between 0 and 1, then $x_i$ must be on the margin).
You can also solve logistic regression using coordinate descent; from an optimization perspective there are additional questions around whether there are closed-form solutions to solving for a single variable at a time, or whether optimization needs to be done to solve or approximately solve. There are numerous methods which solve using for example dual coordinate descent. Here is one such method, which also discusses solving via updates to a single variable.


nginx url rewrite

Question: How to nginx rewrite within location context the following:
/ => /myurl/
/abc/ => /myurl/abc/
Answer: the following will do the trick:
location / {
    rewrite ^ /myurl/$uri;
}

Note that the /abc/ wil match the / prefix (and be rewritten to /myurl/abc/), so it doesn't need a sepperate rewrite
Also note that you need to add an additional "location /myurl/ {...}" block to avoid getting incoming request to that url to get rewritten to /myurl/myurl/... and so on
Comment: cobaco can you elaborate on the second part to prevent unnecessary addons? I think I am running into issues because of that and I'm not sure how to go about it.
Comment: basically: if you go to /myurl/ then at that point $uri = /myurl/. That gets rewriten to /myurl/myurl/ at which point the rewriten url matches the rewrite rule again with $uri=/myurl/myurl/ and so on. By adding a specific `location /myurl/ {}` block you avoid that, because that's a more specific match that matches anything that already has the /myurl/ attached in front, and being more specific it gets precedence
Answer: Looks like this does the trick:
rewrite ^(/.*)$  ^/admin/$1 break;

Any other suggestion?

nginx url rewrite


What is the variance for time series data? How can it be computed?

Question: I have learnt time series courses but may have forgotten some important and basic pieces. For typical times series data say AR data, there is only one observation at a time. How do you define the variance and compute it? Please help! Thank you all.
Answer: It is true that you have one observation at each time, but this is the realization of the stochastic process $\{y_t: t \in 1,2,\dots\}$ behind the data you observe, where for stochastic process I mean a collection of univariate random variables defined on some probability space.
Typically, time series are defined by specifying a model for the individual components of the process, for example an AR(1): $y_t = \phi y_{t-1} + \epsilon_t$, where $\epsilon_t \sim WN(\sigma^2)$, i.e. $\epsilon_t$ is a white noise with variance $\sigma^2$.
So, for a AR(1), assuming that the process is weakly stationary, the variance of each random variable $y_t$ does not depend on time t, and you can compute it as
$Var(y_t) = \phi^2 Var(y_{t-1})+\sigma^2 \implies Var(y_t) = \frac{\sigma^2}{1-\phi^2}
$


Location Background Service that never closes

Question: I am creating an application for device tracking for my company (own company devices)
I created a Background Service using the GooglePlay API to get the updates.
However, when my application is closed, it rarely sends the locations to the server. Rarely does it get update updates.
What should I do? Is there any special permission I can use?
Answer: Use WorkManager for this purpose. Even if your app is being killed process scheduled in WorkManager will be executed.
In your case, you can schedule to check location at a fixed interval and send it to server or wherever you want to send.
From official doc:

The WorkManager API makes it easy to specify deferrable, asynchronous
  tasks and when they should run. These APIs let you create a task and
  hand it off to WorkManager to run immediately or at an appropriate
  time. For example, an app might need to download new resources from
  the network from time to time. Using these classes, you can set up a
  task, choose appropriate circumstances for it to run (like "only while
  device is charging and online"), and hand it off to WorkManager to run
  when the conditions are met. The task is still guaranteed to run, even
  if your app is force-quit or the device is rebooted.

Your task will come under Recurring tasks


Negative Harmony

Question: Is there a quick rule-of-thumb for finding negative harmonies?
Usually they are explained by drawing a circle of pitches and pairing them by lining up the 'positive' and 'negative' pitches. But this needs a diagram. How does one quickly decide the negative of, say, the supertonic chord without having to draw a diagram?
I suspect this may turn out to be a very naive question.
EDIT: I presume the downvote is for not explaining negative harmony. I felt it unecessary to do so (since only someone who knows it can answer and youtube is awash with vids), and impractical, (since I barely understand it.)
Comment: I don't think negative harmony is a common term, where does it come from?
Comment: Are you looking for a quicker method than the ones shown in the "Related" questions on the right side of the page? Unfortunately it's hard for me to imagine a quicker way.
Comment: @PatMuchmore Negative harmony is all the rage on YouTube at the moment.
Comment: I'm not sure if the question is naive or not, but it sure is difficult to answer. It's a bit as if you were asking "How does one quickly find out a relative minor key without drawing a circle of fifths/fourths?" I guess, the only quick way to go about it is to have an eidetic memory and knowing one's chords by heart. Other than that, you could program a tool that would do it for you. That's what I would personally do (as I do not have eidetic memory).
Comment: Beflummoxed by the terms 'negative' and 'positive' harmony.
Comment: @PatMuchmore I've found [this](https://music.stackexchange.com/questions/84709/are-there-any-typical-practical-applications-of-negative-harmony) related question that explains the concept quite nicely, IMHO. And I agree, the above question would benefit from a short explanation.
Comment: @Pyromonk - oh dear. Is it necessary now to have an app for everything? That makes me very un-app-y. Usage over a time should mean that a player is conversant with most of these sort of facts.
Comment: @Tim, I wasn't saying anything about an app. I am a developer professionally, so whenever I am lazy or cannot memorise something, I just program a tool that does it for me. I built a library for midi music automation, for example, to save me hours I would otherwise spend "composing" algorithmic music.
Comment: Thanks for the comments and answers. The nearest to an answer I've seen so far is to count the chord intervals down from the root note rather than up. This is a practical method and it might be the answer I'm after, unless there's a better one. I should have checked other Q&As first - I have now.
Comment: Just to confirm that I'm happy with the answers given. As you were.
Answer: There is a relatively simple algorithm, depending on what you think is relatively simple.  We'll just need to assign some numbers to the chromatic scale, with the major third in the key being 1 and the minor third being -1.  You can extend this as far as you need to, so:

Now just change the sign. I in this key (CEG) is -4 1 4.  Changing signs yields 4 -1 -4, or G Eb C, just as expected.  V7 (4 8 -2 2) becomes -4 -8 2 -2, or C G# F D: the inversion of iiø, as expected.  It doesn't matter if you use 8 for B or -5.
This might seem a little clunky at first, but it's not that much different from on the fly transposition, which is fairly simple to learn how to do.  You could also think about it in scale degrees, instead of trying to get absolute names, and then you would only have to work it once, as long as you're comfortable working with relative degree.

In this chart, you'll see the note transformation, but I've also put the new scale degree in, and you'll notice that the root becomes the 5th, the 2nd becomes the 4th, &c in inverse proportion to the scale degree, so in practice, you can count down from the 5th as your original note goes up.  In this case let's take V7, so scale degrees 5 7 2 4.  That means count down a 5th from G (if in C), a 7th from the G, 2nd from the G and 4th from G, giving us C, Ab, F, and D, the expected iiø.
This also means, if you are familiar with interval inversion, that you can calculate the negative counterpart from the inversion.  Perform the interval inversion (root = root, 2 = b7, 3 = b6, 4 = 5, etc) and then go down a fourth from that, and that will be the negative degree.

So to put this to practice, if your chord contains the root, the 3rd and the 5th, then the inversions are root, b6 and 4.  Go down a fourth from those and you have 5, b3 and root, or G, Eb & C, as expected.  Any one of these methods should work, will not require pre-charting, and should be able to be performed on the fly once you are used to it.
Answer: No, there isn't.  You just have to draw the diagram.  Well, you could, beforehand, draw a giant table of all possible diagrams: every axis, every triad, say.  But looking it up in the table might not be any faster.  Also, see Richard's answer.
Comment: Thanks. I feared as much. But counting the intervals down from the root seems to be a way to do it.
Comment: @PeterJ, that's basically it: create the negative by listing the ascending intervals above the bass/root as descending from the bass/root.

Negative Harmony


Out-of-core data analysis options

Question: I have been using SAS professionally for close to 5 years now.  I have it installed on my laptop and frequently have to analyze datasets with 1,000-2,000 variables and hundreds of thousands of observations.
I have been looking for alternatives to SAS that allow me to conduct analyses on similar sized data sets.  I am curious what other people use for situations such as this.  This certainly isn't "Big Data" in the manner that is used today. Nor are my datasets small enough to hold in memory.  I need a solution that can apply algorithms to data stored on a hard drive.  These are the things I have investigated to no avail:

R - BigMemory can create matrices stored out of memory, but the elements have to be the same mode.  I work with data that is almost a 50/50 split between character and numeric.  The FF package gets closer to what I need, but I don't quite understand which procedures are compatible with it.  I think support is somewhat limited.
Pandas -  I was very excited about a Pythonic alternative to R.  However, it too has to hold all of the data in memory.
Revolution R - This one shows quite a bit of promise.  I have a copy on my home computer (free if you sign up for Kaggle) and have yet to test it as viable alternative to SAS.  Comments on Revolution R as a SAS alternative are much appreciated.

Thanks
UPDATE 1
Editing to add that I am looking for real-life, practical solutions that people have used successfully.  For the most part, SAS lets me chug through big files without worrying one bit about memory constraints.  However SAS is implemented, they figured out how to make memory management transparent to the user.  But, it is with a heavy heart that I used SAS for my job (I have to) and would LOVE a FOSS alternative that allows me to work on "large" data without having to think too hard about where the data is located at a specific time (in memory or on disk).
The closest things I have come across are R's FF package and something on the horizon for Python called Blaze.  And yet, these problems have existed for many years so what have analysts been doing in the mean time?  How are they handling these same issues with Memory limits?  The majority of solutions on offer seem to be:

Get more RAM -- This isn't a good solution, imo.  It's easy to find a dataset that can exceed RAM yet still fit on a hard-drive.  Furthermore, the work flow has to accommodate all of the structures that are created during exploratory data analysis.
Subset the data -- This is fine for exploration but not for finalizing results and reporting.  Eventually, whatever processes are developed on a subset will have to be applied to the entire dataset (in my case, anyway).
Chunk through the data -- This is what I would like to know more about from people who actually implement this work-flow.  How is it done?  With what tools?  Can it be done in a way that's transparent to the user? (i.e., create some on-disk data structure and the frame-work takes care of the chunking under the hood).
Comment: Another option is to store the data in a database (SQL or otherwise). Often, an analysis will only require a subset of data (variables 1:10, but not 1:1000) which will fit in memory. If the subset is still larger than memory, the analysis can be done in chunks (load in the data from the database 1000 observations at a time, appropriately collate the results together at the end).
Comment: @Zelazny7: Adding another 4Gigs to a laptop should be pretty cheap these days. :)
Comment: The newer versions of Stata on 64 bit machines have no problem with datasets this size (because 5GB easily fits in RAM these days) but have you any interest in commercial alternatives to SAS or are you looking only at FOSS?
Comment: Yes, FOSS solutions are what I'm looking for.  I agree that a dataset could fit into 5 gigs of RAM, but it also has to handle of the operations and additional data structures that are created during exploratory data analysis.  Couple this with commercial laptops equipped with a meager 4GB of RAM and the overhead consumed by the OS and Memory Errors show up pretty quickly.
Comment: Be sure to review the [big data](http://stackoverflow.com/questions/tagged/bigdata?sort=votes&pagesize=15) questions on SO, as well as the [high-performance computing](http://cran.r-project.org/web/views/HighPerformanceComputing.html) task view on CRAN for further related discussions, if you have not already.
Comment: I've found [PyTables](http://www.pytables.org/moin) to be a reasonable solution in the past when working with data which wouldn't fit in memory.
Comment: Thanks, and I've looked at PyTables too.  I'd be very interested in seeing an example workflow using PyTables.  Something like: Read in data in chunks, store in PyTables, operate on PyTables columns to make new columns, etc...
Comment: @jthetzel I am exploring the database option.  Python has native support for sqlite3 and pandas has functions that can convert queries into dataframes.  I think this the approach I will end up adopting.  I'd like to award you the bounty.  Would you mind turning your comment into an answer?  Thanks
Answer: if you're maxing out at 500,000 records x 2,000 variables, i would spend a little more money on RAM for your laptop and be done with it.  if you have 16GB, you can probably read the data set you're describing into R directly.  and at that point, you'll be able to do far more - and very quickly.. but you say that's not an option, so:
look at SQL-based packages for R.  these allow you to connect to external databases and access those tables via SQL.  since SQL is pretty universal (and since R is open-source), your code won't be lost if you change jobs or lose access to SAS.  the easiest external database to set up is RSQLite but by far the fastest is MonetDB.R (speed tests)
there are probably a few good solutions to your stated problem, my guess is that just about all of them involve R  ;)
Answer: Maybe it is not so much about the applications/problems you are aiming for, and its characteristics, but more about the algorithms and variants you are using. More concretely, in order to handle big data, many variants based on stochastic gradient descent of popular algorithms, like SVM, have appear which are able to handle that.
Scikit offers support for some of this algorithms (SVM, kNN, kmeans, ...). I guess this nice graph can help you to quickly figure out, if scikit makes sense to you at all.
Hope that helps
NOTE: what follows is a reply on the comment by zelazny7
Now I understood you. What you are looking for is pandas. Take a look at the talks section. There is a presentation comparing Panda's workflow and panda's briefly. Panda lets you import data in different formats and handle bgu files through HDF5 tables integration. Besides, you can interface Scikit.
Comment: Thanks!  The algorithms would definitely have to work `on-line` or on chunks of the data read into memory and written back to disk.  Scikit is great and that's actually what I'd LOVE to use, but what tools/work-flows/approaches exist for the very necessary steps of exploration, munging and data prep before applying these algos?  These languages can handle these steps, but I'm really looking for an actual example from someone who has to deal with these issues out of memory.
Comment: I edited my reply with what you are looking for (or so I believe!)
Comment: What is the easiest way to calculate the maximum data size (in python and pandas) that can be handled smoothly by the current processor and the system as a whole before even starting to execute the dataset?
Answer: You already seem comfortable with SAS, and your datasets are small enough to fit in RAM, but maybe you can't fit enough RAM into your laptop. If you don't mind sticking with SAS, how about you just connect to SAS running remotely on a computer with lots of RAM? I have no idea how that works, but these links might get you started.

http://support.sas.com/documentation/cdl/en/camref/61896/HTML/default/viewer.htm#a000382050.htm
http://support.sas.com/documentation/cdl/en/hostunx/61879/HTML/default/viewer.htm#sasrem.htm

There are other great reasons to use Pandas or R, but I don't think you need to worry about memory limits. If you can't fit enough memory on your laptop, run the Python or R instance elsewhere and connect with SSH, iPython Notebook or RStudio.
Answer: Graphchi is excellent, and can handle huge datasets.  It's a bit of a pain to work with, but it can handle graphical and non-graphical data.
Answer: I recently came across SFrames and GraphLab Create.
These are libraries for Python that offer the kind of functionality you seem to be looking for
From the Pypi site: "SFrame is an scalable, out-of-core dataframe, which allows you to work with datasets that are larger than the amount of RAM on your system." So think of it as the data manipulation functionality and API in Pandas but without puttin all the data in memory first.
SFrame is free and open source as far as I know.
On the other hand GraphLab builds on the SFrame functionality to provide algorithms for doing descriptive and predictive (machine learning)  analytics on data that is stored in SFrames. GraphLab Create create is not free / open source, but has a free demo license.
In any case, depending on how sophisticated your algorithms need to be, SFrame might suffice for you. Or if you require for instance a quick and dirty linear / logistic / ridge regression on your data, you can implement your self it on the  SFrame with some simple SGD method, for example, and avoid having to pay for  Graphlab Create license.
Answer: Have you considered a "Real", non-interpreted language like Fortran?
It seems like the suggestions so far are either very vendor dependent or interpreted.  Interpreted methods are notoriously bad at memory intense applications.  MatLab may be much higher level of a language than "C" but the memory handling optimizations in C can make it handle 100's of times faster data sets that are millions of times larger.
Both "R" and "Python" are wonderful, high-level, technically rich and highly used languages.  They are also interpreted.
You might consider one of the R-on-Hadoop instances.  (Rhipe, others)  This has the advantage of being able to translate R (high level, easy to program) into MapReduce/Hadoop instructions.  Hadoop can make an interesting poor-mans multiprocessing cluster.
http://www.datadr.org/  <-- (Rhipe link)
Fortran has been being developed for decades. It has very efficient memory handling, and compiling. It also has some higher level libraries so it can do very technically sophisticated operations pretty simply. I might do a toy CFD in MatLab, but for something realistic and self-coded, I would use Fortran for the "big-iron" processing and something like MatLab or R for presenting/summarizing data.  Nobody makes commercial CFD software whose "engine" is interpreted instead of compiled.  Several vendors have their moneymaker CFD coded in C or Fortran.  SAS was originally written in C (link).
Fortran and Rhipe are accessible.  MatLab costs money and if my job didn't pay for it then I would be using R or Python right now.
UPDATE:
My point was "compiled".  Python has cython that can literally run the same (fundamental) code ~1000x faster.  That means you can have data that is ~1000x larger and process it in nearly the same time.  Getting Cython to work clean can be a challenge, but tools  like "Sage" wrap it well.  Rcpp allegedly has similar, but I don't personally know that it is as well developed.
If you think about it, the fundamentals of nearly everything you run in scientific computing, even in interpreted languages is compiled Fortran or compiled C.  BLAS is BLAS; you aren't re-inventing the code every time you do EDA.  Your interpreted language is calling those libraries, albeit very inefficiently, when it runs.
As an aside, you might look at JMP.  It has very easy to use interface and is excellent for very visual exploratory data analysis (EDA).
Comment: How do you explore data with a compiled language like fortran? The appeal of interpreted languages, for me, is the REPL which provides rich introspection and allows me to quickly learn what I'm dealing with as well as play with new library functions that I may not be familiar with.
Comment: Rcpp is very well developed (just check out the reverse usage on CRAN), but not really comparable with cython. If you have some defined function, yeah you can take the R input, process it in Cpp and return the data to R. But you won't be compiling a whole R script into Cpp anytime soon.

Out-of-core data analysis options


Multivariate regression and process control

Question: I have a question regarding process control with the use of multivariate regression.
The setup is as follows: say we have some data, representing the results of a plant process. Specifically, several continuous variables $\{x_1,\ldots,x_n\}$ represent inputs of the process (and we can control directly only $x_1, x_2, x_3$) and there is a continuous dependent variable $y$ being the result of this process. Our goal is to model the dependence and to control $y$ with the help of $x_1, x_2, x_3$. So, given real data, to model this dependence we can build the multiple regression with $y=\sum \alpha_i x_i$.
However, to control $y$ with $x_1, x_2, x_3$, it seems better to "reverse" the problem, i.e. to build multivariate regression: the independent variables (inputs) are now $\{x_4,\ldots,x_n,y\}$ and we predict the values of $x_1, x_2, x_3$ to be applied in order to obtain $y$.
And now for the main part: suppose there are some constraints on the inputs $x_1, x_2, x_3$, e.g. we may be forced to put $x_3$ or $\frac{x_1}{x_2}$ equal to some values. In other words, sometimes we might not be able to freely adjust the inputs in order to obtain given value of $y$, but still we want to be as close to $y$ as possible, so it seems like an optimization problem. The question: how such constraints could be introduced in the model?
EDIT:
Let me rephrase the problem by means of a particular constraint $x_1/x_2=u$. Given the model $(x_1, x_2, x_3) = f(x_4,\ldots,x_n, y)$, it has been obviously trained on samples with various values of $x_1/x_2$; for simplicity, suppose that a half of samples has $x_1/x_2\approx 1$ and the other half $x_1/x_2\approx 2$, and these ratios were set during production due to some process-specific constraints, such as availability of quantities $x_1, x_2$ through time.
Now, let's assume the tuple $(x_4,\ldots,x_n, y)$ is such that the model gives $f(x_4,\ldots,x_n, y)=(x_1, x_2, x_3)$ with $x_1/x_2\approx 1$, but due to practical reasons we insist on input $(x'_1, x'_2, x'_3)$ with $x'_1/x'_2=2$.
In that case, can we find $(x'_1, x'_2, x'_3)$ that is optimal, i.e. such that $(x'_1, x'_2, x'_3, x_4, \ldots, x_n)$ gives $y'$ that minimizes $(y-y')^2$? A wild guess would be to use a multiple regression $y=\sum \alpha_i x_i$ and then minimize an objective function $(y-y')^2$ w.r.t. $(x_1,x_2,x_3)$ with $x_1/x_2=2$, or perhaps there is another useful strategy?
Answer: Sounds like you might already have a model $(x_1,x_2,x_3)=f(x_4,\ldots,x_n,y;\hat\theta)$, so I'll just talk about the constraints.
If you have a constraint like $x_1/x_2 = u = \text{const}$, then you actually only need to model $(x_2,x_3) = f(\ldots)$, since you can recover $x_1$ with $x_1=ux_2$. Similarly if $x_3$ is fixed then this is even easier.
If you're constraints are more complicated than this (i.e. nasty and non-linear), you may be able to parameterize your problem, but failing that you could try nonlinear modelling with something like JuMP https://jump.dev/
Comment: If I got it right, you propose to train a new model each time we face a constraint. My idea is a little bit different: suppose the multivariate regression for $(x_1, x_2, x_3)$ has been trained on data with various ratios $x_1/x_2$. Then, based on that single model, I would like to find an optimal $(x_1, x_2, x_3)$ given some particular $x_1/x_2=u=\mathrm{const}$. Does this make sense?
Comment: I was thinking you would bake all of your constraints into a single model, not train a new one with each constraint. If you are having to add new constraints frequently or constraints are changing frequently then I suppose that may not be practical.
Comment: I edited the question, hope it narrows down the problem.