Android: Checking network connectivity return not connected when connected to wifi/3g in Broadcast receiver

Question: I want to check if device in connected or not in broadcastReceiver.
below is my code :
public boolean isOnline(Context context) {
NetworkInfo info = (NetworkInfo) ((ConnectivityManager) context
        .getSystemService(Context.CONNECTIVITY_SERVICE)).getActiveNetworkInfo();

if (info == null || !info.isConnected()) {
    Log.e("UpdateDataReceiver","info: "+info);
    return false;
}
return true;
}

Issue with my code:
above function returns me false (even when wifi connected) when BroadcastReceiver fires in background(when app is in background)
and it returns true when app is in foreground.

info: NetworkInfo: type: WIFI[], state: DISCONNECTED/BLOCKED, reason:
  (unspecified), extra: (none), roaming: false, failover: false,
  isAvailable: true, isConnectedToProvisioningNetwork: false, simId: 0

Device Info: Redmi Note
Comment: It seems BLOCKED could mean powered down to save battery. I tried a network request in this state and it DID go through but with a longer delay than normal.
Comment: Maybe you should extend WakefulBroadcastReceiver
Comment: I've got the same problem and found an issue on Google's issue tracker that seems to fit: https://issuetracker.google.com/issues/37137911
As a quick-fix I used a similar fallback-solution as mentioned in https://stackoverflow.com/a/45231746/1394330
Answer: This is how I'm handling it as it turns out getActiveNetworkInfo will always return you DISCONNECTED/BLOCKED in a specific case even if there is network connection. This is the receive method in the BroadcastReceiver with intent filter ConnectivityManager.CONNECTIVITY_ACTION.
@Override
public void onReceive(Context context, Intent intent) {
    ConnectivityManager conn = (ConnectivityManager) context.getSystemService(Context.CONNECTIVITY_SERVICE);
    NetworkInfo networkInfo = conn.getActiveNetworkInfo();

    NetworkInfo intentNetworkInfo = intent.getParcelableExtra(ConnectivityManager.EXTRA_NETWORK_INFO);

    if (intentNetworkInfo == null) {
        intentNetworkInfo = intent.getParcelableExtra(WifiManager.EXTRA_NETWORK_INFO);
    }

    if (networkInfo == null) {
        networkInfo = intentNetworkInfo;
    } else {
        //when low battery get ActiveNetwork might receive DISCONNECTED/BLOCKED but the intent data is actually CONNECTED/CONNECTED
        if (intentNetworkInfo != null && networkInfo.isConnectedOrConnecting() != intentNetworkInfo.isConnectedOrConnecting()) {
            networkInfo = intentNetworkInfo;
        }
    }

    //do something with networkInfo object
}

I've searched for better solution but no results. The case I've been able to reproduce 100% on my device (Pixel 7.1.2) is the following:

The device must be on low battery < 15% (other devices <20%)
Wifi is on, app is launched
Send app to background turnoff wifi and go to 3g (or vice versa)
Go back to the app
In that situation the app will report DISCONNECTED/BLOCKED from getActiveNetworkInfo.

If you change connectivity while in app it will be ok but if it is on background it wont. This won't happen while you are debugging because it will be charging the device even if the battery is low.
In the example above EXTRA_NETWORK_INFO in ConnectivityManager and WifiManager is actually same string "networkInfo" but I didn't wan't to risk it if in other Android versions they are different, so it is extra boilerplate.
You can directly use the networkInfo from the intent but I wanted to show here that there is a case where actualNetworkInfo is not the actual network info.
Comment: In the example above is for backward compatibility the string is the same for WifiManager "networkInfo".
Comment: Running into the same issue on Nexus 6p and Samsung S8 with power saver mode ON. Interestingly, Samsung S8 remains in power saver even while charging

NetworkInfo from ConnectivityManager.EXTRA_NETWORK_INFO correctly says CONNECTED, but it is deprecated since API 16. I am not sure about using that.
Comment: ConnectivityManager.EXTRA_NETWORK_INFO - It is deprecated as of API 16.
Answer: I believe the way you can do this is,
Register a Broadcast Receiver with an IntentFilter of ConnectivityManger.Connectivity_Action
private BroadcastReceiver receiverDataChange;

private void registerData(){

    try {
        receiverDataChange = new bcr_ToggleData();
        IntentFilter filterData = new IntentFilter();
        filterData.addAction(ConnectivityManager.CONNECTIVITY_ACTION);
        registerReceiver(receiverDataChange, filterData);

    } catch (IllegalArgumentException e) {
        e.printStackTrace();
    }}

Then in your Broadcast receiver class
public class bcr_ToggleData extends BroadcastReceiver {

@Override
public void onReceive(Context context, Intent intent) {

    String action = intent.getAction();

    if (action.equals(ConnectivityManager.CONNECTIVITY_ACTION)) {

        TelephonyManager telephonyManager = (TelephonyManager) context.getSystemService(Context.TELEPHONY_SERVICE);
        int state = telephonyManager.getDataState();
        switch (state){

            case TelephonyManager.DATA_DISCONNECTED: // off
                Log.d("DavidJ", "DISCONNECTED");
                break;

            case TelephonyManager.DATA_CONNECTED: // on
                Log.d("DavidJ", "CONNECTED");
                break;
        }
    }
}

}
This fires off when you go into your settings and turn on/off mobile data.
Hope this helps! :)
Answer:     ConnectivityManager cm =
        (ConnectivityManager)context.getSystemService(Context.CONNECTIVITY_SERVICE);

NetworkInfo activeNetwork = cm.getActiveNetworkInfo();
boolean isConnected = activeNetwork != null &&
                      activeNetwork.isConnectedOrConnecting();

I found this on this google tutorial: http://developer.android.com/intl/pt-br/training/monitoring-device-state/connectivity-monitoring.html. check it out.


Change syslog log file's owner/group?

Question: Is there a way to have syslog create a log file that's not owned by root?
I've got several jboss application servers that I've got logging to a server. Everyone's running CentOS. Right now, all of the logs are going to 'somelog.log' and it's owned by root:root. Can I make that instead be another user?
Answer: If possible for you, migrate to syslog-ng. It allows you to specify file modes per-file basis and is much more granular in every way.

Change syslog log file's owner/group?


actionscript variable name

Question: I am a freshman for actionscript. I get some code from others and get confused on some syntax.
<code>private static var _337833930conversationStartTimeList:ArrayList;
private static var conversationStartTimeList:ArrayList;
</code>
Whether these two represent same variables or different variables?
I think these should be the different ones but the get and set functions seem also work for the former one.
Does anybody help me to explain why is that and better to give me some references for further study.
Thanks
<code>    public static function get conversationStartTimeList() : ArrayList
    {
        return GeneralManager._337833930conversationStartTimeList;
    }// end function

    public static function set conversationStartTimeList(param1:ArrayList) : void
    {
        var _loc_3:IEventDispatcher = null;
        var _loc_2:* = GeneralManager._337833930conversationStartTimeList;
        if (_loc_2 !== param1)
        {
            GeneralManager._337833930conversationStartTimeList = param1;
            _loc_3 = GeneralManager.staticEventDispatcher;
            if (_loc_3 !== null)
            {
                _loc_3.dispatchEvent(PropertyChangeEvent.createUpdateEvent(GeneralManager, "conversationStartTimeList", _loc_2, param1));
            }
        }
        return;
    }// end function
</code>
Answer: These two have different names and they are declared in two separate statements - why would you think they are anything but two separate variables?
Comment: You should also try to not post decompiled code, btw.  If the author doesn't offer any source code, it's probably not intended to be viewed by others.
Comment: Actually in the code, it only has the first statement. But I know that in actionscript, get and set function could access property. But the get and set function's name should be same as the property's name. But in the code, property name is _337833930conversationStartTimeList but function name is conversationStartTimeList. But when I reuse them to test, it works. So I get confused about it. I think the property's name should be also conversationStartTimeList. That is whay I ask whether those two statements represent the same parameter.
Comment: Getter and setter names _should_ be similar, but don't have to be, and they certainly can't be the same as the variable name - there would be a compiler error if they were exactly equal!  So at least, it should be _conversationStartTimeList (starting with an underscore).

actionscript variable name


How to have "dashes" in filenames in ZendFramework 1.12

Question: I have seen a few posts regarding this question but none were answered to the topic on which they were asked. I have a ZendFramework project (v 1.12) and I want to do some changes in the file naming conventions.
The files and folders are named in Pascal case and I want to make the following changes:

rename all the files/folders to lowercase (this works)
I want dashes ("-") in between words. The Zend maual says that dashes could be there in the file names. But then again says that they should be mapped to the class names, (A Class Name cannot contain a -!!). And now there is Zend Router, it translates the - in URL and joins the words in between them back to the Pascal case. How do I manipulate that?

For example, Here is a file name: TestingDemoController.php, I want to change the name as testing-democontroller.php (actually I was thinking of making it as testing-demo.controller.php but don't know how possible is it.) For the previous filename, the URI used to be something like Testing-Demo. How can the router be configured to not treat - as a special character?

Edit: Questions on PSR:

On the suggestion of @apokryfos, I have been reading through PSR releases, PSR-0 and PSR-4 mostly.

looking at my class names (directories separated with underscore until the Terminating class name) I feel that we have used PSR-0 in our project.
On my approach to the change in project structure (having dashes in between file names). This goes against PSR-4,

The terminating class name corresponds to a file name ending in .php. The file name MUST match the case of the terminating class name.

Is the above point, the only point which I am going against?

This is what I feel about the ramifications that can happen on me doing the file name (dashed) and writing my own autoloader:
When looking for the files in libraries, the files will be looked by my autoloader, on failure the search will fallback to the Standard Autoloader which will finally resolve them? This is leading to unnecessary lookups.

Please correct me here if I am understanding it wrong and if am missing something?
Comment: For what it's worth, what you're doing goes against [PSR-4](http://www.php-fig.org/psr/psr-4/) which is a PHP coding standard you should be following.
Comment: The only ramification of going against the PSR-0/4 auto-loading standards is that you have to implement the autoloading yourself instead of using one of the many existing PSR-0 or PSR-4 compliant autoloaders. You will also be unable to convert your project into a more modular structure without bundling your own auto-loader as a dependency for each module.
Comment: Okay thanks, trying to understand it..
Comment: @apokryfos: I have updated the question, can please give some of your inputs regarding my understanding of PSR there?
Comment: Got it, while I will have a look at it myself, but can you say if I am correct on saying, that while loading library files, it will fallback to the standard autoloader causing unnecessary lookups?
Comment: Well, if you have 2 autoloaders then that will happen, in practice the overhead of autoloading is minimal so not something that should be a big concern.
Comment: Okay, thank you :)
Answer: I have done this for a project once.
There are 2 things that need to be done for this to work:
Override Zend Dispatcher
In order to override the standard Zend Dispatcher, create a file called Custom.php under "Zend/Controller/Dispatcher" with contents:
<?php

class Zend_Controller_Dispatcher_Custom extends Zend_Controller_Dispatcher_Standard {

    const DELIMITER = '-';
    public function classToFilename($class)
    {
        $class = trim(strtolower(preg_replace('/([A-Z]+)/', self::DELIMITER . "$1", $class)), self::DELIMITER); //str_replace('Controller', '-controller', $class);
        return str_replace('_', DIRECTORY_SEPARATOR, $class) . '.php';
    }

}

and in your Bootstrap.php of your application path, add this function:
protected function _initControllerDirectoryPath()
    {
            $frontController = Zend_Controller_Front::getInstance();
            $frontController->setDispatcher(new Zend_Controller_Dispatcher_Custom());
    }

This way you will get controllers with a - to be found, eg demo-controller.php. In this case, you will have to change all controllers to the new formats.
Override module resource loader
This is tricky, as you need to define your own autoloader.
Create a file "MyAutoloader.php" under "Zend/Loader" and add the following code:
<?php
class Zend_Loader_MyAutoloader extends Zend_Application_Module_Autoloader {

    public function autoload($class)
    {
        $classParts = explode('_', $class);
        $classParts[count($classParts) - 1] = trim(strtolower(preg_replace('/([^_])([A-Z]+)/', "$1-$2", end($classParts))), '-');
        $class = implode('_', $classParts);
        $classPath = $this->getClassPath($class);
        if (false !== $classPath) {
            return include $classPath;
        }
        return false;
    }
}

and in each module Bootstrap.php, you will need to add:
class Mymodule_Bootstrap extends Zend_Application_Module_Bootstrap
{
    public function __construct($application)
    {
        parent::__construct($application);

        $r    = new ReflectionClass($this);
        $path = $r->getFileName();
        $this->setResourceLoader(new Zend_Loader_MyAutoloader(array(
            'namespace' => $this->getModuleName(),
            'basePath'  => dirname($path),
        )));
    }
}

This will enable to use also module resources eg. "modules/mymodulename/forms/search-test.php"
If you use shared resources, eg forms, models etc, then you also need to register Zend_Loader_MyAutoloader for these in your main Bootstrap by adding:
protected function _initResourcesDirectoryPath()
{
    $loader = new Zend_Loader_MyAutoloader(array(
                'basePath'  => dirname(__FILE__),
                'namespace' => '',
    ));
}

Now you can have a file eg forms/test-shared-search.php
Note that class names remain the same inside the files(eg Form_TestSharedSearch in the last case)
Comment: Great, I was myself looking through all the docs and sources, thanks for the detailed answer. I don't have my code with me now but will certainly let you by tomorrow about how it goes.
Comment: The only thing I am hesitating is to create files in Zend Library itself. Can't I override the functionalities in my application itself? Anyway, I will have a look tomorrow.
Comment: Actually you do not have to create the files in the zend library itself. You can name the classes eg Mylibrary_MyAutoloader and Mylibrary_Dispatcher_Custom, place them under Mylibrary directory and register this new library using registerNamespace() of the autoloader.
Comment: Many thanks, although I am having a few problems, but now I know how to proceed. Great solution :)
Comment: With shared resources, did you mean models, forms etc which are not under a specific module? And I guess you missed `$this->setResourceLoader($loader)` for the Shared Resources. And also do you know any good links for understanding Routing, usage of both the Rest Controllers and normal Zend Action Controllers? (other than the Zend Manual)
Comment: Yes, that is what I mean. And yes, maybe I missed a couple of things. Xm, I am not sure about resources, cannot think of something
Answer: You can do that, but then you either need to use spl_autoload_register to create your own autoloader or include/require manually. Also, even though that is going to help with your file names, class names won't change, and you will definitely to have a custom routing.
That said, apokryfos is definitely right about PSR-4.


Very simple javascript example for getElementById. Why am I getting an error?

Question:

<!DOCTYPE html>

Hello World!

When I access the document, the console error I am getting is:
Uncaught TypeError: Cannot read property 'innerHTML' of null
What am I missing?
Comment: Scripts are evaluated while the document is being parsed. When your script runs, your `` element is not yet part of the DOM.
Comment: Ahh, thank you.  I thought it would look through the entire document.
Comment: It does look through the entire document _that has been parsed so far._ If that script ran after the document ready event, its location in the HTML would not matter.
Comment: May I ask why this one works though?

If I put that before my DIV elements, it does return an index.
Comment: @xproofx - The Element.getElementsByTagName() method returns a live HTMLCollection of elements with the given tag name. The subtree underneath the specified element is searched, excluding the element itself. The returned list is live, meaning that it updates itself with the DOM tree automatically. See https://developer.mozilla.org/en-US/docs/Web/API/Element.getElementsByTagName
Answer: It's because you have the script tag before the element. Try it the other way around:

<!DOCTYPE html>

Hello World!

It's usually advised to put the script tag at the very bottom of your page. There's also a couple of events such as load and DOMContentLoaded that you can use to start your script once things have loaded
Comment: @xproofx `.getElementsByTagName()` **always** returns a NodeList, but in that case it would be an empty one.
Comment: May I ask why this one works though?

If I put that before my DIV elements, it does return an index.


SSM Patch manager Patch policy issue

Question: The requirement is to do the patching of existing EC2 instances. So we decided to work with AWS Patch Manager under the Systems Manager.
Issue: We have added AmazonSSMManagedInstanceCore Policy Role to all instances. Whenever we try to do on-demand patching ( Patch Now ) it is working without error.
But when we need automatically scheduled patching which can be done by "Patch Policy" it gives an error into the installation part of the state manager. Currently, we are having EC2 instances of Amazon Linux2 OS and Ubuntu 16.04 OS. When we tried creating a new instance in a personal account with the same OS Amazon Linux 2, It is giving the same error.
Even tried with SSMFullaccess policy.
Sometimes we find this error.
No IMDS credentials were found on the instance.failed to run commands: exit status 156
Expecting to get the patch policy done without any fail.
Answer: I was facing the same issue I guess, my Patch policy was set up using quick-start and so the manual scan&patch was running fine whereas the quick-start association was failing instead.
The catch is that I'm using a custom role on EC2 instances and solved it adding a missing tag on the ec2-role, as suggested on this page..
https://docs.aws.amazon.com/systems-manager/latest/userguide/quick-setup-patch-manager.html#patch-policy-instance-profile-service-role
It wasn't obvious at all, hope it helps!
Comment: I added this during the patch policy creation time. Still had issue. But currently solved all of them. I will update soon.

SSM Patch manager Patch policy issue


Understanding Tierney Gibbs Sampler Kernel

Question: In subsection 2.2 Tierney (1994) states:

Suppose $X \sim \pi$ and $f$ is a function on the state space $E$. Then the kernel
$$
P(x, A) = P(X\in A \mid f(X) = f(x))
$$
leaves $\pi$ invariant.

However doesn't seem to provide any intuition or proof as to why that's the case. I know that for the kernel to leave $\pi$ invariant we must have
$$
\pi = \pi P \qquad \qquad \pi(A) = \int P(x, A)\, d\pi(x) = \int Q(f(x), A) \, d \pi(x)
$$
How do we show this and what's the intuition behind this?
Edit
Following comments here is my attempt. I will try to use the Law of Total Expectation. First, we can write
$$
\mathbb{E}_{\pi}[P(x, A)] = \mathbb{E}_\pi[P(X \in A \mid Y = y)]
$$
By the TOWER property we know that if $\nu$ is the distribution for $y$ then
$$
\mathbb{E}_\nu[\mathbb{E}_\pi[P(X \in A \mid Y=y)] = P(X\in A) = \pi(A)
$$
Now I only need to find what is the distribution $\nu$. Suppose $\pi$ is defined on $E$. Then $f(E)$ is image of the function $f$ and $y \in f(E)$. This means that our distribution $\nu$ must be defined on a sigma algebra of $f(E)$, for instance $\sigma(f(E))$
$$
\nu: \sigma(f(E))\to [0, 1]
$$
Since $E = f^{-1}(f(E))$ we can say $\nu$ is the pushforward of $\pi$ by $f^{-1} \circ f$
$$
\nu = (f^{-1} \circ f)_* \pi = \pi
$$
Therefore
$$
\int d\pi(x) \int P(X \in A\mid F(X) = f(x)) d\pi(x) = \pi(A)
$$
???
Comment: You indeed need to prove the last equality holds.
Comment: The intuition is that the move is according to $\pi$ restricted to one level set of $f$. Integrating over $x$ recovers all level sets,
Comment: Note that the kernel is not of practical interest for MCMC, since the entire sequence of simulations remains at the same level set $f(x)=f(x_0)$.
Comment: The expression$$\mathbb E_π[P(x,A)]=\mathbb E_π[P(X∈A∣Y=y)]$$should write$$\mathbb E_π[P(X,A)]=\mathbb E_{fπ}[P(X∈A∣Y)]$$as there is only one random variable left to integrate.
Comment: @Xi'an Thanks! I gave it a go but I didn't quite manage to prove it, do you mind having a look?
Answer: It looks like you have a good idea of the kind of tools you could use to prove this statement, but I was a bit unclear about your attempted proof, in particular what you meant by the measure/distribution $\nu$.
Intuitively, you compute $P(X\in A) = \pi(A)$ in two steps: first integrate across each level set of the function $f$, then integrate over all level sets of $f$; this is equivalent to integrating over the whole space $E$. You can see this most clearly in the case where $f$ takes on only $1$ or $2$ values. You could try working through these cases in detail to develop more intuition.
For the general case, it helps to identify $P(X \in A \mid f(X)=f(x))$ as an integral over a level set of $f$.  In fact, $P(X \in A \mid f(X)=f(x))=\mathbb{E}[1_{X\in A}\mid Y=f(x)]=\varphi(f(x))$ where the function $\varphi$ is such that $\varphi (Y)=\mathbb{E}[1_{X\in A}\mid Y]$. (See e.g. Definition 8.24 in Probability Theory by Klenke for how this function is obtained. It's also discussed in this answer.)
By Theorem 4.10 (image measures) in the same book, or Lemma 6.12 in Probability with Martingales by Williams, we have that $ \int P(x, A)\, d\pi(x) =\int\varphi(f(x))\, d\pi(x)=\mathbb{E}[\varphi (f(X))]$. This is just $\mathbb{E}[\varphi(Y)]=\mathbb{E}[\mathbb{E}[1_{X\in A}\mid Y]]$, which is equal to $\pi(A)$ by the law of total expectation.
Comment: @Yves I've included additional references for anyone who doesn't have access to Klenke.
Comment: I do not understand why someone downvoted this answer, which I find fine although I do not have access to the cited book. Maybe an explanation will come?


Do Boosted tree models result in only one final tree?

Question: I'm currently reading Intro to Statistical learning. On the chapter about Tree models I'm not sure if my intuition about the formula for boosting is correct

From the looks of it, it seems like shallow trees are built on top of one another with a shrinkage parameter. Would the end result of this summation be one large tree compromised of the sequentially built trees? Or am I completely off on this?
Answer: As the other answer mentions, the direct result is a collection of trees.
Depending on the operation used to combine trees, however, these can be combined into a single tree spanning the whole ensemble.
It is very easy to see how.
See this question for an example: Is the sum of two decision trees equivalent to a single decision tree?
Singe GBDTs are (almost always) simply stacked (i.e., added) sequentially, this also applies to them.
This means that the ensemble, in the end, is mathematically equivalent to a single decision tree.
It does not mean it is necessarily possible to learn the same decision tree by GBDTs (since the split criteria are different) though, but the final structure is itself a single polytree graph.
Answer: You're right that boosting is a summation of many "shrunken" trees. I guess for the intuition, though, boosting doesn't result in one final "tree", per se, at least in the sense of a single decision tree that you're able to go down splits and eventually end up at a node with a predicted value.
With boosting, you take a set of predictors and you input them into all $B$ of your trees, and each tree outputs a contribution, and all these $B$ contributions are then summed into a final boosted model output.
If I were to visualize a boosted model, I wouldn't be able to draw just a single tree. I'd have to draw something more like a series of $B$ decision trees in a row, write the contribution of each decision tree under it, and add $+$ signs in between each contribution to end up with the total / model output.
I'd say it's something more like a bunch of apple trees working together that each contribute a portion of a single apple that gets output.
Comment: You can merge trees into a single tree. It doesn't mean it's a more useful representation, nor is it central to GBDTs, but it is surely possible.


If a^(N-1) (is not≡) 1 mod N for some a relatively prime to N, then it must hold for at least half the choices of a < N.

Question:
Here, in the proof of the above lemma, I'm not able to understand why for every b less than N that passes the Fermat's test, there is a twin (a.b) that fails the Fermat's test.
Comment: It is proved in the picture you attached.
Comment: Suppose $a^{N-1}=c\neq 1$ (all equations are $\mod N$, of course) and that $b^{N-1}=1$. Then $(ab)^{N-1}=a^{N-1}b^{N-1} = c\cdot 1 = c$. But $ab\neq a$ (why?).  So there are always at least as many numbers $d$ with $d^{N-1}\neq 1$ as those with $d^{N-1}=1$: as long as there's at least one of the former, then there's at least one of the former for each of the latter.
Answer: Let $C$ be the set of those $c\in \{1,..., N-1\}$ for which $\gcd (c,N)=1.$ Let B be the set of those $b\in C$ for which $b^{N-1}\equiv 1\pmod N.$ Let $A =C$ \ $B.$ Assume $a\in A.$
For $b\in B$ let $f(b)\in \{0,1,...,N-1\}$ such that $ab\equiv f(b) \pmod N.$
Then $\gcd (f(b),N)=\gcd (ab,N)=1$ because $\gcd (a,N)=\gcd(b,N)=1.$ And $f(b)\ne 0$ because $$f(b)=0\implies N|ab \implies N|b \text {... (because } a\ne 0 \text { and } \gcd (a,N)=1)....$$ $$\implies b=0 \text {... (because } |b|<N) ...$$ $$\implies b\not \in B \text {... (by def'n of } B). $$ $\bullet \;$ So $f(b)\in C.$ And  $f(b)^{N-1}\equiv a^{N-1}\not \equiv 1 \pmod N$ so $f(b)\in A.$
For $b,b'\in B$ we have $$f(b)=f(b')\iff N|a(b-b') \iff N|(b-b') \text {...  (because } a\ne 0 \text { and } \gcd (a,N)=1)...  $$ $$\iff b-b'=0 \text {... (because } |b-b'|<N).$$
$\bullet \;$ So $f$ is a $1$-to-$1$ function from $B$ into $A.$ So $A$ has at least as many members as $B$ does.
Since $A\cap B=\emptyset$ and $A\cup B=C,$ and $A$ has at least as many members as $B$ does, therefore at least half of the members of $C$ belong to $A.$
Comment: If $N$ is composite and $A=\emptyset$ then $N$ is called a Carmichael number. See Wikipedia for more about them if you're interested.


Customize error in rails 3

Question: I have messages error from
render :json=> {"status"=>"failed",:errors=>@user.errors.messages}.to_json
{
    "status": "failed",
    "errors": {
        "Email": [
            "has already been taken"
        ],
        ,
        "Username": [
            "has already been taken"
        ]
    }
}

But i want to custom error:
{
        "status": "failed",
        "errors": {
            "error": [
                "Email has already been taken"
            ],
            ,
            "error": [
                "Username has already been taken"
            ]
        }
    }

Please help me to resolve this problem. Thank for your support.
Comment: Try accessing the Email and UserName value from `@user.errors.messages` and you cant index two values with the name "error"
Comment: why do you need this json structure? If you just want to change the message content, you can try http://stackoverflow.com/questions/6966263/rails-custom-validation-message
Answer: You be able to implement method my_custom_errors_messages in the User model. Or in the module and include it in your models. Where you will change errors has whatever you want.
You also could take a look at full_messages() it might help you.
Comment: please give me detail your solution.Thank so much.
Comment: What kind of details? Define method, use enumerable method like `map` and prepare Hash you want.
Comment: Read about modules: http://ruby-doc.org/core-2.0.0/Module.html Read about enumerable:  http://ruby-doc.org/core-2.0.0/Enumerable.html

Customize error in rails 3


multivariate mean?

Question: [edit : because my question was ambiguous, I decided to rewrite it entirely, with some simplification but a lot more details on the experimental design]
Four independent 10m*10m plots each received sewage sludge from the same water treatment facility (this is precisely where pseudoplication occurs).
After a year of waiting, one "representative" 1kg soil sample was sampled from each plot.
Ten 1g-sub-samples were sampled from each soil sample, then suspended in 10ml water and agitated for 2 days in order to "equilibrate".
Only then, 1 (arbitrary) unit of labeled molecule was added into each soil suspension.
We want to follow the decrease of labeled molecule quantity as time goes on.
After 1 min of agitation, remaining labeled molecule in water of one soil suspension was measured destructively, giving a value "v" between 0 and 1. The value of w = 1-v was recorded.
Same after 2, 3, 4, 5, 6, 7, 8, 9 and 10 minutes of agitation.
So far, we have recorded 10 w values for each soil sample.
plot of w with time of agitation :

Because v decreases with time, w is increasing with time and can be efficiently modeled by the following model equation :
      w ~ a * time^b

This model was fit onto each of the 4 sets of 10 w values, giving four sets of (a,b) parameters.
My problem is : How to calculate / estimate a set of global (a,b) parameters ?
I think calculating average of each parameter (=> a_mean and b_mean) is not right. Fitting a model on all pooled 40 w values is no more right.
NB : As a side note, if I had one, and only one, parameter of interest (for example, a maximum value, or a mean) by soil sample, one good option would be to average those four values into one global average.
Thanks !
Comment: Do you have any idea (optimally: a distribution) *how* the w-vectors from different plots are different from each other? Can you provide the data or a plot?
Comment: @A.Donda plot added. w-vectors are not really different in practice.
Answer: Your model assumption is
$$
w = a ~ t^b
$$
with some small random variations.
Take the logarithm of that equation:
$$
\log w = \log a + b ~ \log t.
$$
You now have a linear relationship between $\log w$ and $\log t$ with two parameters, $\log a$ and $b$. Determining the parameters is a standard linear regression problem, and if for simplicity we assume normally distributed errors with constant variance, the parameters can be estimated by ordinary least squares.

I don't know which software you use, but in Matlab you would do the following: Define the time points:
t = (1 : 10)';

Let's assume that your $w$-values are contained in a matrix W with 10 rows for the different time points and 4 columns for the different samples. Set up a design matrix where the first column consists of repetitions of $\log t$ and the second is constant 1
X = [repmat(log(t), size(W, 2), 1), ones(numel(W), 1)];

and construct a vector of logarithmized data
y = log(W(:));

Now you can perform the linear regression using Matlab's backslash operator:
parest = X \ y;

The estimates of the parameters are
b = parest(1);
a = exp(parest(2));

Comment: @Rodolphe, well it is "right" if you are willing to make the assumption that you have normally distributed errors in the logarithmized relation. If you think it is not right, you should explain why so that an answerer can deal with the underlying problem; just arbitrarily excluding a legitimate solution doesn't make sense. Btw., the standard free alternative to Matlab is Octave, and all my code should run with it unchanged.
Comment: In your own answer you assume that the true parameters are different and normally distributed; also a legitimate approach. But in that case you could just as well estimate the parameters on each data set and then average, which you also declared "not right". You're not playing fair here! :)
Comment: my reason for thinking it is not right to pool all datas together is because I think my problem is a two I dimensional  equivalent to the zero-dimensional case of measuring, in four non independent fish tanks, the length of ten fish in each = pseudoreplication.
Comment: This is true that I come back on my words in my answer. However I tried to explain why. If someone proves my first thoughts were wrong I will be happy to read it.
Comment: From what I understand, you seem to explain how to estimate as many (a,b) as columns in the W matrix. However my question is about how to get a global (a,b)  while taking into account that soil samples are pseudoreplicates.
Comment: @Rodolphe, no, read again. You get exactly one estimate of (a,b). Try it out for yourself.
Comment: I just achieved to try it by myself with FreeMat (I don't have matlab actually, and I am not familiar at all with the syntax so it took me some time... I use R). Now I understand that your way of doing this estimates one set of parameter by pooling all data together and not taking into account the relationship between datas from the same soil. I am sorry but this is exactly what I said in my question to be wrong. Thanks for showing me a bit of matlab syntax however.
Answer: Thanks to some simple R simulations, I came up with a satisfying answer to my own question.
The question asks what is the right way of estimating the true / global  set of parameters of a given model, given the model has been fitted to N different datasets, yielding N  vectors of parameters.
The answer is : calculate vector average from these N vectors.
How I came up with this answer that contradicts what I though at first is quite well demonstrated by a simulation (= backing up my answer with personal experience) :
Let's imagine we observe two variables X (independent variable), and Y (dependent variable). Assuming Y is linearly dependent on X, we can construct the following linear model :
 Y = a * X + b

Assume we know by construction that a_TRUE = 2 and b_TRUE = 10.
In the case of a and b being orthogonal to each other, one can draw a sample of N "a" values, and another sample of N "b" values.
 a_sample <- rnorm(10,a_TRUE,1)
 b_sample <- rnorm(10,b_TRUE,2)

here is one possible plot of the true equation (black solid) and all 10 individual equations (dashed red) :

By construction, a_TRUE can be estimated by averaging all individual vectors.
Here (sorry, I forgot to set seed so it is not reproducible), b_estimate = 9.7 +- 0.7 (95% CI), and a = 1.5 +- 0.8
By analogy, in the nonlinear case, it seems OK to average over all 4 (a,b) vectors.
However, with only four values to average by parameter, it is likely that 95% confident intervals will be huge.
I don't know, thought, if this is still OK when the parameters are correlated.
Comments are welcome.

multivariate mean?


Como separar com condicoes em R?

Question: O meu objetivo e separar um endereco com certas condicoes
Eu resultado final seria assim:
id   Cod_postal Zona    Tipo_estrada    Nome_estrada    numero  bloco   apartamento extra
1000 8400       Lisboa  Avenida         Ameixoeira 5    8       NA      ap.A        NA

Mas como eu fiz nao separa bem:
id   Cod_postal Zona    Tipo_estrada    Nome_estrada    numero       bloco  apartamento extra
1000 8400       Lisboa  Avenida                         Ameixoeira   5      8           ap.

Ate agora o meu codigo e esse aqui:
  select(Id,Cod_postal,cod_x) %>%
  separate(cod_x, into = c("Cod_postal","Zona","Tipo_estrada","Nome_estrada","numero","bloco","aparamento","extra"), sep = " ") -> test

Acho que e suposto usar o mutate ou algo do genero.
Edit: Amostra
1000 8400 Lisboa Avenida Ameixoeira 5 8 ap. A

Ps:(Eu sei que facilmente era separar com um espaco mas eu nao posso alterar a base de dados)
Ps2:O que eu pretendo e saber como separar pelo espaco e como exigir uma condicao de um certo campo (Por exemplo numero so pode ter numero)
Comment: Por favor, esclareça seu problema ou forneça detalhes adicionais a fim de destacar exatamente o que você precisa. Da forma como está escrito atualmente é difícil dizer com exatidão o que você está perguntando.
Comment: Bom dia. Para ajudar, seria necessário que vc adicionasse uma amostra da base para saber como a mesma está organizada.
Comment: eu tenho um endereco e pretendo separa-lo como indiquei no 1 treco de codigo, mas da maneira que eu fiz nao esta separando bem
Comment: O endereço está em uma "string" única ? Segue algum padrão ? Qual a entrada do dado ?
Comment: Adicionei mais informacao no edit
Comment: Sem um separador fica difícil a cidade pode ter um nome Lisboa ou nome composto 'Vila Nova de Gaia' p ex , aqui no Brasil é comum lugares sem número em zonas rurais , não tem como separar na origem ?
Comment: nem por isso porque sao cerca de 200.000 registos e separar 1 por 1 iria demorar semanas
Comment: Olá @nmexlxc , Bem vindo ao [pt.so], para entender melhor o funcionamento do site vale apena dar uma olhada em nosso [Tour]. Agora especificamente sobre sua pergunta, como ela é sobre [tag:r], deixo um link que pode ajuda-lo a melhora-la e dessa forma ajudar a comunidade a entender melhor sua duvida - [Como criar um exemplo mínimo reproduzível em R](https://pt.meta.stackoverflow.com/a/825/3774)
Answer: Na falta de um exemplo com a estrutura das variáveis "bloco" e "extra" fica difícil contemplar essas variáveis. No entanto, segue uma sugestão para separar as demais variáveis.
A função separate() é, de fato, uma boa opção. O problema é que vc precisa (se entendi corretamente) que o nome da estrada venha junto com seu número; da mesma forma que o termo "ap." venha agregado ao que se segue. A questão é que não dá para usar o separador " " porque ele separaria estas duas circunstâncias acima mencionada.
Uma possível solução seria criar algum caracter que separasse os termos que vc deseja, sem interferir no espaço em branco entre o termo estrada e o número que vem em seguida, e do termo "ap." do que vem em seguida.
Para este exemplo vou utilizar o underscore para separar os termos de interesse. E para adicionar este underscore vou utilizar a função str_replace().
A função str_replace() tem os seguintes parâmetros:

string =  com a string a ser manipulada;
pattern =  o padrão que será objeto da substituição;
replacement =  com a estrutura que se pretende.

Assim, primeiro crio um exemplo:
library(tidyverse)

df <- tibble(endereco = c("1000 8400 Lisboa Avenida Ameixoeira 5 8 ap. A"))

# A tibble: 1 × 1
  endereco
  <chr>
1 1000 8400 Lisboa Avenida Ameixoeira 5 8 ap. A

E aplico a função str_replace():
df_2 <-
  df %>%
  mutate(endereco = str_replace(string = endereco,
                                  pattern = "(.*) (.*) (.*) (.*) (.*\\s.*) (.*) (.*\\s.*)",
                                  replacement = "\\1_\\2_\\3_\\4_\\5_\\6_\\7"))

A interpreção do parâmetro pattern =  é a seguinte:

cada () representa uma coluna. Como a intenção é dividir a string "endereco" em 7 colunas, criamos, 7 conjuntos de (). Ficaria assim:

pattern = "()()()()()()()"

do lado de fora do parêntese, fica aquilo que não será capturado. No caso, não queremos capturar os espaços. Então a continuação do preenchimento é:

pattern = "() () () () () () ()"

e agora, informamos qual será o conteúdo de cada coluna. Com base no exemplo que vc deu, a seguinte estrura se adequa bem:

pattern = "(.*) (.*) (.*) (.*) (.*\\s.*) (.*) (.*\\s.*)"

O .* significa qualquer coisa em qualquer quantidade.
Assim, uma leitura do padrão acima seria:

(.*)  qualquer coisa seguido de espaço em branco seguido de;
(.*)  qualquer coisa seguido de espaço em branco seguido de;
(.*)  qualquer coisa seguido de espaço em branco seguido de;
(.*)  qualquer coisa seguido de espaço em branco seguido de;
(.*\\s.*)  qualquer coisa seguido de espaço em branco seguido de qualquer coisa seguido de espaço em branco seguido de;
(.*)  qualquer coisa seguido de espaço em branco seguido de;
(.*\\s.*) qualquer coisa seguido de espaço em branco seguido de qualquer coisa.

resultado:
df_2

# A tibble: 1 × 1
  endereco
  <chr>
1 1000_8400_Lisboa_Avenida_Ameixoeira 5_8_ap. A

Agora utiliza-se o _ como separador na função separate():
df_2 %>%
  separate(col = endereco,
           into = c("id",
                    "Cod_postal",
                    "Zona",
                    "Tipo_estrada",
                    "Nome_estrada",
                    "numero",
                    "apartamento"),
           sep = "_",
           convert = TRUE)

Resultado:
# A tibble: 1 × 7
     id Cod_postal Zona   Tipo_estrada Nome_estrada numero apartamento
  <int>      <int> <chr>  <chr>        <chr>         <int> <chr>
1  1000       8400 Lisboa Avenida      Ameixoeira 5      8 ap. A

Como separar com condicoes em R?


Exception in thread "main" java.lang.RuntimeException: Found multiple defaults.yaml resources while executing Storm

Question: I am trying to execute the tutorial found here http://virajonlinetutor.blogspot.in/
Kafka storm Topology code is :
package com.storm;

import storm.kafka.KafkaSpout;
import storm.kafka.SpoutConfig;
import storm.kafka.StringScheme;
import storm.kafka.ZkHosts;
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.spout.SchemeAsMultiScheme;
import backtype.storm.topology.TopologyBuilder;

public class KafkaStormTopology {
public static void main(String[] args) {
  ZkHosts zk = new ZkHosts("10.25.3.208:2181");
  SpoutConfig config = new SpoutConfig(zk,"deepthy","",
"KafkaStorm");

 config.scheme = new SchemeAsMultiScheme(new StringScheme());

  config.forceFromStart = true;

  TopologyBuilder builder = new TopologyBuilder();
  builder.setSpout("KafkaSpout", new KafkaSpout(config), 1);
  builder.setBolt("Bolt", new FileBolt(), 1).globalGrouping("KafkaSpout");

  LocalCluster cluster = new LocalCluster();

  Config conf = new Config();
  conf.setDebug(true);
  cluster.submitTopology("SampleTopology", conf, builder.createTopology());
 }
}

My pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.deepthy.storm</groupId>
<artifactId>Storm</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>Storm</name>
<url>http://maven.apache.org</url>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<repositories>
    <repository>
        <id>clojars.org</id>
        <url>http://clojars.org/repo</url>
    </repository>
    <repository>
        <id>central</id>
        <url>http://repo1.maven.org/maven2/</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>3.8.1</version>
        <scope>test</scope>
    </dependency>
      <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-core</artifactId>
        <version>0.9.2-incubating</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>0.9.2-incubating</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.2.2</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.3</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.storm.KafkaStormTopology</mainClass>
</manifest>
 </archive>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
 </goals>
 </execution>
</executions>
</plugin>
</plugins>
</build>
</project>

I get the following error:
Exception in thread "main" java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar. [jar:file:/usr/hdp/18.104.22.168-2557/storm/lib/storm-core-0.10.0.2.3.0.0-2557.jar!/defaults.yaml, jar:file:/usr/hdp/126.96.36.199-2557/storm/S1.jar!/defaults.yaml]
    at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:140)
    at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:167)
    at backtype.storm.utils.Utils.readStormConfig(Utils.java:191)
    at backtype.storm.config$read_storm_config.invoke(config.clj:121)
    at backtype.storm.command.config_value$_main.invoke(config_value.clj:22)
    at clojure.lang.AFn.applyToHelper(AFn.java:161)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at backtype.storm.command.config_value.main(Unknown Source)

I tried changing the scope from 'compile' to 'provided' in pom.xml ,but getting the same error.
Any suggestions regarding how to solve this.
Comment: What command are you running to upload the jar?
Answer: Your jar contains the file defaults.yaml which in not allowed. You need to exclude the file from your jar assembly step. I would recommend to use maven-jar-plugin instead of maven-assembly-plugin and to include all needed files "manually" -- this also help to "minimize" your jar to only contain stuff you need:
<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>

        <executions>
                <execution>
                    <id>YOUR-ID</id>
                    <phase>package</phase>
                    <goals>
                        <goal>jar</goal>
                    </goals>
                    <configuration>
                        <finalName>JAR-FILE-NAME-PREFIX</finalName>
                        <classifier>JAR-FILE-NAME-SUFFIX</classifier>

                        <archive>
                            <manifestEntries>
                                <mainClass>com.storm.KafkaStormTopology</mainClass>
                            </manifestEntries>
                        </archive>

                        <includes>
                            <include>packages/to/include/**/*.java</include>

                        <includes>
                    </configuration>
                </execution>
        </executions>
</plugin>

Answer: exclude default.yaml file in pom file.
in your shade plugin add

                                    :

                                        META-INF/.SF
                                        META-INF/.DSA
                                        META-INF/.RSA
                                        .yaml


How to get common roots of unity of $ z^{6}=1 $ and $ z^{21}=1 $?

Question: How to get common roots of unity of $z^{21}=1$ and $z^{6}=1 $ for $ z\in\mathbb C $?
I know that $ z^{n} =1 $ has roots $ z=e^{\frac{2\pi k }{n}i} $ where $ k\in \{0,1,2,...,n-1\} $
Comment: Do you mean $z^6=1?$
Comment: Find the roots of $z^3$ as $\left(z^3\right)^2=z^6$ and $\left(z^3\right)^7=z^{21}$.
Comment: Note that $z^{21-3\times 6}=z^3=1$.
Comment: I have corrected!!
Comment: In $z^{21}=1$ for what values k we have $e^{i\pi/3}$?? And how
Answer: $$  \gcd( 21, 6 ) = ???    $$
$$ \frac{ 21 }{ 6 } = 3 +  \frac{ 3 }{ 6 } $$
$$ \frac{ 6 }{ 3 } = 2 +  \frac{ 0 }{ 3 } $$
 Simple continued fraction tableau:
$$
 \begin{array}{cccccc}
 & & 3 & & 2 & \\
  \frac{ 0 }{ 1 }   &   \frac{ 1 }{ 0 }   & &   \frac{ 3 }{ 1 }   & &   \frac{ 7 }{ 2 }
 \end{array}
 $$
$$  $$
$$ 7 \cdot 1 - 2 \cdot 3 = 1 $$
$$  \gcd( 21, 6 ) = 3  $$
$$ 21 \cdot 1 - 6 \cdot 3 = 3 $$
==================================================================
a different example:
$$  \gcd( 7654321, 1234567 ) = ???    $$
$$ \frac{ 7654321 }{ 1234567 } = 6 +  \frac{ 246919 }{ 1234567 } $$
$$ \frac{ 1234567 }{ 246919 } = 4 +  \frac{ 246891 }{ 246919 } $$
$$ \frac{ 246919 }{ 246891 } = 1 +  \frac{ 28 }{ 246891 } $$
$$ \frac{ 246891 }{ 28 } = 8817 +  \frac{ 15 }{ 28 } $$
$$ \frac{ 28 }{ 15 } = 1 +  \frac{ 13 }{ 15 } $$
$$ \frac{ 15 }{ 13 } = 1 +  \frac{ 2 }{ 13 } $$
$$ \frac{ 13 }{ 2 } = 6 +  \frac{ 1 }{ 2 } $$
$$ \frac{ 2 }{ 1 } = 2 +  \frac{ 0 }{ 1 } $$
 Simple continued fraction tableau:
$$
 \begin{array}{cccccccccccccccccc}
 & & 6 & & 4 & & 1 & & 8817 & & 1 & & 1 & & 6 & & 2 & \\
  \frac{ 0 }{ 1 }   &   \frac{ 1 }{ 0 }   & &   \frac{ 6 }{ 1 }   & &   \frac{ 25 }{ 4 }   & &   \frac{ 31 }{ 5 }   & &   \frac{ 273352 }{ 44089 }   & &   \frac{ 273383 }{ 44094 }   & &   \frac{ 546735 }{ 88183 }   & &   \frac{ 3553793 }{ 573192 }   & &   \frac{ 7654321 }{ 1234567 }
 \end{array}
 $$
$$  $$
$$ 7654321 \cdot 573192 - 1234567 \cdot 3553793 = 1 $$
Answer: $$ \begin{align}\{ z : z^6=z^{21}=1, z\in\mathbb C \}&\Longleftrightarrow \{ k : k=o(z),\ k|6,\ k|21 \}\\
&\Longleftrightarrow\{ k : k=o(z),\ k|\gcd (6, 21) \}\\
&\Longleftrightarrow\{ k : k=o(z),\ k|3 \}\\
&\Longleftrightarrow\{ k : k=o(z),\ k=1\ \text{or}\ k=3 \}\\
&\Longleftrightarrow\{ z\in \mathbb C : z^3=1 \}\\
&\Longleftrightarrow\{ z : z=1\ \text{or}\ z=e^{\frac{2\pi i}{3}}\ \text{or}\ z=e^{\frac{4\pi i}{3}}  \} \end{align}$$
Answer: $z$ will satisfy both of $z^{21}=1$ and $z^6=1$ iff $z^3=1$.  One direction is trivial,  the other follows from $z^6=1\implies z^{21}=z^3$.
Thus $ z\in \{ 1,e^{\frac{2\pi i}3},e^{\frac{4\pi i}3} \} $.


Golang io.Pipe vs node.js Pipe

Question: I'm new to golang and I'm having problem understanding go's io.Pipe. Is this similar to node.js' .pipe? And how should I use it? Is it possible to use it with 1 read file and a write file?
Thank in advance guys.
Answer: No, they are not precisely similar. io.Copy(dat io.Writer, src io.Reader) is quite enough to read and write files, like this:
input := bufio.NewReader(os.Stdin)
output := bufio.NewWriter(os.Stdout) // buffer output like C stdlib
io.Copy(output, input)               // copy entire file
output.Flush()

io.Pipe() (*PipeReader, *PipeWriter) will produce piped Reader and Writer for you when you have not them but code expect them, like this:
type id struct{
  name string
  age int
}

payload := id{"John", 25}
requestBody, jsonPayload := io.Pipe()
request := http.NewRequest("POST". "http://www.example.com", requestBody) // NewRequest expect io.Reader
encoder := json.NewEncoder(jsonPayload) // NewEncoder expect io.Writer
err := encoder.Encode(payload)
response, err := client.Do(request)

Golang io.Pipe vs node.js Pipe


Term frequency/inverse document frequency (TF/IDF): weighting

Question: I've got a dataset which represents 1000 documents and all the words that appear in it. So the  rows represent the documents and the columns represent the words. So for example, the value in cell $(i,j)$ stands for the times word $j$ occurs in document $i$. Now, I have to find 'weights' of the words, using tf/idf method, but I actually don't know how to do this. Can someone please help me out?
Comment: The tf-idf-Statistic For Keyword Extraction - http://www.joyofdata.de/blog/tf-idf-statistic-keyword-extraction/
Answer: there is package tm (text mining) http://cran.r-project.org/web/packages/tm/index.html which should do exactly you need:
#read 1000 txt articles from directory data/txt
corpus  <-Corpus(DirSource("data/txt"), readerControl = list(blank.lines.skip=TRUE));
#some preprocessing
corpus <- tm_map(corpus, removeWords, stopwords("english"))
corpus <- tm_map(corpus, stripWhitespace)
corpus <- tm_map(corpus, stemDocument, language="english")
#creating term matrix with TF-IDF weighting
terms <-DocumentTermMatrix(corpus,control = list(weighting = function(x) weightTfIdf(x, normalize = FALSE)))

#or compute cosine distance among documents
dissimilarity(tdm, method = "cosine")

R is a functional language so reading code can be tricky (e.g. x in terms)
Answer: Wikipedia has a good article on the topic, complete with formulas.  The values in your matrix are the term frequencies.  You just need to find the idf: (log((total documents)/(number of docs with the term)) and multiple the 2 values.
In R, you could do so as follows:
set.seed(42)
d <- data.frame(w=sample(LETTERS, 50, replace=TRUE))
d <- model.matrix(~0+w, data=d)

tf <- d
idf <- log(nrow(d)/colSums(d))
tfidf <- d

for(word in names(idf)){
  tfidf[,word] <- tf[,word] * idf[word]
}

Here's the datasets:
> colSums(d)
wA wC wD wF wG wH wJ wK wL wM wN wO wP wQ wR wS wT wV wX wY wZ
 3  1  3  1  1  1  1  2  4  2  2  1  1  3  2  2  2  4  5  5  4
> head(d)
  wA wC wD wF wG wH wJ wK wL wM wN wO wP wQ wR wS wT wV wX wY wZ
1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0
2  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0
3  0  0  0  0  0  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
4  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0  0
5  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0  0  0  0  0  0
6  0  0  0  0  0  0  0  0  0  0  1  0  0  0  0  0  0  0  0  0  0
> head(round(tfidf, 2))
  wA wC wD wF wG   wH wJ wK wL wM   wN wO wP   wQ wR wS wT   wV  wX  wY wZ
1  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 0.00  0  0  0 0.00 2.3 0.0  0
2  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 0.00  0  0  0 0.00 0.0 2.3  0
3  0  0  0  0  0 3.91  0  0  0  0 0.00  0  0 0.00  0  0  0 0.00 0.0 0.0  0
4  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 0.00  0  0  0 2.53 0.0 0.0  0
5  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 2.81  0  0  0 0.00 0.0 0.0  0
6  0  0  0  0  0 0.00  0  0  0  0 3.22  0  0 0.00  0  0  0 0.00 0.0 0.0  0

You can also look at the idf of each term:
> log(nrow(d)/colSums(d))
      wA       wC       wD       wF       wG       wH       wJ       wK       wL       wM       wN       wO       wP       wQ       wR       wS       wT       wV       wX       wY       wZ
2.813411 3.912023 2.813411 3.912023 3.912023 3.912023 3.912023 3.218876 2.525729 3.218876 3.218876 3.912023 3.912023 2.813411 3.218876 3.218876 3.218876 2.525729 2.302585 2.302585 2.525729

Comment: Thanks for your help! But is it possible to obtain some value for each word which represents some weighting (instead of a whole matrix)? Now we have a whole matrix of weights. I'm doing some feature selection and want to use tf/idf as a filter method...
Comment: @ABC tf-idf by definition refers to the full matrix of weights.  Perhaps you are interested in the idf weights alone, which you would get by `log((number of docs)/(number of docs containing the term))`.  You could also just filter out the infrequent terms.
Comment: Very clear! Really appreciated.
Answer: Your code has an error: colSums computes the number of occurence in the corpus, not the number of texts with the word.
A version computing such would be:
tfidf=function(mat){
  tf <- mat
  id=function(col){sum(!col==0)}
  idf <- log(nrow(mat)/apply(mat, 2, id))
  tfidf <- mat
  for(word in names(idf)){tfidf[,word] <- tf[,word] * idf[word]}
  return(tfidf)
  }

Answer: I am late to this party, but I was playing with the concepts of tc-idf (I want to emphasize the word 'concept' because I didn't follow any books for the actual calculations; so they may be somewhat off, and definitely more easily carried out with packages such as {tm: Text Mining Package}, as mentioned), and I think what I got may be related to this question, or, in any event, this may be a good place to post it.

SET-UP: I have a corpus of 5 long paragraphs taken from printed media, text 1 through 5 such as The New York Times. Allegedly, it is a very small "body", a tiny library, so to speak, but the entries in this "digital" library are not random: The first and fifth entries deal with football (or 'soccer' for 'social club' (?) around here), and more specifically about the greatest team today. So, for instance, text 1 begins as...

"Over the past nine years, Messi has led F.C. Barcelona to national
  and      international titles while breaking individual records in
  ways that seem otherworldly..."

Very nice! On the other hand you would definitely want to skip the contents in the three entries in between. Here's an example (text 2):

"In the span of a few hours across Texas, Mr. Rubio suggested that Mr.
  Trump had urinated in his trousers and used illegal immigrants to tap
  out his unceasing Twitter messages..."

So what to do to avoid at all cost "surfing" from the text 1 to text 2, while continuing to rejoice in the literature about almighty Barcelona F.C. in text 5?

TC-IDF: I isolated the words in every text into long vectors. Then counted the frequency of each word, creating five vectors (one for each text) in which only the words encountered in the corresponding text were counted - all the other words, belonging to other texts, were valued at zero. In the first snippet of text 1, for instance, its vector would have a count of 1 for the word "Messi", while "Trump" would have 0. This was the tc part.
The idf part was also calculated separately for each text, and resulted in 5 "vectors" (I think I treated them as data frames), containing the logarithmic transformations of the counts of documents (sadly, just from zero to five, given our small library) containing a given word as in:
$\log\left(\frac{\text{No. documents}}{1\, +\, \text{No. docs containing a word}}\right)$. The number of documents is 5. Here comes the part that may answer the OP: for each idf calculation, the text under consideration was excluded from the tally. But if a word appeared in all documents, its idf was still $0$ thanks to the $1$ in the denominator - e.g. the word "the" had importance 0, because it was present in all texts.
The entry-wise multiplication of $\text{tc} \times \text{idf}$ for every text was the importance of every word for each one of the library items - locally prevalent, globally rare words.

COMPARISONS: Now it was just a matter of performing dot products among these "vectors of word importance".
Predictably, the dot product of text 1 with text 5 was 13.42645, while text 1 v. text2 was only 2.511799.
The clunky R code (nothing to imitate) is here.
Again, this is a very rudimentary simulation, but I think it is very graphic.
Answer: There is a new R package which can do this: textir: Inverse Regression for Text Analysis
The relevant command is tfidf, the example from the manual:
data(we8there)
## 20 high-variance tf-idf terms
colnames(we8thereCounts)[
order(-sdev(tfidf(we8thereCounts)))[1:20]]


Confidence Interval for the average of a set of proportions

Question: If I have a dataset of 500+ observations (people in this case) that has the % of allotted dollars that were spent for each person (so, I have each persons allotted dollars and their dollars spent), and I want to show the average % of dollars spent (in this case the average is 80%), how would I also calculate some measure of "confidence" for that average?  Is it correct to just calculate a confidence interval using a Wilson Score?  I'm having trouble finding the right method, given that the average isn't really the proportion from a set of binomial data, but rather the average of a set of proportions...
Thanks!
Comment: Is it possible for a person to spend part of a dollar? It doesn't seem to me that your data are binomial.
Comment: Yes, right -- so would it be appropriate to just use the normal distribution to calculate a confidence interval for the mean?
Comment: For a guide to using the beta regression in R (which in the simplest case could estimate just an intercept & SE), you could see my answer here: [Remove effect of a factor on continuous proportion data using regression in R](http://stats.stackexchange.com/a/94766/7290).
Answer: The Wilson score interval is for binomial proportions, which you don't have here -- you essentially have a continuous proportion, which - while it will share the property that the variance will tend to be smaller if the proportion is near 0 or 1 - won't typically have a variance of the same form as the binomial.
You will expect to have non-constant variance across different proportions, so it's also unlikely that you can reasonably argue for some constant-variance assumption. [For variables constrained to lie in $[0,1]$ with mean $p$, the variance cannot exceed $p(1-p)$ which in turn $\leq \min(p,1-p)$, so as the mean approaches either 0 or 1, the variance must be small -- smaller than the gap between the mean and the nearest boundary -- whereas in the middle region it may be relatively large.]
Unless the data occur naturally in groups of similar proportion (so you could measure variance within groups you might treat as homogeneous), you'll likely have to make some sort of model assumption, such as a beta model, or you'll also need to model how the variance relates to the mean (it sounds like you might have enough data to do that).
If the variance of the proportions is approximately proportional* to $p(1-p)$, a beta model might be suitable.
* for a beta random variable, the constant of proportionality being $\frac{1}{\alpha+\beta+1}$
Once you have an estimated variance for each proportion, if you also assume either that the proportions are independent or are able to assume/identify some model for the correlation structure between them, you can compute the standard error for the average, at which point a normal approximation for the average proportion should work very nicely and a confidence interval would be easily constructed.
If you have potential covariates, you would likely be best off modelling both the mean and the variance, perhaps via beta regression.
--
An alternative (though related) possibility is to consider transformation to some scale where the variance is more nearly constant, and then using that as a way of getting an estimated variance for each proportion so that an appropriate (if approximate) variance of the average of the proportions could be calculated. I'm not sure this gains us anything over modelling the variance more directly, but some people may find it conceptually easier.
Note that I am not talking about computing an average on the transformed scale and trying to make an interval for the average on the original scale from that*, but simply getting an approximate variance model for the original proportions out of being able to say "with this transformation, the variance is seen to be nearly constant" and backing the proportion variances out of the estimate of constant transformed-scale variance and the transformation (via Taylor series approximation, say).
*(I guess you might try to do something like that, but it would need a correction for bias)
To that end, this reference may be helpful:
Rocke, David M. (1993),
"On the Beta Transformation Family,"
Technometrics, 35:1 (Feb), pp. 72-81
Comment: Thanks Glen_b.  I think I'm in over my head here, but quick question -- are you saying that it's necessary to calculate a variance for each observation?  How does that work, given that each proportion is just data point, not a distribution?
Comment: The point of my post from the paragraph beginning "*Unless the data occur...*" onward is to explain how you do that. There are four alternatives discussed there - (i) grouping similar proportions (such as based on some variable which the proportions relate to) and treating the groups as homogeneous; (ii) assuming some distribution, such as a beta distribution, where the variance is related to the parameters that also define the mean; (iii) modelling the variance (e.g. as a function of the proportion); (iv) back out variance via back-transformation from a variance-stabilizing transformation.
Comment: My first instinct (in the absence of seeing the data) would be to consider whether a common beta distribution would be appropriate, or whether some model that incorporates variation in parameters might be necessary (perhaps with predictors, possibly a mixed model or some mixture model). An obvious predictor to consider is dollars-allocated.


Is there an advantage to using blocks over functions in Objective-C?

Question: I know that a block is a reusable chunk of executable code in Objective-C. Is there a reason I shouldn't put that same chunk of code in a function and just called the function when I need that code to run?
Answer: It depends on what you're trying to accomplish. One of the cool things about blocks is that they capture local scope. You can achieve the same end result with a function, but you end up having to do something like pass around a context object full of relevant values. With a block, you can do this:
<code>int num1 = 42;
void (^myBlock)(void) = ^{
    NSLog(@"num1 is %d", num1);
};

num1 = 0; // Changed after block is created

// Sometime later, in a different scope

myBlock();              // num1 is 42
</code>
So simply by using the variable num1, its value at the time myBlock was defined is captured.
From Apple's documentation:

Blocks are a useful alternative to traditional callback functions for
  two main reasons:

They allow you to write code at the point of invocation that is
  executed later in the context of the method implementation. Blocks are
  thus often parameters of framework methods.
They allow access to local variables. Rather than using callbacks
  requiring a data structure that embodies all the contextual
  information you need to perform an operation, you simply access local
  variables directly.
Comment: Great explanation. Yet I have a question for reason 1. Take completion handler block for example. If no multi-threading involves,i.e. the method and its block are both running on main queue, there should be no difference between putting the completion code inside the block and outside the block. Because the completion code would run AFTER the method finishes in both cases. Is it correct?
Comment: Yes, that's true, but there are many cases where a method needs time to do something asynchronously, and does so without blocking the main thread (not necessarily just multithreading, see NSRunLoop). There are application where the same method may sometimes run synchronously and call the completion block before it returns, and other times run asynchronously. It's nice to allow for both possibilities without writing different (calling) code. Another thing is that completion blocks often have arguments, which essentially allows for the method to return multiple values.
Comment: Session 308 from WWDC 2011 ("Blocks and GCD in Practice") highlights the advantages of blocks over ordinary functions (as covered in this answer) with a nice example to do with sorting.
Answer: As Brad Larson comments in response to this answer:

Blocks will let you define actions that take place in response to an
  event, but rather than have you write a separate method or function,
  they allow you to write the handling code right where you set up the
  listener for that event. This can save a mess of code and make your
  application much more organized.

A good example of which i can give you is of alert view, it will be good if i decided at time of creation of alert view what will happen when i dismiss that instead i write the delegate method and wait for that to call. So it will be much easier to understand and implement and also it provides fast processing.
Comment: Please do not plagiarize things others have written elsewhere without proper citation. This will get you suspended from this site very quickly.


Limit doesn't turn out as expected

Question: I'm trying to find the limit of the following function at $x=\frac{\pi}{4}$ and as $n$ approaches infinity:
$$\frac{\ln(\tan x)-\tan^{n}x\lfloor\sin(\tan\frac{x}{2})\rfloor}{1+\tan^{n}x}.$$
The left-hand limit is $\ln(\tan x)$ as expected, but apparently the right-hand limit is $-\lfloor\sin(\tan\frac{x}{2})\rfloor$.
Now, since the $\tan^{n}x$ in the numerator, which approaches infinity, is multiplied with $0$ (since $\sin(\tan\frac{x}{2}$) lies between $0$ and $1$, and the floor function turns it to $0$), shouldn't it end up as $0$? And while the log value is a small positive value, the denominator approaches infinity, so shouldn't the final right hand limit come out as zero?
What's the error I'm making here?
Comment: Oh, sorry, apparently they're both the same! When you evaluate the expression with the floor function with x =$\frac{π}{4}$, it ends up as zero! Is this right? Should I delete this?
Answer: On the interval $\left(\frac{\pi}{8},\frac{3\pi}{8}\right)$, we find that $\lfloor \sin(\tan(x/2))\rfloor=0$, so on that interval we have that
$$ \frac{\ln(\tan x)-\tan^{n}x\lfloor\sin(\tan\frac{x}{2})\rfloor}{1+\tan^{n}x}=\frac{\ln(\tan x)}{1+\tan^nx} $$
But this function is continuous on that interval, for any integer $n$ and equals $0$ when $x=\frac{\pi}{4}$.
So
$$ \lim_{n\to\infty}\lim_{x\to\pi/4}\frac{\ln(\tan x)-\tan^{n}x\lfloor\sin(\tan\frac{x}{2})\rfloor}{1+\tan^{n}x}=0 $$
Comment: Isn't the interval $(0, \frac{\pi}{4}$?
Comment: If you are taking a limit as $x$ approaches $\pi/4$ from the left, then you only need to consider an interval to the left of $\pi/4$, but the undirected limit is $0$ regardless.
Comment: Yes, so shouldn't the interval be $(0, \frac{\pi}{4})$ and not $(\frac{\pi}{8},\frac{3\pi}{8})$?
Comment: There is no 'the interval'. The point is that there exista 'an interval' containing $\pi/4$ over which the function simplifies, and the limit of the simplified function is $0$ at $\pi/4$. The interval $(0,\pi/4)$ is not relevant to the issue.

Limit doesn't turn out as expected


Is making in-place operations return the object a bad idea?

Question: I'm talking mostly about Python here, but I suppose this probably holds for most languages.  If I have a mutable object, is it a bad idea to make an in-place operation also return the object? It seems like most examples just modify the object and return None.  For example, list.sort.
Comment: I'm not 100% sure, but most of the time, there's no need for an in-place operation to return an object. You're not creating a new object that needs to assigned, after all. Additionally, there are analogs to each in-place operation to make the fact that you're returning something to do further operations on it explicitly obvious. (e.g. `list.sort` vs. `sorted(list)`, `list.reverse` vs. `reversed(list)`)
Comment: I think it's all about consistency. Python is pretty consistent about methods on mutable objects being in-place operations. As long as you are consistent about it, there shouldn't be an issue with in-place operations returning an object or object reference.
Comment: But why is it like that in the first place?
Answer: Yes, it is a bad idea.  The reason is that if in-place and non-in-place operations have apparently identical output, then programmers will frequently mix up in-place operations and non-in-place operations (List.sort() vs. sorted()) and that results in hard-to-detect errors.
In-place operations returning themselves can allow you to perform "method chaining", however, this is bad practice because you may bury functions with side-effects in the middle of a chain by accident.
To prevent errors like this, method chains should only have one method with side-effects, and that function should be at the end of the chain.  Functions before that in the chain should transform the input without side-effects (for instance, navigating a tree, slicing a string, etc.).  If in-place operations return themselves then a programmer is bound to accidentally use it in place of an alternative function that returns a copy and therefore has no side effects (again, List.sort() vs. sorted()) which may result in an error that is difficult to debug.
This is the reason Python standard library functions always either return a copy or return None and modify objects in-place, but never modify objects in-place and also return themselves.  Other Python libraries like Django also follow this practice (see this very similar question about Django).
Comment: Yes, the point is that `sort` fails immediately when it is used for its return value (because it returns `None`), instead of silently causing a side-effect that the developer may not intend to cause.
Comment: Agreed as a general rule, but I think there are exceptions in some specific cases, which are not so rare. Eg1: When the semantic of the method is clearly an in place operation, like jQuery's `.empty()`. Eg2: When the API is so commonly used that everybody knows it from the first getting started and has no version that return's a copy, like jQuery's `.append()`
Comment: Just because the method name is a present tense verb, doesn't mean that people will find it obvious that the operation acts in-place.  It took me a long time after learning Python before I always remembered that `list.sort` acted in-place, even though the name sounds like it should do that.
Comment: Isn't there still confusion with ending a chain with an in-place operation?  `a.sort()` and `a[:2].sort()` are going to do completely different things (I guess it would be different if you used something like a numpy `array`, which uses views).  Maybe the point is that `sort` returning `None` protects you from thinking that `a[:2].sort()` does anything useful?
Answer: Returning the modified object from the method that modified it can have some benefits, but is not recommended in Python. Returning self after a modification operation will allow you to perform method chaining on the object, which is a convenient way of executing several methods on the same object, it's a very common idiom in object-oriented programming. And in turn, method chaining allows a straightforward implementation of fluent interfaces. Also, it allows some functional-programming idioms to be expressed more easily.
To name a few examples: in Python, the Moka library uses method chaining. In Java, the StringBuilder class allows multiple append() invocations on the same object. In JavaScript, JQuery uses method chaining extensively. Smalltalk takes this idea to the next level: by default, all methods return self unless otherwise specified (therefore encouraging method chaining) - contrast this with Python, which returns None by default.
The use of this idiom is not common in Python, because Python abides by the Command/Query Separation Principle, which  states that "every method should either be a command that performs an action, or a query that returns data to the caller, but not both".
All things considered, whether it's a good or bad idea to return self at the end is a matter of programming culture and convention, mixed with personal taste. As mentioned above, some programming languages encourage this (like Smalltalk) whereas others discourage it (like Python). Each point of view has advantages and disadvantages, open to heated discussions. If you're a by-the-book Pythonist, better refrain from returning self - just be aware that sometimes it can be useful to break this rule.
Comment: Thanks for the excellent answer, especially the link to the Command/Query Separation Principle, which helps me attach a label to some design trade-offs I've been mulling over lately.
Answer: The answers here about not returning from in-place operations messed me up for a bit until I came across this other SO post that links to the Python documentation (which I thought I read, but must have only skimmed). The documentation, in reference to in-place operators, says:

These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self).

When I tried to use the in-place operation without returning self, then it became None. In this example, it will say vars requires an object with __dict__. Looking at the type of self there shows None.
# Skipping type enforcement and such.
from copy import copy
import operator
import imported_utility # example.
class A:
    def __init__(self, a, b):
        self.a = a
        self.b = b
    def one(self, scaler):
        self *= scaler
        return imported_utility(vars(self))
    def two(self, scaler):
        tmp = self * scaler
        return imported_utility(vars(tmp))
    def three(self, scaler):
        return imported_utility(vars(self * scaler))
    # ... addition, subtraction, etc.; as below.
    def __mul__(self, other):
        tmp = copy(self)
        tmp._inplace_operation(other, operator.imul)
        return tmp
    def __imul__(self, other): # fails.
        self._inplace_operation(other, operator.imul)
    # Fails for __imul__.
    def _inplace_operation(self, other, op):
        self.a = op(self.a, other)
        self.b = op(self.b, other)

* works (two and three), but *= (one) does not until self is returned.
    def __imul__(self, other):
        return self._inplace_operation(other, operator.imul)
    def _inplace_operation(self, other, op):
        self.a = op(self.a, other)
        self.b = op(self.b, other)
        return self

I do not fully understand this behavior, but a follow-up comment to the referenced post, says without returning self, the in-place method is truly modifying that object, but rebinding its name to None. Unless self is returned, Python does not know what to rebind to. That behavior can be seen by keeping a separate reference to the object.
Answer: I suppose it depends on the use case.  I don't see why returning an object from an in-place operation would hurt, other than maybe you won't use the result, but that's not really a problem if you're not being super-fastidious about pure functionalism.  I like the call-chaining pattern, such as jQuery uses, so I appreciate it when functions return the object they've acted upon, in case I want to use it further.


Interpretation of interaction terms if main effect is insignificant

Question: I have read the questions posted here about how to deal with the situation of significant interaction terms but insignificant 'main' effects, and my main take-away is that it depends on your research question how to proceed and interpret your results.
I would appreciate your thoughts on my interpretation in the context of my research question.
Hypothesis: The relation between height and weight is stronger for males than for females.
$$Weight = a + b_1 \times Height + b_2 \times Male + b_3 \times (Height \times Male) + e$$
Results:
$b_1$ is not significantly different from zero.
$b_3$, the interaction term, is positive and significantly different from zero.
Can I conclude that I found evidence in support of my hypothesis (that the relation between height and weight is stronger for males than for females)?
Thank you!
Comment: It might depend on what you mean by "stronger".  At least two senses are natural in this context--statistical significance and size of effect. The latter is usually more meaningful, because the former is largely an artifact of sample size and data variation.  But how, then, are you measuring effect size when there is an *interaction*? It seems to me that at a minimum you will need to account for $b_1$ in your interpretation.
Comment: but isn't the fact, that there is an significant interaction (with a positive value) evidence that the effect size is bigger for males than for females?
Comment: sorry @whuber I confused b2 with b1. I edited my original post.
Answer: Rather than saying the relationship is stronger, I think it's more precise to say that weight increases significantly more quickly with height for males than for females. Strength of relationship would be measured by measures like $R^2$, and these are affected not only by the rate of increase of one variable with another, but by the amount of noise in the data. e.g. if the data were something like this:
maleheight <- rnorm(1000, 70, 3)
femaleheight <- rnorm(1000, 65, 2.5)
maleweight <- maleheight*2.2 + rnorm(1000, 0, 20)
femaleweight <- femaleheight*1.3 + rnorm(1000, 0, 10)
height <- c(maleheight, femaleheight)
weight <- c(maleweight, femaleweight)
male <- c(rep(1, 1000), rep(0, 1000))
data <- data.frame(cbind(height, weight, male))

and the model
m1 <- with(data, lm(weight~height + male + height*male))
summary(m1)

shows your pattern, but the relationship looks stronger for women
Comment: It wouldn't change things. $R^2$ is a measure of strength of relationship. The coefficient of the intercept term is a measure of whether the slope of the line would be different for men and women, regardless of whether the coefficient for male is significant, or whether the coefficient for height is significant.
Comment: thank you @Peter Flom! Sorry, I confused the names of coefficients in my original post. You're example corresponds to the situation where the dummy 'Male' is not significant. However, how would you interpret the results, if the coefficient 'Height' is not significant while the interaction term (Height * Male) is?


Two-handed "attack" return on a serve?

Question: So I went over the Volleyball Canada rulebook for 2018-2019, but I couldn't find anything about a certain situation that occurs often when I play volleyball with friends/colleagues.
What will sometimes happen is that, during a service, a player on the front-row of the receiving team will jump and return the ball to the serving team by hitting it with both hands at the same time (as if he was trying to "spike" it with two hands). We're wondering whether this move is legal or not:

Is returning a serve from the front row by jumping legal?
Is hitting the ball (in a spike-like motion) with two hands at the same time (contact is made with both hands) legal?

If that helps, when this happens, the ball is usually not over the net (net height), but below it. From our point of view, it looks like a full blown "block" or "attack". Not a receive, nor a set attempt.
Answer: I don't believe that the difference between one hand or two makes a difference in this scenario, but I've found relevant rules to reference.
Returning or blocking a serve is illegal
Listed under section 13.3 (Faults of the Attack Hit) of the 2018-2019 Volleyball Canada rulebook is rule 13.3.4 which describes a fault triggered by attacking an opponent's serve.

13.3.4 A player completes an attack hit on the opponent's service, when the ball is in the front zone and entirely higher than the top of the net (13.2.4).

If you attempt to return the serve from the front row and the ball is higher than the top of the net, you are committing a fault.
Additionally, rule 14.5 addresses blocking a serve rather directly.

14.5 BLOCKING THE SERVICE
To block an opponent's service is forbidden.

So regardless of how the ball is hit, if you attempt to block or return a serve from the front row, you lose the point.
Finally, I'll say that technically you can hit the ball with both hands so long as the contact with both hands/arms occurs simultaneously.

9.2.3 The ball may touch various parts of the body, provided that the contacts take place
simultaneously.

Two-handed "attack" return on a serve?


I want to help Finder Overlay FullPath!(List Overlay)

Question: class_addMethod(finder_class, @selector(FO_drawIconWithFrame:),
                    class_getMethodImplementation(self_class, @selector(FO_drawIconWithFrame:)),"v@:{CGRect={CGPoint=dd}{CGSize=dd}}");

    old = class_getInstanceMethod(finder_class, @selector(drawIconWithFrame:));
    new = class_getInstanceMethod(finder_class, @selector(FO_drawIconWithFrame:));
    method_exchangeImplementations(old, new);

// AND//

FO_drawIconWithFrame
{
...
NSString *path = [self objectValue];
...
}

i want to FullPath. but [self node] <- ERROR
Answer: Did this work ?
NSURL *url = [[NSClassFromString(@"FINode") nodeFromNodeRef:[(TNodeIconAndNameCell *)self node ]->fNodeRef] previewItemURL];

NSString *path = [url path];


Updating WordPress 3.6 to 3.7 via admin area on Nginx VPS hangs and fails

Question: So I have a few WordPress sites running on my VPS (Ubuntu 12.10, Nginx, php-fpm 5.4)
The sites are all on seperate vhosts and use their own config files (albeit similar to each other) and vary in complexity. One is very simple and uses minimal plugins.
When I try to update core on any site via the admin area I click the "Update Now" button (which should run the script in wp-admin/update-core.php the page hangs for a minute or two before going to a blank admin page (i.e. the wp-admin menu bars and header bar are there but there is no content in the body of the page). Visiting another admin page via the still menu bar reveals that the core has not been updated.
Checking the error log I see this entry:

2013/10/29 23:20:48 [error] 9384#0: *5318248 upstream timed out (110: Connection timed out) while reading upstream, client: --.---.--.---, server: www.mysite.com, request: "POST /wp-admin/update-core.php?action=do-core-upgrade HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "mysite.com", referrer: "http://mysite.com/wp-admin/update-core.php"

This didn't happen in the past on older updates and the rest of the site including updating plugins works fine.
Any ideas? Could it be as simple as a time-out error? I find that unlikely as the server should munch though a wp upgrade in seconds.
Comment: Try to strace a PHP upgrade process to find what could be causing the issue, could it be a network related timeout, or there is something else in play.
Comment: I don't think it's network related as the rest of the site works fine and updating plugins works too. How would I trace a php upgrade process?
Answer: Two things you'd want to check with WP updates from my experience.

network problems, do a netstat on server to see if there are some
connections from php process that is updating and an external server.
some servers are slower than others.
limited number of php-fpm workers, I've seen WP updates calling themselves via HTTP so it locks
all available workers.

try:

php-fpm's slowlog
higher nginx's side timeout on fastcgi
strace is the best thing next to nutella. use it.
Comment: I'm having trouble with the slowlog. When I edit `/etc/php5/fpm/pool.d/www.conf` and uncomment the slowlog lines I get a 502 bad gateway error
Answer: WordPress update is a slow process.
Add
request_terminate_timeout = 300s

to your www.conf
You need to increase fastcgi read timeout in your nginx vhost file
fastcgi_read_timeout 300s

Also, check your execution time in php ini, or redefine it in www.conf for php-fpm
php_admin_value[execution_time] = 300

Comment: Can we clarify which www.conf we are talking about. I am still confused as to what config files apply to what. I thought nginx.conf is Nginx's and php-fpm's is called php-fpm.conf...
Comment: OK I increased the timeout in the vhost config file, went to update and it updated in about 2 seconds. Very strange, but it works!


Sending the token to server paypal using angularjs

Question: I have successfully acquired a single-use credit card token, i need to send this token to my server to complete the billing process. i will probably want to use Angular's built-in $http module or a third-party REST API library (such as RestAngular) for sending the credit card token to my server.
i use $http module like this:
<code>$http({

             url: 'https://api.sandbox.paypal.com/v1/payments/payment',
             method: 'GET',
             headers: {

                 'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + $localStorage.accessTokenPayPal,
             },

            }).success(function (data, status, headers, config) {
                         //console.log(data);
                           alert("valide: "+ JSON.stringify(data));

             }).error(function (error) {
                         //console.log(data);
                          alert("error:"+JSON.stringify(error));
             });
</code>
but i found always this message :
<code>{"count":0}
</code>
what's the paroblem please ??
Answer: Which version of Angularjs are you using? Since 1.4 version of angularjs, the $http doesn't support the success and error methods. Please see the deprecation notice at this link. Please use the .then with success and failure callbacks. If this is not the issue with your problem, please give us a detailed error message or a console log message for the call.
Thank you,
Soma.


Postfix configuration in Ubuntu is not sendin e-mail to Yahoo

Question: Is the first time that I configure a Postfix Mail service in Ubuntu. The situation I'm facing is that recover password e-mails are being sent to G-mail accounts, but no e-mails are being sent to Yahoo e-mails accounts.  I configured Postfix for Google Apps which is what I'm using to manage e-mails for my domain.  From the mail error log I got:
<code>Nov 22 22:46:41 ip-172-31-xx-xx postfix/smtp[25837]: 6AEE443548: to=<firstname.lastname@example.com>, relay=alt1.aspmx.l.google.com[64.233.xxx.xx]:25, delay=1.3, delays=0.01/0/0.92/0.37, dsn=5.1.1, status=bounced (host alt1.aspmx.l.google.com[64.233.xxx.xx] said: 550-5.1.1   Learn more at 550 5.1.1  https://support.google.com/mail/answer/6596 g9si5019491qhg.69 - gsmtp (in reply to RCPT TO command))
Nov 22 22:46:42 ip-172-31-xx-xx postfix/cleanup[25835]: 13C7443549: message-id=<firstname.lastname@example.com>
Nov 22 22:46:42 ip-172-31-xx-xx postfix/qmgr[1196]: 13C7443549: from=<>, size=3433, nrcpt=1 (queue active)
Nov 22 22:46:42 ip-172-31-xx-xx postfix/bounce[25838]: 6AEE443548: sender non-delivery notification: 13C7443549
Nov 22 22:46:42 ip-172-31-xx-xx postfix/qmgr[1196]: 6AEE443548: removed
</code>
My instance is an AWS Ec2 Instance using Ubuntu, and Apache.
If you need to gave a look to another .cf mail configuration file please let me know so that I can provide it to you.
I'm assuming is a special Yahoo servers configuration that is not permitting to receive e-mails.  I've search to see if my domain is blacklisted, but everything appears to be ok in the records that I've read.
How an I solve this problem?  Do I have to add an special configuration to my  Postfix so I can send e-mails to Yahoo?
Regards,
Nelitza M.
Than you very much for your time.  I really appreciate it.
Comment: Why are you trying to relay Yahoo mail through Google?
Answer: There may be more going on but with this message it appears that the address in invalid.
550, "5.1.1", The email account that you tried to reach does not exist. Please try double-checking the recipient's email address for typos or unnecessary spaces. For more inform
https://support.google.com/mail/answer/6596
Comment: Is my own e-mail address the one I'm using to test, and is well written.
Comment: I don't have enough info to tell you more... Just telling you what your error is saying. :)
Comment: I know, thank you.  Others users from Yahoo are experimenting the same thing...Maybe its a google app configuration.  I'll keep everyone posted about the issue.  If anyone else has something else in mind, please share it I will implement your solution and provide feedback if it solved it or not.
Comment: Are you able to relay to other domains besides google.com?
Comment: Is not sending to Outlook either.  I assume its only sending to Google Apps related accounts and to Gmail accounts. How do I add others e-mails provider to the Postfix configurations or how do I enable to send it to other emails provider different from Google Apps or G-mail accounts?
Comment: I found this that seems similar. http://unix.stackexchange.com/questions/91209/do-we-need-to-configure-postfix-for-using-google-apps-gmail
Comment: I came to the conclusion that the problem was related to the Google App third party configuration in my Postfix. What I proceeded to do is to remove it, install it again using the standard Ubuntu configuration.  Now its working for all the email providers.

Thank you for the feedback Eddie Dunn.
Answer: How I resolved my problem is to re-install Postfix using the following configuration:
https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-postfix-as-a-send-only-smtp-server-on-ubuntu-14-04
Now its working for every e-mal provider.


Intuition behind the Fourier series

Question: I have a simple question:
What is the (analytical) intuition behind the Fourier series:
$f(t)=\frac{a_{0}}{2}+\sum_{k=1}^{\infty}\left(a_{k}\cos\left(kt\right)+b_{k}\sin\left(kt\right)\right)$
I have read in an article that this expression results from $f(t)=\sum_{k=1}^{\infty} d_n\cos (nt+\phi_n)$ using some trigonometric theorems. But the question is still: why can I write every function as the second expression?
Comment: Arguably, there is no good analytical intuition for being able to write $f$ as a Fourier series. When Fourier first claimed such a thing, he was banned from publishing for over a decade because the prominent Mathematicians at the time thought his claim was blatantly false.
Comment: I think you should just keep going, intuition will come. This is NOT obvious at all, otherwise it would not have been such a great breakthrough.
Comment: You can read this: https://betterexplained.com/articles/an-interactive-guide-to-the-fourier-transform/
Comment: Actually you can't "write every function" in general. For example, if $f(t)$ is not periodic it is impossible. Also, [convergence](https://en.wikipedia.org/wiki/Convergence_of_Fourier_series) can be tricky if $f(t)$ is not integrable. See also the Gibbs phenomenon.
Answer: I assume that you are familiar with Taylor expansion. In principle this just says that all (analytic) functions can be written as a sum of polynomials. This is in the beginning not trivial at all, but seems to work just fine. I assume that you got used to this fact.
Same thing can be said about the Fourierseries. It's not obvious that this is indeed possible, but it might just work (and as mathematics show, it does indeed work). Fourierseries is in a way similar to Taylor series. We write a function as an infinite sum of other functions.
The space of all functions can be seen as a vector space (if you don't know this, maybe just ignore what's coming next). We can try to find, as with any vector space, a base. Apparently all the sin and cosine fucntions happen to be a base for this vector space.
Answer: Any function $ f(x)$, between limits $ a $ and $b$ can be expressed as sum of sinusoidals . This is based on the same logic as that of the Taylor/ Maclaurin series, but here for a bounded range of input.
You can basically prove it in reverse by determining the values of the constants by differentiating on both sides w.r.t x and substituting x=0.
Answer: The reason it works is that ∫ fn*(f1+f2+f3+...) = ∫ (fnf1) + ∫ (fnf2) + ∫ (fn*f3) +....and multiplying any 2 sine waves of frequency n1 and frequency n2 (of period p/n1 and p/n2) and integrating over one period p will always equal zero unless n1=n2. See graph of sin2(x)

Intuition behind the Fourier series