Perhaps a more advanced way to prove Cantor–Bernstein

Question: I was told the following, and am having a lot of trouble understanding how this can be true.
Apparently if we have any two arbitrary functions $f:X \to Y$ and $g: Y \to X$ then there exist subsets $X_1$ and $X_2$ of $X$ and $Y_1, Y_2$ of $Y$ such that $X_1 \cup X_2=X$ , $X_1 \cap X_2= \emptyset$, $Y_1 \cup Y_2= Y, Y_1 \cap Y_2 = \emptyset$ and moreover $f(X_1)=Y_1$ and $g(Y_2)=X_2$.
Finally, this can be used to actually prove the Cantor–Bernstein theorem.
Here are my questions, how do we know that such subsets even exist? What would they have to be? Would it be related to fixed points? And even if it is true, how are we able to prove the theorem just from it?
Thanks all for time and help
Comment: Check out the proof given [on Wikipedia](https://en.wikipedia.org/wiki/Schröder–Bernstein_theorem), using sequences of elements alternating from $X$ and from $Y$ ($A$ and $B$ on the page). I think it's very clear.
Comment: I doubt anyone would say one can prove the Cantor Bernstein theorem "just" from this. It would be one step among others, one idea among others in the proof.
Comment: Well the professor said it. He is notorious for being overly difficult and unwilling to respond, so to clarify is not an option. This is for an undergrad course, first course in logic. What he said is verbatim what I wrote above. (Only a few weeks ago we began defining what a set and even a function were). I am very confused because I try my best to keep up, and am completely stuck with this. I dont like to make excuses, but I really am not sure this is possible for me to do with my background and what we have learnt in this course or any other for that matter.
Comment: I don't offhand know a proof that starts by proving the existence of such $X_1,X_2,Y_1$, and $Y_2$; it falls out indirectly from the proof that I gave in detail [here](http://math.stackexchange.com/a/225635/12042). On the same webpage **Michael Greinecker** gives another nice proof of the theorem.
Comment: I think the cited theorem does imply Cantor-Bernstein quite directly. If, as in the hypothesis of the C-B theorem, $f$ and $g$ are one-to one, then, after getting $X_1,X_2,Y_1,Y_2$ as in the cited result, you could map $X$ bijectively to $Y$ by $(f\upharpoonright X_1)\cup((g\upharpoonright Y_2)^{-1})$.
Comment: Thanks but I still am not sure how to get the bijection after we assume injectiveness
Answer: I. Lemma (Knaster–Tarski Fixed Point Theorem). If a function $\varphi:\mathcal P(A)\to\mathcal P(A)$ satisfies the condition
$$X\subseteq Y\implies\varphi(X)\subseteq\varphi(Y)$$
for all $X,Y\subseteq A,$ then $\varphi(S)=S$ for some $S\subseteq A.$
Proof. Let $\mathcal F=\{X:X\subseteq\varphi(X)\}$ and let $S=\bigcup\mathcal F.$ First note that, if $X\in\mathcal F,$ then $X\subseteq S$ and $X\subseteq\varphi(X)\subseteq\varphi(S).$ Since every member of $\mathcal F$ is a subset of $\varphi(S),$ it follows that $S=\bigcup\mathcal F\subseteq\varphi(S),$ that is, $S\in\mathcal F.$ From $S\in\mathcal F$ it follows that $\varphi(S)\in\mathcal F,$ whence $\varphi(S)\subseteq S,$ and so $\varphi(S)=S.$
II. Theorem (Banach Mapping Theorem). For any two functions $f:X\to Y$ and $g:Y\to X$ there exist subsets $X_1,X_2$ of $X$ and $Y_1,Y_2$ of $Y$ such that $X_1\cup X_2=X,\ $$X_1\cap X_2=\emptyset,\ Y_1\cup Y_2=Y,\ Y_1\cap Y_2 = \emptyset$ and moreover $f(X_1)=Y_1$ and $g(Y_2)=X_2$.
Proof. Define a function $\varphi:\mathcal P(X)\to\mathcal P(X)$ by setting
$$\varphi(S)=X\setminus g(Y\setminus f(S)),$$
and observe that $S\subseteq T\implies\varphi(S)\subseteq\varphi(T).$ Hence, by the lemma, there is a set $X_1\subseteq X$ such that $X_1=\varphi(X_1)=X\setminus g(Y\setminus f(X_1)),\ $ i.e., $$X\setminus X_1=g(Y\setminus f(X_1)).$$
Let
$$Y_1=f(X_1),$$
$$Y_2=Y\setminus Y_1=Y\setminus f(X_1),$$
$$X_2=g(Y_2)=g(Y\setminus f(X_1))=X\setminus X_1,$$
and everything is fine.
III. Corollary. (Cantor—Bernstein Theorem). If there are injections $f:X\to Y$ and $g:Y\to X,$ then there is a bijection $h:X\to Y.$
Proof. With $X_1,X_2,Y_1,Y_2$ as above, let $h=(f\upharpoonright X_1)\cup(g\upharpoonright Y_2)^{-1}.$
Comment: I think you meant $X_1=\varphi(X_1)=X\setminus g(Y\setminus f(X_1))$ rather than $X_1=\varphi(X_1)=X\setminus g(Y\setminus f(S))$.
Comment: @DungLe Thanks for the correction.
Answer: Let $f:X\to Y$ and $g:Y\to X$ be injections. For $p\in X,$ consider the sequence $$(p,g^{-1}(p), f^{-1}g^{-1}(p), g^{-1}f^{-1}g^{-1}(p),...)$$  which may terminate after finitely many terms (when an inverse fails to exist), or may continue without end.
Let $O$ be the set of $p\in X$ for which the sequence has an odd number of terms. Let $E$ be the set of $p\in X$ for which the sequence has an even number of terms. Let $I$ be the set of $p\in X$ for which the sequence has no end.
Now let $h(p)=f(p)$ if $p\in O,$ and $h(p)=g^{-1}(p)$ if $p\in E \cup I.$
(1A).Clearly if $\{p_1,p_2\}\subset O$ or if $\{p_1,p_2\}\subset E$ then $h(p_1)=h(p_2)\implies p_1=p_2.$
(1B).Suppose $p_1\in O$ while $p_2\in E\cup I.$ Then $$h(p_1)=h(p_2)\implies f(p_1)=g^{-1}(p_2)\implies p_1=f^{-1}g^{-1}(p_2).$$ But if $p_2 \in E\cup I$ and if $ f^{-1}g^{-1}(p_2)$ exists then $f^{-1}g^{-1}(p_2)\in E\cup I,$ so $p_1\in E\cup I, $ a contradiction to $p_1\in O.$
(1C).Therefore $h:X\to Y$ is an injection.
(2A).  For $q\in Y,$ if $g(q)\in E\cup I$ then $h(g(q))=g^{-1}g(q)=q.$
(2B).If $g(q)=x\in O,$ then $g^{-1}(x)$ exists and the sequence $(x,g^{-1}(x), ...)$ has an odd number of terms. So $f^{-1}g^{-1}(x)$ exists, and it also belongs to $O.$ So $h(f^{-1}g^{-1}(x))=f(f^{-1}g^{-1}(x))=g^{-1}(x)=g^{-1}(g(q))=q.$
(2C). Therfore $h:X\to Y$ is a surjection.
I think that drawing a heuristic picture, with $X$ and $Y$ as parallel lines, may help you to grasp this.
Comment: Hi @Daniel! Let $SE(p)$ is the sequence of $p$. In (1B) $SE(p_2)=(p_2,g^{-1}(p_2), f^{-1}g^{-1}(p_2)=p_1,...) \implies card(SE(p_1))+2= card(SE(p_2))$. This implies $card(SE(p_1))$ is even, or $card(SE(p_1))=\infty \implies p_1 \in E\cup I$ (Contradiction). Is my reasoning correct?
Comment: @DungLe. Yes. Correct.
Comment: Your solution is a very nice variant of Julius König's proof :)
Comment: @DungLe. See "Introduction To Topology And Modern Analysis" by Simmons. (His notation differs.).... Textbooks seem to vary between this proof and another,  which is long & complicated, that I have  never tried to read all of.... You may notice that I said "or may continue without end" rather than "or is an infinite sequence" so it is valid in the absence of the Axiom of Infinity
Comment: I'm just recently exposed to set theory. Regarding "or is an infinite sequence", I think that when we mention "infinite sequence", we implicitly appeal to the set of natural numbers $\mathbb{N}$, and in turn appeal to Axiom of Infinity. But I still not get why  "or may continue without end" need not to appeal to Axiom of Infinity? Please explain this point in a bit more detail!
Comment: @DungLe. To explain it in detail would be about a chapter of  a book.....
Comment: Anyway, thank you so much @Daniel ^^


Drupal 7: can't get ckEditor module configured for inline editing

Question: Drupal 7: I using ckEditor module for editing within the textarea fields and I noticed on the ckeditor website that you can do inline editing like:
http://ckeditor.com/demo#inline
I can get the ckEditor module to work for regular editing but I can't seem to get it configured for inline editing like the demo listed above.
Answer: I believe you need Quick Edit, which is the Drupal 7 backport of the Spark core initiative for Drupal 8.


ssl.conf SSLCipherSuite not working

Question: I have an AWS server and have been testing it with https://www.ssllabs.com/ssltest/ to determine if I'm setting things up correctly.
Type: Amazon Linux AMI 2
Apache version: 2.4.39
OpenSSL: 1.0.2k-fips

I'm looking at the Cipher Suites in the results and it's showing a lot of 'weak' Ciphers.
I've attempted to make changes to the SSLCipherSuite within ssl.conf but nothing seems to work (as in, nothing seems to even change the Cipher Suites being listed). I've even commented out SSLCipherSuite and it doesn't affect the output of ssllabs at all!
My ssl.conf has this:
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:!aNULL:!MD5
SSLHonorCipherOrder on

I have always restarted Apache after each change.
I can only guess it might be pulling the cipher suites from somewhere else?
I'm at a loss and any help would be appreciated.
Comment: "I can only guess it might be pulling the cipher suites from somewhere else? " No. As for configuration, look at https://wiki.mozilla.org/Security/Server_Side_TLS#Recommended_configurations and https://ssl-config.mozilla.org/ that will help you give proper configuration directives for various software based on the  security "level" you want to achieve.
Answer: I left out I was using Letsencrypt as certificate provider. I was able to fix the issue and would like to leave this answer up for others who might run into this issue as well.
In my httpd.conf, there was
IncludeOptional conf.d/*.conf
IncludeOptional sites-enabled/*.conf
Include /etc/httpd/sites-available/virtualhosts-le-ssl.conf

Inside my virtualhosts-le-ssl.conf was this line:
Include /etc/letsencrypt/options-ssl-apache.conf

I opened that file and it had SSL settings so I can only assume it was indeed overwriting my SSLCipherSuite settings. What's baffling is I can use ssl.conf to overwrite my SSLProtocol settings but can't do the same with SSLCipherSuite.
After changing the SSLCipherSuite inside virtualhosts-le-ssl.conf, I was able to successfully modify the Cipher Suite list being shown in ssllabs and have since tweaked it into something more secure.
Hopefully, this helps someone else.
Cheers!

ssl.conf SSLCipherSuite not working


How to throttle WCF service per client

Question: I'm developing a service that will be exposed on the internet to only a few select clients. However I don't want one client to be able to call the service so often that they prevent another client calling the service, or being responded to within a reasonable time. I realise that WCF has a number of throttling configuration settings built in, but I understand that these are only for the service as a whole.
Is there any built in mechanism that will enable me to configure the service such that a single client can only carry out (eg.) 10 concurrent calls or similar?
This question is related to another question here:
Best way to secure a WCF service on the internet with few clients
where I'm still trying to determine if I need to, and how I will identify individual clients.
Comment: I don't think there is a configurable way to achieve what you want. You could get the client ip address in the wcf and block out a particular ip from exceeding a call limit.
Answer: The phrase you're looking for here is Rate limiting. And, no, there's no built-in way to rate-limit a WCF service. As you said, you can play with the WCF feature set around service throttling, but this is a service-level setting and not per-client.
In order to implement rate limiting the general guidance seems to be to use an in-memory collection (or something like redis for scale-out scenarios) to perform fast look-ups against the incoming user string or IP address. Then you can define some limiting algorithm around that information.
More info here and here.
Answer: Firstly if you are using some kind of load balancer it is best to implement it there. For instance NGINX have rate limiting capabilities: http://nginx.org/en/docs/http/ngx_http_limit_req_module.html .
Secondly you should consider using built in IIS rate limiting capabilities called dynamic IP restrictions: http://www.iis.net/learn/get-started/whats-new-in-iis-8/iis-80-dynamic-ip-address-restrictions .
If both of this are not enough for you because you need custom logic for this you can always implement it on the application level. This can be done in multiple ways.
Let's start with some reusable rate limiting logic:
public interface IRateLimiter
{
    bool ShouldLimit(string key);

    HttpStatusCode LimitStatusCode { get; }
}

public interface IRateLimiterConfiguration
{
    int Treshhold { get; set; }
    TimeSpan TimePeriod { get; set; }
    HttpStatusCode LimitStatusCode { get; set; }
}

public class RateLimiterConfiguration : System.Configuration.ConfigurationSection, IRateLimiterConfiguration
{
    private const string TimePeriodConst = "timePeriod";
    private const string LimitStatusCodeConst = "limitStatusCode";
    private const string TreshholdConst = "treshhold";
    private const string RateLimiterTypeConst = "rateLimiterType";

    [ConfigurationProperty(TreshholdConst, IsRequired = true, DefaultValue = 10)]
    public int Treshhold
    {
        get { return (int)this[TreshholdConst]; }
        set { this[TreshholdConst] = value; }
    }

    [ConfigurationProperty(TimePeriodConst, IsRequired = true)]
    [TypeConverter(typeof(TimeSpanConverter))]
    public TimeSpan TimePeriod
    {
        get { return (TimeSpan)this[TimePeriodConst]; }
        set { this[TimePeriodConst] = value; }
    }

    [ConfigurationProperty(LimitStatusCodeConst, IsRequired = false, DefaultValue = HttpStatusCode.Forbidden)]
    public HttpStatusCode LimitStatusCode
    {
        get { return (HttpStatusCode)this[LimitStatusCodeConst]; }
        set { this[LimitStatusCodeConst] = value; }
    }

    [ConfigurationProperty(RateLimiterTypeConst, IsRequired = true)]
    [TypeConverter(typeof(TypeNameConverter))]
    public Type RateLimiterType
    {
        get { return (Type)this[RateLimiterTypeConst]; }
        set { this[RateLimiterTypeConst] = value; }
    }
}

public class RateLimiter : IRateLimiter
{
    private readonly IRateLimiterConfiguration _configuration;
    private static readonly MemoryCache MemoryCache = MemoryCache.Default;

    public RateLimiter(IRateLimiterConfiguration configuration)
    {
        _configuration = configuration;
    }

    public virtual bool ShouldLimit(string key)
    {
        if (!string.IsNullOrEmpty(key))
        {
            Counter counter = new Counter {Count = 1};
            counter = MemoryCache.AddOrGetExisting(key, new Counter { Count = 1 }, DateTimeOffset.Now.Add(_configuration.TimePeriod)) as Counter ?? counter;
            lock (counter.LockObject)
            {
                if (counter.Count < _configuration.Treshhold)
                {
                    counter.Count++;
                }
                else
                {
                    return true;
                }
            }
        }

        return false;
    }

    public HttpStatusCode LimitStatusCode
    {
        get { return _configuration.LimitStatusCode; }
    }

    private class Counter
    {
        public volatile int Count;
        public readonly object LockObject = new object();
    }
}

public class RateLimiterFactory
{
    public IRateLimiter CreateRateLimiter()
    {
        var configuration = GetConfiguration();
        return (IRateLimiter)Activator.CreateInstance(configuration.RateLimiterType, configuration);
    }

    public static RateLimiterConfiguration GetConfiguration()
    {
        return ConfigurationManager.GetSection("rateLimiter") as RateLimiterConfiguration ?? new RateLimiterConfiguration();
    }
}

static class GetClientIpExtensions
{
    private const string XForwardedForHeaderName = "X-Forwarded-For";
    private const string HttpXForwardedForServerVariableName = "HTTP_X_FORWARDED_FOR";
    private const string HttpRemoteAddressServerVariableName = "REMOTE_ADDR";

    public static string GetClientIp(this Message message)
    {
        return GetClientIp(message.Properties);
    }

    public static string GetClientIp(this OperationContext context)
    {
        return GetClientIp(context.IncomingMessageProperties);
    }

    public static string GetClientIp(this MessageProperties messageProperties)
    {
        var endpointLoadBalancer = messageProperties[HttpRequestMessageProperty.Name] as HttpRequestMessageProperty;
        if (endpointLoadBalancer != null && endpointLoadBalancer.Headers[XForwardedForHeaderName] != null)
        {
            return endpointLoadBalancer.Headers[XForwardedForHeaderName];
        }
        else
        {
            var endpointProperty = messageProperties[RemoteEndpointMessageProperty.Name] as RemoteEndpointMessageProperty;
            return (endpointProperty == null) ? string.Empty : endpointProperty.Address;
        }
    }

    public static string GetClientIp(this HttpRequest request)
    {
        string ipList = request.ServerVariables[HttpXForwardedForServerVariableName];
        return !string.IsNullOrEmpty(ipList) ? ipList.Split(',')[0] : request.ServerVariables[HttpRemoteAddressServerVariableName];
    }
}

This uses configuration, proper segregation using interfaces and default MemoryCache. You can easily change the implementation to abstract the cache. This would allow to use different cache providers like redis for example. This could be useful if you want to have a distributed cache for multiple servers running same service.
Now having this code as a base we can add some implementations using it. We can add a IHttpModule:
public class RateLimiterHttpModule : IHttpModule
{
    private readonly IRateLimiter _rateLimiter;

    public RateLimiterHttpModule()
    {
        _rateLimiter = new RateLimiterFactory().CreateRateLimiter();
    }

    public void Init(HttpApplication context)
    {
        context.BeginRequest += OnBeginRequest;
    }

    private void OnBeginRequest(object sender, EventArgs e)
    {
        HttpApplication application = (HttpApplication)sender;
        string ip = application.Context.Request.GetClientIp();
        if (_rateLimiter.ShouldLimit(ip))
        {
            TerminateRequest(application.Context.Response);
        }
    }

    private void TerminateRequest(HttpResponse httpResponse)
    {
        httpResponse.StatusCode = (int)_rateLimiter.LimitStatusCode;
        httpResponse.SuppressContent = true;
        httpResponse.End();
    }

    public void Dispose()
    {
    }
}

Or a WCF only implementation that will work for any transport level:
public class RateLimiterDispatchMessageInspector : IDispatchMessageInspector
{
    private readonly IRateLimiter _rateLimiter;

    public RateLimiterDispatchMessageInspector(IRateLimiter rateLimiter)
    {
        _rateLimiter = rateLimiter;
    }

    public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
    {
        if (_rateLimiter.ShouldLimit(request.GetClientIp()))
        {
            request = null;
            return _rateLimiter.LimitStatusCode;
        }
        return null;
    }

    public void BeforeSendReply(ref Message reply, object correlationState)
    {
        if (correlationState is HttpStatusCode)
        {
            HttpResponseMessageProperty responseProperty = new HttpResponseMessageProperty();
            reply.Properties["httpResponse"] = responseProperty;
            responseProperty.StatusCode = (HttpStatusCode)correlationState;
        }
    }
}

public class RateLimiterServiceBehavior : IServiceBehavior
{
    public void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) { }

    public void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters) { }

    public void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
    {
        var rateLimiterFactory = new RateLimiterFactory();

        foreach (ChannelDispatcher chDisp in serviceHostBase.ChannelDispatchers)
        {
            foreach (EndpointDispatcher epDisp in chDisp.Endpoints)
            {
                epDisp.DispatchRuntime.MessageInspectors.Add(new RateLimiterDispatchMessageInspector(rateLimiterFactory.CreateRateLimiter()));
            }
        }
    }
}

public class RateLimiterBehaviorExtensionElement : BehaviorExtensionElement
{
    protected override object CreateBehavior()
    {
        return new RateLimiterServiceBehavior();
    }

    public override Type BehaviorType
    {
        get { return typeof(RateLimiterServiceBehavior); }
    }
}

You can similarly do an action filter for ASP.NET MCV. Check it out here: How do I implement rate limiting in an ASP.NET MVC site? .
Comment: From a Code review point of view, isn't `TerminateRequest` a little violent a way of proceeding. Wouldn't a `await Task.Delay` make more sense?
Comment: What would Task.Delay achieve?
Comment: It would "Delay" the response, hence rate without killing the request off. Granted you would then need to implement some kind of concurrency checking/queuing logic...But that seems like the correct way to rate limit.
Comment: Additionally your `TerminateRequest` would encourage users to repeatedly make requests, which would use up more resources...Which is the whole point of rate limiting...
Comment: Other standard rate limiters like nginx or ISS just terminate the request. So I am not convinced.
Answer: You could alter your configuration like this:
 ConcurrencyMode:=ConcurrencyMode.Single
 InstanceContextMode:=InstanceContextMode.Single

Then, in code, set up two service-level variables:

one string variable to hold the ID of the last requestor
one integer variable for the number of visits.

With each request where the ID of the inbound user == to last saved user, increment +1 your integer variable. After request 10, return a denial to the user. If the user's different, reset the variables and process the request.
It's not a configuration solution - it's configuration and code, but it would work.
Comment: A single instance of the service that only executes one call at a time?! That's the best implementation of "one client able to prevent another client calling the service" I've ever seen.
Comment: Yes, but it won't prevent the user from returning 1/1000 of a second later with another request. Basically, it's just queuing the requests.
Comment: You could host a singleton for each client. When you get a new client, just host another service instance. Great scale out strategy. Wait....

How to throttle WCF service per client


Scraping the portion of HTML code (not text) using selenium in python

Question: I want to use Selenium to scrape a portion of HTML code (not all) from a complete web page for use in my current project. I do not want the inner text of the HTML, but the actual code and store it in an external text file or a new HTML file.

Limitation: I cannot use any other library because the website is dynamic and rich in J.S only Selenium will work.
Comment: Which portion of `HTML code` are you looking for?
Comment: Please read https://meta.stackoverflow.com/questions/303812/discourage-screenshots-of-code-and-or-errors why you should not post errors or code as screenshots.
Comment: it is a sample code it is not actual code,just want people to grab the idea.
Comment: @DebanjanB you can check the sample code/
Comment: @ZeeshanAhmad Can you update the question with text based _HTML_ and the exact block of _HTML_ you are trying to _scrap_?
Answer: No, its possible to extract the selected html with selenium. Either you can extract complete html or you will have to extract each element one by one:
to extract whole header tag (including header) using outerHTML:
elem = driver.find_element_by_class_name("header cf")
source_code = elem.get_attribute("outerHTML")

to extract everything inside nav(including nav):
elem = driver.find_element_by_class_name("nav")
source_code = elem.get_attribute("outerHTML")

Comment: But how can i put the limit to the scraping, i mean how can i define the endpoint?
Comment: you don't define end point, you only define starting point and outerHTML will return the HTML of the element itself plus all the children elements
Comment: another thing you can do is to split the html after scrapping to get the required portion, see: https://stackoverflow.com/questions/42422360/python-split-at-tag-regex


Can we detect if internet in turned off by the user in the background?

Question: I am creating an application in which the location is continuously sent to the server in X minutes. Now the issue is if the user turns off the internet while in background or suspended mode, Can we send a local push notification?
Comment: I suspect you want this, but obviously notifications only will work when you are in the foreground. https://stackoverflow.com/questions/1083701/how-to-check-for-an-active-internet-connection-on-ios-or-macos
Answer: Yes it is possible to get notification in local, Go through this link for your refrence.
Hope this question will help you.


Como conseguir documentação das DLL's do Windows 7?

Question: Onde consigo uma documentação completa do funcionamento e de como usar as funções/procedures das DLL's do Windows 7?
Comment: A resposta resolveu o seu problema? Acha que pode aceitá-la? Se não souber como faz, veja o [tour]. Isso ajudaria muito indicar que a solução foi útil para você e dar uma indicação que houve uma solução satisfatória. Também pode votar em qualquer pergunta ou resposta que achar útil no site todo (quando tiver 15 pontos).
Answer: Toda documentação deste tipo fica no Docs Microsoft. Especificamente a API. O principal do Windows.


Are there any tools for comparing DNSBL effectiveness?

Question: I just added a new DNSBL to the list of ones that I use on my small mailserver.  I set it up to be first on the list because I wanted to see what it missed but other lists caught.
I'm watching mail logs out of the corner of my eye while I do other things, just to make sure that things looked sane, and I started wondering about how to measure the relative effectiveness of various lists.  I suppose I could give each list a week at the top of the heap and then measure how much the other lists catch that week, but that seems like a real hack.
Are there any tools or techniques for measuring the relative effectiveness of different blocklists on a specific server?  I can measure false positives by user complaint volume, so my concern is addresses that one blocklist misses but another catches.
Answer: If all you want to do is determine "which is the minimal set of blacklists I should subscribe to in order to catch the maximum amount of spam", then what you really want to do is:

Tally all of your blocked spam by IP address (so analyse your mail logs to get a table of blocked IPs, with the count of how many spams came from each, perhaps)
Run all those IPs through all of your blacklists (trivial to do, just needs a quick DNS lookup for each IP)
Score each blacklist against how many of the IPs (and how many spams came from each IP) each blacklist caught.

This is complicated by the time-varying nature of the contents of each of the blacklists, so perhaps having a logcheck-type process (running regularly, analysing new log entries) is the way to go, which can then get reported back to you as required.
All of this isn't all that tricky to rig up with a few lines of shell or <code>$SCRIPTING_LANGUAGE_OF_CHOICE</code>.
Of course, unless you're running so many blacklists that the time required to go through them all is excessive, the more interesting question is "how many spams didn't get caught by my blacklists, and what additional blacklists would catch those without contributing excessively to my false-positive rate."  Again, something that looks up "passed" IP addresses in a set of blacklists you're considering and records the results would be instructive.
Comment: It's common that such simple scripts don't get released, as the effort of packaging, documenting, maintaining, releasing, etc exceeds the effort of just rewriting the sucker.  Add on top of that the need to rewrite the log parsing part of the program for each MTA, and voila!
Comment: Yeah, I was thinking about doing something like that.  I'm surprised that someone hasn't already written it-- it's only a few lines of code.
Answer: In addition to DNSBL you may wish to consider is GeoIP blocking.  This is when you take a look at the IP address the connection is coming from, determine the originating country, and then block the connection.
We find that most of our spam comes from countries that we would NEVER expect to receive an email from (China, Brazil, Singapore, etc.)
By blocking these countries completely we have reduced our spam load a HUGE amount before any other sort of more CPU intensive checking is made.
I would estimate that at least 90% (probably more) of all of our spam is caught by GeoIP and less than 10% is actually processed against DNSBL, spamassassin, whitelist, etc.
We used MaxMind's free GeoIP database but I know there are others out there.
To answer your question directly though about DNSBL, we use the following:
sbl-xbl.spamhaus.org
bl.spamcop.net
dnsbl.cyberlogic.net
dnsbl.ahbl.org
We haven't had to update/change from the above in quite some time.  I would say that (for us) spamhaus.org tends to catch most stuff that GeoIP doesn't.
Your mileage may vary.
Answer: If you have spamassassin apply the blocklists, which it (effectivly) does in paralell you can use its logs to do those hit rate calculations.
Once you know what blocklists to trust you can reconfigure your mailservers to block on the most trusted RBL's.
Comment: SpamAssassin is going to be way more CPU-intensive than MTA-level blocking, no?  I'm pretty sensitive to that, though I guess I could put up with it for a few days.
Comment: Absolutely, which is why, once you know the safe blocklists you have your MTA filter them.


Can I reference field item & label in node.html.twig?

Question: In Drupal 8, I defined a content type (say example) and made a separate template for it: node--example.html.twig.
I wonder if I can reference the item or label of a specify field in node--example.html.twig, instead of reference them in field.html.twig? By which I mean can I use something like the following, which I know is not working?
{{ content.field_example_field.label }}

Comment: Item(as a Field API field attached to an entity) yes, but not the label due to the structure. Although maybe something like {{ content.myfield.und[0]['#title'] }}could potentially work.
Answer: If you can't find the label somewhere inside of content, then the code, which will prepare the label for rendering did not run at this point.
If you are inside the node twig, then there is a fully loaded node object. You can use this to get the field definition and the label:
{{ node.field_myfield.fieldDefinition.label }}

Or the first field item:
{{ node.field_myfield.0.value }}

Comment: That's what I was looking for, thx.


Calculus I Integral

Question: $\int(\sqrt x + \sec x \tan x)\, dx$  using $u$-substitution.
So far as this class has been taught...  the class knows that in order to use $u$-substitution, a $u$ and its derivative must be present in the integral.  I cannot make sense of a $u$.  Help please.
Comment: $\frac d{dx}\sec x = \sec x \tan x$
Answer: Hint. One may observe that
$$
\int \sqrt{x}\:dx=\int x^{1/2}\:dx=\frac23x^{3/2}+C
$$ and
$$
\int \sec x \tan x\:dx=-\int \frac{-\sin x}{(\cos x)^2}\:dx=-\int \frac{(\cos x)'}{(\cos x)^2}\:dx=\frac1{\cos x}+C.
$$
Answer: Hint:
Split the integral into $\int \sqrt{x}\;dx + \int \sec(x)\tan(x)\,dx$ and solve each accordingly. A substitution you can use is $u=\tan(x)$.
Answer: The first thing I would do is split the integral up into $\int\sqrt{x}dx+\int sec(x)tan(x)dx$ because this allows you to see the steps a bit more readily. The square root is a simple integral where you can use the power rule (I will leave this up to you).
Now let us consider what $sec(x)\cdot tan(x)$ is in more intuitive trig functions. We can look at this as $\frac{1}{cos(x)}\cdot \frac{sin(x)}{cos(x)}dx$. This makes our integral look like this: $\int \frac{sin(x)}{(cos(x))^2}dx$ which arguably has a more obvious u-substitution.
Hopefully this helps guide you in the right direction!

Calculus I Integral


Find all the prime integer solutions to $q^2(p-1)=(p+1)(q+1)$

Question: Let $p,q$ be prime numbers. Find all the integer solutions to:
$$q^2(p-1) = (p+1)(q+1)$$

I am almost sure that $q=2$,$p=7$ is the only solution. Thus I assumed that $p$ and $q$ were both odd to reach a contradiction, but so far I haven't made any progress that way.
Comment: I notice that $q$ cannot be very big, for example.  What have you tried?
Comment: I am almost sure that $q=2, p=7$ is the only solution. Thus I assumed that $p,q$ were both odd to reach a contradiction, but so far I haven't done any progress on that way.
Answer: We show how to solve a different problem, finding the integer solutions of $q^2(p-1)=(p+1)(q+1)$. There are not many.
We can have $p=1$, which forces $q=-1$. If $p\ne -1$, then $\left|\frac{p+1}{p-1}\right|\le 3$.
Note that if $|q|\ge 4$ then $\left|\frac{q^2}{q+1}\right|\gt 3$.
So the only candidates for $q$ lie in the interval $-3\le q\le 3$.  Try them all. In each case we get a linear equation for $p$.
If we are really only interested in primes, the only numbers that need to be tried are $q=2$ and $q=3$.
Comment: Please explain a little bit more ... How you got if $p \not = -1$, then $\left|\frac{p+1}{p-1} \right| \leq 3 $...Didn't get this
Answer: You're on the right track assuming that $p,q$ are odd.
Here's the step you may be missing: if $p>2$ is odd, then $(p-1, p+1) = 1$. But $(p+1)$ must divide both sides of the equation. So $(p+1)\mid q^2$. Hence $q$ is even.
Comment: $(p−1,p+1)=2$ if $p$ is odd.
Comment: WOAH thanks....
Comment: How did I miss that? :P
Comment: Happens to all of us :)
Comment: Right you are... give me a sec
Comment: Hmm... seems I can't delete it or fix it. Any ideas? I'm sure this is gonna be the correct kind of approach
Comment: Try reworking a solution from $q^2 | p+1$, since $q^2$ is coprime to $q+1$.
Comment: That gives $q \equiv -1$ (mod $p$)... I couldn't make that work. I've reduced it to the case $p \equiv 1$ (mod $4$)...
Answer: If we assume $p,q$ are positive primes, we can proceed by first showing $q=2$ and then deduce $p=7$.
Note $q-1 = (q^2-1)/(q+1) \lt (p+1)/(p-1)$.  For any  positive prime $p$, the right hand side is at most 3, so $q$ must be less than 4.  Thus $q$ is either 2 or 3.
If $q=3$ we can refine this estimate a little.  Since $q^2$ divides $p+1$, because $q$ and $q+1$ are coprime, for $q=3$ would mean $p+1$ is a multiple of 9.  This first happens for prime $p=17$.  In any case $(p+1)/(p-1)$ would be less than 2, so $q \lt 3$.  This contradiction implies $q=2$.
Plugging that value for $q$ into the original equation quickly gives $p=7$.
For completeness sake you should examine whether one or both $p,q$ can be negative primes.
Comment: Negative numbers are typically considered not prime.
Answer: $q^2(p−1)=(p+1)(q+1)$
This can be written as:
$\frac{q^2}{q+1} = \frac{p+1}{p-1} = 1 + \frac{2}{p-1}$
In other words, subtracting 1 from both sides,
$\frac{q^2 - q - 1}{q+1} = \frac{2}{p-1}$
Two cases here:
1) p = 2. So $q^2 - q - 1 = q+1$.
Gives: $q^2 - 2q - 2 = 0$. Easy to see this has no integer solutions (just solve the quadratic equation).
2) p is an odd prime. Let p = 2k + 1.
Then $q + 1 = k(q^2 - q - 1)$.
So $(q+1)(k+1) = kq^2$.
Now, q and q+1 are co-prime. So are k+1 and k. Unless k or q is 1, this can only mean:
$q+1 = k$ and $k+1 = q^2$
So $q+2 = q^2$
Solving the quadratic, q = 2 or q = -1. 'q is prime' makes us reject q = -1 case. So q = 2. So p = 2k+1 = 2(q+1) + 1 = 7.
Now, suppose k = 1. Then we get:
$2(q+1) = q^2$. This again gives $q^2 - 2q - 2 = 0$ which has no integer solutions.
q = 1 gives:
$2(k+1) = k$. This gives k = -1/2, which is not an integer.
So p = 7, q = 2 is the only solution to this.


on scroll "sticky-element" will need stick to before "footer-area" not on bottom of page

Question: I have one "sticky-element" div, on page load which I have been set position:fixed with bottom right aligned.
Requirement : on page scroll I would like to set it stick to just before on my "footer-area".
Issue : I have handled successfully css and js part on load, but I am not able to find logic that how can i add another class to my "sticky-element" once "footer-area" will start visible on window.
HTML

lots of code and other div nested in this as well

jquery
$(window).on("load",function(){
$(window).scroll(function() {
    var scroll = $(window).scrollTop();
    if(){
        $(".sticky-element").addClass("some-class");
    }

    else {
        $(".sticky-element").removeClass("some-class");
    }
});

});
css
.sticky-element { position:fixed; }

.sticky-element.some-class { position:static; }
In above one if() my logic is that I would like to use if "footer-area" is visible on window than only add class will works.
Please suggest if anyone has Simple and short(not too much complex) coding way for this.
Thanks in advance
Answer: Try something like that:

lots of code and other div nested in this as well

   $(window).on("load",function(){
    $(window).scroll(function() {

        var bottomOfWraper = $('#wraper').offset().top + $('#wraper').outerHeight();
        var bottomOfWin = $(window).scrollTop() + $(window).height();

        if( bottomOfWin > bottomOfWraper){
            $(".sticky-element").addClass("some-class");
        }

        else {
            $(".sticky-element").removeClass("some-class");
        }
    });
  });

Comment: Thanks @Tomas for such a beautiful and clear solution, Nice logic you have been used, I have try this and only issue I am find here is page will flicker once i reached to my footer area. I will try to fix it from my end to resolve it, rest logic is working fine.

In case you need to check css for this, I have been updated in my question css part as well.
Comment: Sorry, I do not understand what your problem is.


why is this expression used when using requestAnimationFrame

Question: why is this if is used when used requestAnimationFrame??

if (!window.requestAnimationFrame)
        window.requestAnimationFrame = function(callback, element) {
            var currTime = new Date().getTime();
            var timeToCall = Math.max(0, 16 - (currTime - lastTime));
            var id = window.setTimeout(function() { callback(currTime + timeToCall); },
              timeToCall);
            lastTime = currTime + timeToCall;
            return id;
        };
Comment: Are you asking why an existing piece of code written by someone else is written a certain way? Then please edit your question to add the context—where did you find this code, and what is the overall code used for? If this isn't what you're asking, then it's too unclear what you want to ask, so again you should edit your question to make it clear.
Answer: That piece of code implements requestAnimationFrame. It defines a global function requestAnimationFrame (all members of window are global variables) if such a function doesn't exist.
Here's a short explanation via comments of what it does:
if (!window.requestAnimationFrame) // if requestAnimationFrame doesn't exist
  window.requestAnimationFrame =
    function(callback, element) { // define requestAnimationFrame
      /* implementation detail */
    };

Such pieces of code are known as polyfills - code that implement features of newer versions of the browser/javascript engine in older browsers/javascript engines.
Comment: Note that since polyfills are written in javascript itself it may not be as fast as a natively implemented function which may be written in C. But the whole point of polyfills is to allow you to use new features in old browsers.


Binding ZURB Foundation dropdown plugin to dynamically-inserted element

Question: I have some client-side JavaScript that dynamically inserts some elements into the DOM. Those elements contain the markup for a dropdown element using the Dropdown plugin:

    Invalid

Invalid File

    This file is not a supported file type.

The problem is that the dropdown plugin isn't detecting this new element, so it is not adding the dropdown behaviors to the new elements.
As a test, I tried hard-coding the HTML directly in the HTML source, and the dropdown loads fine in that context. So I have Foundation and the dropdown plugin configured correctly. It's just not binding the functionality to the dynamically-generated elements.
Is there a bit of JavaScript that I can run to bind the dropdown functionality to #resource-upload-label-1234? I was looking at the source for the plugin, and it wasn't evident to me how to do this.
Answer: For efficiency you can use $(document).foundation('reflow');

To be even more efficient "target the actual type of foundation item
  you need to 'reflow' $(document).foundation('orbit', 'reflow');"

(I'm quoting from: http://foundation.zurb.com/forum/posts/1766-reflow)
Answer: One solution that I just found that works is to call $(document).foundation(); again after inserting the new elements.
I am not sure if this is the "right way," but it works. If anyone knows of a more appropriate solution, don't be shy, and post it as an answer!
According to a comment on zurb/foundation#3885, it appears that calling foundation() again should not cause any problems, and perhaps foundation() is designed to be used this way.


Smiley icons in the comments

Question: I installed the Smiley module for drupal 7.
Filtered enabled smileys for plan text format.
I want to enable the smile in the comments field where the body of the comments is a plain text.
Answer: You need to rearrange text filters - make smiley filter last. If you put it before "strip HTML tags", it will add tags but they'll be stripped.
Comment: Come on, don't paste descriptions of options here, it makes your comment hard to read and we can check them by name if we need. And please try to structure your posts in sentences. I can't see a word about filters order in your comment now.
Comment: I have selected in the properties of the field:

- Truncate html
Check this option if the May Contain html text input field and you want to truncate it safely. This Will Also overwrite the maxlength validation from core, I know That it will strip the tags before checking the length.

- Force truncate text
Check this option if you want that the html (or the text) That the user inserts into the field to be truncated.

but still does not work
Comment: ok sorry. however, I tried to truncate the properties of the field (long text) but nothing happens.
Comment: @illoto what properties? You mean content? And what, it's not truncated? But only when smileys are enabled? If it's independent problem, ask in independent question, please.

Smiley icons in the comments


Prevent Captive Portal auto-close after authentication (Android)

Question: So the simple idea is that we have a RADIUS server setup to allow users to authenticate with our Ruckus controller via user credentials. On authentication the user should be redirected to a page that allows them to manage MAC authenticated devices.
On everything we have tested so far, including "older" Android devices this seems to be no issue and things run as they should. However with Lollipop (5.0+) versions of Android the captive portal has changed quite a bit, and part of that change is to automatically close the captive portal that launches when you join the network. Because we want them to be redirected to a MAC device management page after authentication so they can add the device they currently logged in with and avoid having to log in again, this is bad.
What I have tried:

Detecting if the browser is being launched in a mobile device and popping an alert in onbeforeunload that attempts to keep the browser open.
Opening a new browser window, pointing to the redirection URL, when successful authentication is detected (essentially managing the redirect ourselves).
Performing Option 2, and then Option 1 on the redirected URL

What won't work:

Asking users to disable the captive portal option on their device. Not trying to point general users to advanced controls.
Creating an open network to access the MAC manager, it must be behind some authentication.

Solution For Now:
We are unhappy with this solution, but for now we are simply asking users to authenticate with the network and then open their browser and go to the basic login portal page (non-network authentication) that users use to manually add devices they can't connect with (like printers, gaming devices, etc.). Though this works, it is a pain for users to have to login, open a browser, manually enter a url, and login again.
This isn't a problem that people haven't run into, see here, I just haven't been able to find a solution from anyone that has run into the problem. Certainly there is some way of utilizing javascript or something to keep the browser open in this situation. If not, anyone have any better ideas for managing things?
Comment: Did you find a solution ? I'm looking for the same behaviour
Comment: @MtotheK no luck as of yet. I will update with an answer if I find one, if you could keep us SO folk posted if you find something on your end it would be appreciated. Frustrating for sure.
Comment: @zgc7009 any luck with this issue?
Comment: @CamiloAguilar unfortunately not. Google is aware of the functionality, know that people that need redirects aren't happy, and still see it as intended functionality. They provide very little control over the process in general it seems.
Comment: managed to solve this, by firewalling off the connectivity checks post authentication, and reverse proxing in the 204 when we need it to close.
check my answer in the thread.
Comment: I am using TPlink EAP 225 for captive portal with external portal authentication. I have the same problem when user login via captive browser it closes the browser after authentication. Did you find a solution?
Answer: We have managed to keep the UAM Browser / captive portal browser open on lollipop by adding firewall rules blocking :

clients3.google.com
clients1.google.com ,
android.clients.google.com
connectivitycheck.android.com
connectivitycheck.gstatic.com

Thus after the user is authenticated the UAM / Captive Browser stays open.
You can keep UAM open as long as you need, you can close it by invoking a reverse proxied 204 redirect to google's connectivity page.
Comment: Hi, could you elaborate on this? I'm trying to do this on OPNsense. If you block these domains the connection will be interpreted as with no internet right? And until you unlock them it will stay open. Did it get it right. I don't understand what you mean by a reverse proxied 204 redirect...
Comment: you need to block them on the "authed" side of the network, not the captivated part.
Comment: can we move this to chat?
Comment: how can we make the reverse proxied 204? Thanks!
Comment: Hi @JohannCombrink, How can we reverse proxied 204, I have a captive portal configured on tplink EAP device using an external portal page with radius server authentication. Can you please guide me on how I can implement reverse proxied 204? Thanks
Comment: window.location = "http://connectivitycheck.gstatic.com/generate_204" from my Captive page will close the current page? We found it worked in some cases and some end up getting 200 response instead of 204. Hence, our Captive page is not closing in such cases. Also, observed "connection: close" header in success cases but we are unsure how to force such header from Javascript mentioned above.
Comment: maybe checkout https://wiki.squid-cache.org/SquidFaq/ContentAdaptation
Comment: This worked well on Cisco Meraki. @DominikVogt comment was also needed.
Comment: @skidadon I am looking at how to do this with Cisco Meraki - care to expand on what worked for you?
Answer: This appears to be new Captive Portal behavior in Android devices since the release of Lollipop (5.0).
We have not yet discovered a workaround. If there is an explicit way to disable the auto-dismissal it is probably only documented in the Android codebase available here (I've been looking, but haven't found anything definitive yet):
https://android.googlesource.com/platform/frameworks/base
FYI, we've also noticed Android uses CloudFront CDN for its captive network detection. Our captive portal solution originally used CloudFront for assets, so we had to whitelist CloudFront subnets in pre-auth ACLs. Whitelisting CloudFront subsequently caused captive network detection to fail on recent Android devices. We had to abandon CloudFront CDN to restore captive portal functionality for Android devices.
Comment: So we have a matrix of solutions that come with various trade-offs and responsibilities. I can edit the above answer to better describe the scenarios. Recapture options: 1) delay ACL-upgrade controller redirects until they click some bottom-of-page "browse Internet" button; 2) whitelist Android captive network detection URLs for Android-only fallback to DNS-redirect-like captive methods; 3) use something like Cisco's captive-bypass for DNS-redirect-like methods on all devices. Whitelist options: 1) whitelist external ad resources, blacklist all else; 2) blacklist top sites, whitelist all else.
Comment: If you wouldn't mind outlining those in the answer as a guide for others, it would be appreciated. While it doesn't solve our problem exactly, it is certainly better than the *shrug* answer I have been able to give people to this point.
Comment: Thanks for the detailed response. This is certainly a headache, we keep coming back to it hoping we, Google, or someone else has come up with a solution for the problem and it just doesn't seem like anything is even moving in that direction. It isn't _that_ uncommon for redirects to be an important part of hotspotting, it is weird to me that this "issue" has been allowed to linger without at least an explanation as to why.
Comment: We've found a solution where, for the time being, we're changing when we redirect to the controller to authenticate the client to move them from the pre-auth ACL to the full access ACL until they explicitly click a "Browse the Internet" button. This solution requires our clients to whitelist all external resources they use in their welcome page banner ads and landing URLs.
Comment: We've identified and tested a second solution where we could whitelist Android's connectivitycheck.gstatic.com IP (w/ hostname-to-IP resolution monitoring to update whitelists on IP assignment changes). This causes Android devices to fallback to DNS-redirect-like captive methods. This solution also requires our clients to whitelist all external resources they use in their welcome page banner ads and landing URLs.
Comment: We have a client in Syndey that is working with Cisco to identify a better way to recapture the Android audience, perhaps by enchancing Cisco's captive-bypass feature. The second solution above is similar to an Android-only captive-bypass. We're provisionally opting for the first solution because it gives basically the same experience for Android and iOS (where Android captive browser disappears after clicking "Browse the Internet", while iOS captive browser does not).
Comment: I managed to get a working solution, firewall the connectivity checks, and reverse proxy to the 204 check to close it, see my answer in the thread
Answer: Why don't you just hold captive portal opened after authentication? You can always allow access to every site except captive checking sites.
Tested and working both Android and iOS in all versions.
If you need to access cookies/shared storage from default browser (not captive ios/android sandboxed browser), you gotta hop out it before authentication.
Comment: My question is _how_ do you hold it open. The OS automatically closes the captive portal after authentication.
Comment: cite: "You can always allow access to every site except captive checking sites."
Comment: That would essentially be an open network. We need the captive portal to trigger any time the user connects to the network. We aren't whitelisting. Thank you for your input though.
Comment: I mean after authentication, of course. Sorry for misunderstanding. Our flow: user connects to the wifi, captive portal is shown, user chooses authentication method, internet access is allowed, user follows ad/link/ in captive, user closes captive whenever he/she wants.
Comment: The problem we are having is that as soon as the user is authenticated the captive portal closes. There is no way to continue on within the captive portal once there is authentication, it won't stay open. If you are able to keep it open please let us know how. Whitelisting won't keep the portal open, it will simply allow the user to browse without authentication, at least to the best of my knowledge.
Comment: I think that @lukyer means to disable captive "internet detection" method but that would keep it open in annoying way, though is a good alternative. Furthermore, what I've done in the past is to show an special web on captive asking the user to navigate through other browser like Chrome, or Safari saying that "This page needs special features".
Comment: You can keep UAM open as long as you need by firewalling off the connectivity check urls, you can close it by invoking a reverse proxied 204 redirect to google's page, see my answer in the thread.


413 - request entity too large in lighthttpd

Question: When saving configuration settings in Magento I get "413 - request entity too large". We are running lighttpd.
In php.ini memory_limit = -1
In lighttpd.conf server.max-request-size = 1000000
I have restarted lighttpd and the problem is still there.
Answer: What is post_max_size in php.ini? This controls how much data php will accept. I suspect it is set too low for your situation.
The default is 8M. Try post_max_size = 25M and see if that solves it.
Comment: Sorry, didn't help.
Comment: Sorry, that didn't help. Thing is that this is text being posted. So it shouldn't be that heavy.
Answer: There was a regression in 1.4.36, which should be fixed in 1.4.37: http://redmine.lighttpd.net/issues/2660 (another regression related to not existing upload directory was fixed in https://redmine.lighttpd.net/projects/lighttpd/repository/revisions/3046, not released yet)


Are these variables ordinal?

Question: I am analysing some survey data, but before running some tests I'd like to be sure about the level of measurement of some variables:
income group:
Below £10,000, £10,000-19,999, £20,000-39,999, £40,000 or above
Age group:
Below 18 years old, 18-25 years old, 26-35 years old etc.
Number of travels by train:
Less than 2 times, 2-4 times, 5-6 times, More than 6 times
I would treat them as ordinal, but I've found discording info on the web.
Thanks a lot!
Answer: The first two are definitely ordinal: the next category differs from the previous one on an underlying quantitative trait. The last one is a frequency of some kind, so while some nitpickers would see otherwise, I would treat it as ordinal for most analyses, too.
Answer: Stevens' typology of level of measurement is not really adequate for describing the whole range of variable types we deal with in practice (and analysis recommendations based only on it are certainly less than adequate in many situations).
Just calling those ordinal throws out the specific information in the bin-labels (bin-boundaries).
Strictly speaking you're dealing with interval-censored data, and in some cases it may be valuable to make use of that, rather than just shoehorn this into the more general but lower-information "data are ordinal" category. For example, by adding a distributional model to the interval-censored data you'd be able to estimate moments (means, and variances, for example), which you can't do if you ignore the information in the bin-boundaries.
For example, a common model for income is lognormal. If you add that assumption, you can estimate average income (via maximum likelihood, for example), or use it in various other forms of analysis that may be difficult or hard to justify otherwise.
For some kinds of things, treating them just as ordinal may be fine. On the other hand, don't toss out any information you could possibly use later.
(Does this make me a nitpicker? Perhaps it does -- but often that's an essential characteristic for a statistician so I'm not too bothered by it.)
Comment: I will mostly do some association analysis, like Somers' D, so I think I can measure all of them as ordinal in this case, right?
Comment: There's nothing wrong in calling them ordinal (though pay attention to Stask's comment in relation to the counts), aside from the fact that it potentially wastes information.

Are these variables ordinal?


Complex regression plot in R

Question: I need to draw a complex graphics for visual data analysis.
I have 2 variables and a big number of cases (>1000). For example (number is 100 if to make dispersion less "normal"):
x <- rnorm(100,mean=95,sd=50)
y <- rnorm(100,mean=35,sd=20)
d <- data.frame(x=x,y=y)

1) I need to plot raw data with point size, corresponding the relative frequency of coincidences, so plot(x,y) is not an option - I need point sizes. What should be done to achieve this?
2) On the same plot I need to plot 95% confidence interval ellipse and line representing change of correlation (do not know how to name it correctly) - something like this:
library(corrgram)
corrgram(d, order=TRUE, lower.panel=panel.ellipse, upper.panel=panel.pts)

but with both graphs at one plot.
3) Finally, I need to draw a resulting linar regression model on top of this all:
r<-lm(y~x, data=d)
abline(r,col=2,lwd=2)

but with error range... something like on QQ-plot:

but for fitting errors, if it is possible.
So the question is:
How to achieve all of this at one graph?
Answer: Does the picture below look like what you want to achieve?

Here's the updated R code, following your comments:
do.it <- function(df, type="confidence", ...) {
  require(ellipse)
  lm0 <- lm(y ~ x, data=df)
  xc <- with(df, xyTable(x, y))
  df.new <- data.frame(x=seq(min(df$x), max(df$x), 0.1))
  pred.ulb <- predict(lm0, df.new, interval=type)
  pred.lo <- predict(loess(y ~ x, data=df), df.new)
  plot(xc$x, xc$y, cex=xc$number*2/3, xlab="x", ylab="y", ...)
  abline(lm0, col="red")
  lines(df.new$x, pred.lo, col="green", lwd=1.5)
  lines(df.new$x, pred.ulb[,"lwr"], lty=2, col="red")
  lines(df.new$x, pred.ulb[,"upr"], lty=2, col="red")
  lines(ellipse(cor(df$x, df$y), scale=c(sd(df$x),sd(df$y)),
        centre=c(mean(df$x),mean(df$y))), lwd=1.5, col="green")
  invisible(lm0)
}

set.seed(101)
n <- 1000
x <- rnorm(n, mean=2)
y <- 1.5 + 0.4*x + rnorm(n)
df <- data.frame(x=x, y=y)

# take a bootstrap sample
df <- df[sample(nrow(df), nrow(df), rep=TRUE),]

do.it(df, pch=19, col=rgb(0,0,.7,.5))

And here is the ggplotized version

produced with the following piece of code:
xc <- with(df, xyTable(x, y))
df2 <- cbind.data.frame(x=xc$x, y=xc$y, n=xc$number)
df.ell <- as.data.frame(with(df, ellipse(cor(x, y),
                                         scale=c(sd(x),sd(y)),
                                         centre=c(mean(x),mean(y)))))
library(ggplot2)

ggplot(data=df2, aes(x=x, y=y)) +
  geom_point(aes(size=n), alpha=.6) +
  stat_smooth(data=df, method="loess", se=FALSE, color="green") +
  stat_smooth(data=df, method="lm") +
  geom_path(data=df.ell, colour="green", size=1.2)

It could be customized a little bit more by adding model fit indices, like Cook's distance, with a color shading effect.
Comment: @Tal The interpretation of the ellipse is the same as in the `corrgram` package: it shows 95% pairwise confidence region assuming a bivariate normal distribution centered on the mean and scaled by SD(x) and SD(y). I'm not a big fan of this when used in a scatterplot, though. But see Murdoch & Chow, [A graphical display of large correlation matrices](http://www.jstor.org/pss/2684435), Am Stat (1996) 50:178, or Friendly, [Corrgrams: Exploratory displays for correlation matrices](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.19.1268&rep=rep1&type=pdf), Am Stat (2002) 56:316.
Comment: @chl +1, nice graph, and short code.
Comment: @mpiktas Thanks. This led me to realize I didn't work with the right sample, in fact :-)
Comment: Almost looks like the one I need, but with real numbers I faced the following problems:
1) `df.new <- data.frame(x = seq(min(x), max(x), 0.1))` is better.
2) Ellipse is drawn at 0;0 position, which is not correct and it`s size is also strange (too small). Also tryed `library(car) dataEllipse(df$x, df$y, levels=0.95:1, lty=2)` but it drops all.
3) The curve (like on correlogramm) is missing. I almost reproduced it by calling `library(car) cr.plots(m0)` but the range of data is incorrect.

Use first 2 lines from my code instead of yours to reproduce.
Comment: @Yuriy Ok, I will update my code (no need to make any edit in the meantime), but I cannot see how we could get overlap with real-valued random variates with your $(x,y)$ setting; this is the reason why I use boostrap with replacement (this ensures that ~ 2/3 of the original units are present). `car::dataEllipse` does provide the same facilities than in the `ellipse` package, but it is probably less easy to customize. I guess the superimposed curve is just a *loess*, so it is not difficult to add.
Comment: @Yuriy Done. There were two errors in my previous code: (1) as there are replicates in the data.frame, length of `x` and `table(x)` will differ, hence the `cex` param was recycled; (2) I forgot to update the location and scale of the ellipse (not clearly apparent when using standard gaussian variates). I hope there are no other ones.
Comment: Got `Error in ellipse(cor(df$x, df$y), scale = c(sd(df$x), sd(df$y)), centre = c(mean(df$x),  :
  center must be a vector of length 2` and no ellipse :( Everything else is excellent. Thank you very much.
Comment: @Yuriy It works for me; maybe there's a conflict with another function having the same name (in the car package)? Try to source all the code in a new R session. The picture was generated this way.
Comment: Hi chl, great piece of code.  Might I as what does the ellipse actually mean? (p.s: I think I'll repost your answer on my blog, it is a cool piece of code...)
Comment: @chl ggplot version is even nicer. It will be better to change colors and line styles: dashed red for ellipse and bolder (dark?)green for loess line... just a suggestion. Nice work. – Yuriy Petr
Comment: chl, thank you for the explanation (and the links to the papers!), just to be sure - the confidence regions are for the **expectancy** of the bivariate distribution?
Comment: @Tal Yes, this is always in reference to a theoretical, not empirical, distribution.
Answer: For point 1 just use the cex parameter on plot to set the point size.
For instance
x = rnorm(100)
plot(x, pch=20, cex=abs(x))

To have multiple graphs in one plot use par(mfrow=c(numrows, numcols)) to have an evenly spaced layout or layout to make more complex ones.
Comment: +1 for the tip about `cex`, but I think the OP wants all stuff on the same plotting region, not on separate ones.
Comment: Ahh... now I understand the question. Well, then he can just use `curve` or `points` to overplot the three graphs ;)

Complex regression plot in R


How to solve this complex limit $ \lim_{N \to \infty} \arg \left( \frac{1}{Nj+1}\right) $

Question: How to solve this limit
$$
 \lim_{N \to \infty} \arg \left( \frac{1}{Nj+1}\right)
$$
The prove that I have seen is using that the $\arg$ of a quotient is the diference beetween the num and den i,e:
$$
\arg \left( \frac{1}{Nj+1}\right)=\arg \left( 1\right)-\arg \left( Nj+1\right)= 0-\arg \left( Nj+1\right)=-\arg \left( Nj+1\right)
$$
Then how $A \to \infty$ then I can ignore the $1$(This do not convice me very much) and I get
$$
= -\arg \left( Nj\right)
$$
So $\lim_{N \to \infty}-\arg \left( Nj\right)=\lim_{N \to \infty}-\arg \left( j\right)= -90^{deg}$
But in this last line I can really forget de N?, and why?
Comment: You should really be thinking geometrically, with a picture drawn in your imagination. The points you're looking at are $(1,N)$ in the plane, and the line joining them to the origin is getting more and more nearly vertical. So the answer (before the minus sign) is $90^\circ$ and $-90^\circ$ when you take account of the sign. Of course the "$N$" vanishes after you've performed the limiting process.
Answer: Well, assuming that $\text{j}^2=-1$:
$$0^\circ=0\le\arg\left(\frac{1}{1+\text{N}\text{j}}\right)=\arg\left(1\right)-\arg\left(1+\text{N}\text{j}\right)=0-\arctan\left(\frac{\text{N}}{1}\right)\le2\pi=360^\circ$$
And:
$$\lim_{\text{N}\to\infty}\arctan\left(\text{N}\right)=\frac{\pi}{2}=90^\circ$$
Answer: We could first simplify :
$$\frac1{Ni+1}=\frac{1-Ni}{1+N^2}\implies \arg\frac1{1+Ni}=\arctan(-N)\xrightarrow[n\to\infty]{}-\frac\pi2$$
In degrees, the above would be $\;-90^\circ\;$
Comment: I do not understand How you get the second equality. @DonAntonio
Comment: @JuanMuñoz It is the definition of argument when the real part of the complex number is positive...


pyspark - read json files

Question: I am trying to read this json files.
{
    "data": [{
            "id": "c1",
            "type": "corporate",
            "tenor": "10.3 years",
            "yield": "5.30%",
            "amount_outstanding": 1200000
        },
        {
            "id": "g1",
            "type": "government",
            "tenor": "9.4 years",
            "yield": "3.70%",
            "amount_outstanding": 2500000
        },
]}

Code
    df = spark.read.option("multiline", True).json("sample_input.json")
    df.select(col("data")).show()
However this reads everything into single column. Is there a way I can apply schema using id, type, tenor and other columns?
Comment: after loading the data with your existing code, just run `df = df.selectExpr('inline(data)')`
Comment: @jxc Thanks. it solved the issue
Answer: If you try to load multi-line json with permissive mode then you can able see dataframe correctly,
df = spark.read.option("multiline", "true").option("mode", "PERMISSIVE").json("sample_input.json")
df.printSchema()

root
 |-- data: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- amount_outstanding: long (nullable = true)
 |    |    |-- id: string (nullable = true)
 |    |    |-- tenor: string (nullable = true)
 |    |    |-- type: string (nullable = true)
 |    |    |-- yield: string (nullable = true)

If you remove the data element from json and try to load then it will load correctly like this.
Json content:
[{
            "id": "c1",
            "type": "corporate",
            "tenor": "10.3 years",
            "yield": "5.30%",
            "amount_outstanding": 1200000
        },
        {
            "id": "g1",
            "type": "government",
            "tenor": "9.4 years",
            "yield": "3.70%",
            "amount_outstanding": 2500000
        }]

Ouput Schema:
df = spark.read.option("multiline", "true").option("mode", "PERMISSIVE").json("sample_input.json")
df.printSchema()

df.printSchema()
root
 |-- amount_outstanding: long (nullable = true)
 |-- id: string (nullable = true)
 |-- tenor: string (nullable = true)
 |-- type: string (nullable = true)
 |-- yield: string (nullable = true)

In both option you have correct data frame once it is correctly populated extract and transform data the way you want.

pyspark - read json files


Importance of removing correlated predictors in non linear regression and classification

Question: Removing linear combinations and correlated predictors from a dataset is something I do when I have to fit a linear regression. I wonder though how this technique affects in general the outcome of e.g. SVM or Random Forest.
Comment: (1) I have no idea (2) you don't need to remove correlated predictors in linear regression (the coefficients for the correlated variables will be poorly defined but not always an issue) (3) I assumed the correlated variables would be retained in a random forest with relative importance spread between correlated variables, while in a linear model variable selection will often get rid of some of the correlated variables.
Comment: You don't need to remove correlated predictors. In fact, they're almost always correlated.
Answer: In principal it shouldn't affect the performance of an SVM very much, because the SVM uses regularisation to avoid over-fitting (c.f. ridge regression), which is also effective in making the optimisation problem well posed once more.  However, the problem is then to find a good value for the regularisation parameter, $C$.
Also the SVM is based on an approximation of a bound on generalisation performance that is independent of the dimensionality of the feature space, so at least for linear SVMs, it would be reasonable for it to have little effect.  For non-linear SVMs, it will depend on the sensitivity of the kernel to the correlation.  For the common RBF kernel, it will mean that the induced feature space will be a little more sensitive to a particular direction in the input space, but probably not enough for it to be problematical.
In my experience, it is all too easy to make the performance of kernel methods worse by manually tinkering with the input representation in order to improve performance.  Generally this will just end up over-fitting the test data, as the operator has become part of the model selection procedure in a way that is not properly taken into account in performance estimation.  A good example of this is provided by machine learning challenges (e.g. kaggle) that have a leader board.  Quite often those that work hard to move up the leaderboard find themselves near the bottom of the final rankings (which are determined on a separate sample of data).  The reason is often "over-tuning".
Answer: In linear regression model,
$\mathbf{y} = \mathbf{X}\beta$
the unique solution for $\beta$ is,
$\hat{\beta} = (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$
where we need to calculate the inverse of the obtained matrix: $\mathbf{X}^T\mathbf{X}$. Note that the resulted for a solution to exist, the matrix must be non-singular or in other words, the basis vectors must be linearly independent. Therefore, any redundant basis model must be eliminated but sometimes the data behaves in a way that the $\mathbf{X}^T\mathbf{X}$ becomes very close to being singular. In this situation, one solution is to use singular value decomposition (SVD).
Comment: Removing the co-linear predictors is necessary, not an option, because then as I said your model suffers from singularity. However asking whether adding new predictors improve the fit is rather a complicated matter which must be addressed in the light of model comparison/selection. You may see that having more predictors improves the fit, but it's not guaranteed that the observed improvement is brought about so called over-fitting. Having more predictors is equal to additional Occam's penalty (Occam's razor). As a rule of thumb, a good model is the one with the fewest parameters.
Comment: Right, but I am wondering how removing collinear and correlated predictors affect the SVM and Random Forest fit. Does it improve the fit in general?
Comment: I agree with everything you say, my question is related to SVM and Random Forest though. Does removing a collinear predictor improve a SVM/RF fit in general? Should collinear predictors be removed before applying a SVM/RF fit? Are there some rule of thumb? Thanks!
Comment: There's no way I can tell yes or no. Just investigate it yourself on your data using some standard techniques like try k-fold cross-validation or bootstrapping


How to DockerFile commands in bash instead of sh, does bash alias work for sh shell?

Question: I started with dockerizing my python hello world program, and the dockerfile looks something like this.
FROM ubuntu:16.04
MAINTAINER Bhavani Ravi

RUN apt-get update
RUN apt-get install -y software-properties-common vim
RUN add-apt-repository ppa:jonathonf/python-3.6
RUN apt-get update

RUN apt-get install -y build-essential python3.6 python3.6-dev python3-pip python3.6-venv
RUN apt-get install -y git

# update pip
RUN python3.6 -m pip install pip --upgrade
RUN python3.6 -m pip install wheel

RUN echo 'alias python=python3.6' >> ~/.bash_aliases
COPY hello.py .
ENTRYPOINT python hello.py

The problem is now when I run the image I get /bin/sh: 1: python: not found.
As you can see in the dockerfile, I have set up bash_alias. When I override entrypoint and run the image with /bin/bash and then use python command it works.

How to make the alaias work for all command environments
Can you run dockerfile commands in bash instead of shell
Comment: this has to do with base image you are extending, if it supports both bash and sh then you can run both
Comment: Aliases don't even work for *bash* in a noninteractive context unless explicitly enabled; they're a user portability extension only.
Comment: Is there a reason you aren't setting `ENTRYPOINT python3.6 hello.py`?
Comment: Not really. I am used to setting up aliases and using them. So tried it the same way and ended up there.
Answer: Maybe try wrapping each command in /bin/bash -c "yourcommand" if bash is available on your Docker image it will force it.
Example
RUN /bin/bash -c "apt-get update"
...
ENTRYPOINT /bin/bash -c "python hello.py"


How can a site programmatically detect if cron has not run in a long time?

Question: I'm developing a module intended to proactively warn the admin if there are problems with the site. The idea is to periodically check the system status report and sends an email notice if certain statuses have gone from REQUIREMENT_OK to REQUIREMENT_WARNING or REQUIREMENT_ERROR. I'm planning on firing this check in hook_cron().
But what if the problem with the site is that cron is broken? What is a safe and performant way for the site to periodically check if the 'cron_last' variable is too old, that doesn't rely on cron?
I would like to keep the module self-contained and avoid external dependencies or "phone home" solutions.
Comment: Interesting question...but given that PHP doesn't have the ability to periodically execute a script, it's hard to imagine how you could do this in a self-contained module. You'll need _something_ to execute that script periodically, whether it's cron, ping service or something different. You could probably even find a PHP extension which adds that ability, but that would obviously count as a dependency
Comment: What @Clive said. This is PHP, you need *something* that triggers your script. There really is no other way. Btw, our https://www.drupal.org/project/monitoring project does this, among many other things. We explicitly rely on something external to call us so we're not depending on the core cron, we just provide services and API's to run our checks.
Comment: Actually, Drupal core already does this check on [`system_admin_config_page()`](https://api.drupal.org/api/drupal/modules!system!system.admin.inc/function/system_admin_config_page/7). Find a hook that suites your need, and add some checking to avoid calling `system_check(TRUE)` too often.
Comment: system_status(TRUE) just tells me if there is a status issue, but not the details. The output I'll generate is closer to system_status(FALSE), but filtered down, and formatted for email. I think hook_cron() does suit my need, with the important exception of checking whether cron itself has been running. Would a hook_init() call be a terrible thing if all it did was check 'cron_last'?
Comment: @Berdir https://www.drupal.org/project/monitoring is impressive. Munin and Nagios are off the table for the project I'm working on, but I might try building a simple email alert system on top of Monitoring instead.
Answer:
What is a safe and performant way for the site to periodically check
  if the 'cron_last' variable is too old, that doesn't rely on cron?

As said before by @Clive, you a script running periodically to do this check for you. Having said so, the simplest way is to create a basic module that checks the last time Drupal Cron ran succesfuly and acts based on that. You could include a drush command so you can call it via a server cronjob. E.g.

Check if cron hasn't run in X minutes and send e-mail if so, using  Drush
1) Create a file named cronwatch.drush.inc
2) Create the drush command that you will invoke via a crontab / automated task in your server. For instance, you would be running drush cronwatch every X minutes.
/**
 * Implements hook_drush_command().
 */
function cronwatch_drush_command() {
  $items = array();
  $items['cronwatch'] = array(
    'description' => dt('Check last time Drupal cron ran and send an email if necessary.'),
  );
  return $items;
}

3) Add a function to check if Drupal Cron hasn't run in longer than X minutes, send an alert email.
/**
 * Callback function for drush cronwatch command.
 */
function drush_cronwatch() {
  // Minutes since the last time cron run.
  $last_cron = variable_get('cron_last', 0);
  $minutes_ago = (time() - $last_cron) / 60;

  // How many minutes to allow without having run the cron.
  // This number should be larger than the frecuency of Drupal cron. E.g. If
  // Drupal cron is set to run every 60m, this number could be 70.
  $tolerance = 70;

  // If the cron last ran longer than tolerance minutes ago, it means that it
  // probably failed.
  if ($minutes_ago >= $tolerance) {
    $to = 'firstname.lastname@example.com';  // E-mail to get the alerts
    $message = t('Drupal cron could not run successfuly. Last succesful run was !minutes minutes ago.', array('!minutes' => floor($minutes_ago)));
    drush_mail_send_email($to, 'Drupal Cron failed', $message);
  }
}

4) Finally, this is the function to send an e-mail via drush:
/**
 * Send an e-mail to a specified e-mail address from within Drush.
 * @see http://drupal.org/files/mail.drush_.inc__0.txt
 */
function drush_mail_send_email($to, $subject = '', $body = '') {
  // Define $from and headers
  if (!$from = drush_get_option('from')) {
    // E-mail to send the alerts from. Replace here if desired.
    $from = ini_get('sendmail_from');
  }
  $headers = array();
  $headers['From'] = $headers['Sender'] = $headers['Return-Path'] = $headers['Errors-To'] = $from;

  // D7 implementation of drupal_mail
  if (function_exists('drupal_mail_system')) {
    // Prepare the message.
    $message = drupal_mail('drush', 'key', $to, NULL, array(), $from, FALSE);

    $message['subject'] = $subject;
    $message['body'] = array();
    $message['body'][] = $body;
    $message['headers'] = $headers;

    // Retrieve the responsible implementation for this message.
    $system = drupal_mail_system('drush', 'key');
    // Format the message body.
    $message = $system->format($message);
    // Send e-mail.
    $message['result'] = $system->mail($message);
    $result = $message['result'];

    // D6 implementation of drupal_mail_send
  }
  else {
    $message = array(
      'to' => $to,
      'subject' => $subject,
      'body' => $body,
      'headers' => $headers,
    );
    $result = drupal_mail_send($message);
  }

  // Return result.
  if ($result) {
    drush_log(dt('E-mail message sent to !to', array('!to' => $to)), 'ok');
  }
  else {
    drush_set_error('DRUSH_MAIL_ERROR', dt('An error occurred while sending the e-mail message.'));
    watchdog('cronwatch', 'An error occurred while sending the e-mail message with Drush.', array(), WATCHDOG_ERROR);
  }
}

And there you go. If Drupal Cron is set to run -let's say- every 60 min, and for some reason failed, the next time this script is executed, it will send you an alert e-mail and you can act based on that.
---- EDIT: Added code to check without depending on Drush / Server Cronjobs ----
Check if cron hasn't run in X minutes and send e-mail if so (no Drush / Server cronjobs)
You could also run the checks without Drush nor server cronjobs. For this, you will necessarily need to hook your checks to a certain event in your website (e.g. every page load).
In your cronwatch.module file:
1) In this case, we check on every page load, that is almost guaranteed to run periodically. However, you can use any other hook to trigger your checks.
Let's find out when the Drupal cron check was last executed.
/**
 * Implements hook_page_alter().
 *
 * Finds out when was the last time Drupal cron check was executed. If it's
 * been longer than the desired frequency, proceeds with a new check.
 *
 * We depend on page loads in order to emulate a 'periodically running' event,
 * as an alternative to a server cronjob / scheduled task.
 */
function cronwatch_page_alter() {
  // Defines how often we want to run the checks (in minutes).
  // We need this flag variable, since we don't want to make a check on
  // every single page load.
  $frequency = 60;

  // Only proceed if it's been longer than 60 minutes since the last check.
  $last_check = variable_get('cronwatch_last_check_timestamp', 0);
  $minutes_ago = (REQUEST_TIME - $last_check) / 60;

  if ($minutes_ago < $frequency) {
    return;
  }

  // More than 60 min have passed since the last check. So, let's proceed with
  // a new check.
  if (cronwatch_check()) {
    // Check was succesful. Now update the 'last check timestamp' variable so
    // the next check runs an hour from now.
    variable_set('cronwatch_last_check_timestamp', REQUEST_TIME);
  }
}

2) Make the actual check for the last time Drupal cron ran. If more than X minutes have passed, send an email.
/**
 * Checks the last time Drupal cron ran, and acts based on that.
 */
function cronwatch_check() {
  // Minutes since the last time cron run.
  $last_cron = variable_get('cron_last', 0);
  $minutes_ago = (time() - $last_cron) / 60;

  // How many minutes to allow without having run the cron.
  // This number should be larger than the frecuency of Drupal cron. E.g. If
  // Drupal cron is set to run every 60m, this number could be 70.
  $tolerance = 70;

  // If the cron last ran longer than tolerance minutes ago, it means that it
  // probably failed.
  if ($minutes_ago >= $tolerance) {
    $to = 'email@example.com';  // E-mail to get the alerts
    $message = t('Drupal cron could not run successfuly. Last succesful run was !minutes minutes ago.', array('!minutes' => floor($minutes_ago)));
    $mail_sent = cronwatch_mail_send_email($to, t('Drupal Cron failed'), $message);

    if ($mail_sent) {
      return TRUE;
    }
  }
  return FALSE;
}

3) Function to send an e-mail
/**
 * Sends an e-mail from the cronwatch module.
 */
function cronwatch_mail_send_email($to, $subject, $body) {
  // Let's get the mail ready to be sent.
  $module = 'cronwatch';
  $key = 'alert_mail';
  $from = variable_get('site_mail', ini_get('sendmail_from'));
  $language = language_default();

  // Build the mail, but don't send it yet.
  $message = drupal_mail($module, $key, $to, $language, array(), $from, FALSE);

  // Prepare the message.
  $message['subject'] = $subject;
  $message['body'][] = $body;

  // Retrieve the responsible implementation for this message.
  $system = drupal_mail_system($module, $key);

  // Format the message body.
  $message = $system->format($message);

  // Send e-mail (Drupal will log any error).
  $message['result'] = $system->mail($message);

  return $message['result'];
}

Only drawback is you depend on page loads, so accuracy would be affected by a website with very few or no visits (in that case, do you even need this?). And, of course, this code could be refactored, but this should do the trick.
Comment: This approach would certainly work, and extra credit for the exceptionally well-commented code. This does sidestep the need for external dependencies (not counting Drush), but for the moment I'm holding out for a solution that doesn't require additional setup or server access beyond enabling the module. When I hand the project off, I want the number of potential maintenance points to be at a minimum.
Comment: I see. I have updated my answer to include an option that works without Drush nor any server setup. Your site only needs to have visits ;)
Comment: Nice! I was considering something similar using hook_init(). I think depending on traffic I'll also want to acquire a lock on cronwatch_check(), but I can take it from here.
Comment: Awesome! Certainly, you could use hook_init() too, which is run only in non-cached pages, though.
Answer: This is a pretty heavy handed solution, but you could implement a hook in a module that runs this check for you. Try to find a page that isn't hit very frequently and run it whenever that page is loaded.
Another possibility would be to install the Services module then hit a url from a remote script that kicks off your module to do the check. This introduces a pretty big dependency as well (another server), but it might not be bad if you do have another server. You could have servers checking on each other across a number of projects (maybe I'm overthinking this...)
Comment: I want the module to be general-use, so I don't want to make judgments about which page is not hit frequently. Interesting thought, though.  And Services is definitely too heavy for what I have in mind.
Comment: If it needs to be more general purpose, it could run anytime someone hits a node edit form, or the content admin page. You could expose it to rules and let the user select an event that triggers your module. If it isn't based on something intrinsic to the site itself, you are going to be faced with some kind of external dependency.


Is there any way to generate url alias instead of this taxonomy/term/%/faq, in views

Question: I have created a page view taxonomy/term/%/faq. It works fine but I need to replace taxonomy/term/% with its url alias.
Comment: Try using [pathauto](https://www.drupal.org/project/pathauto) which will help you create aliases for taxonomy pages.
Answer: Pathauto just allows you to generate the URLs automatically so I believe you would still have the issue.
You will need something like https://www.drupal.org/project/path_alias_xt to extend the alias functionality to the other subpaths like the node/%/edit, taxonomy/term/%/faq etc.
Comment: https://www.drupal.org/project/subpathauto is another option.
Answer: I did it that way : using page_manager (with ui) & panels, creating a panel page with an URL /taxonomy/term/% and disabling (not deleting) default taxonomy_term  view (with its URL /taxonomy/term/% )
Then i copied default taxonomy_term view to a new one using different URL (different than /taxonomy/term/%), i added a block (and disabled 'page' displayID) to this view acting exactly like default taxonomy_term view page.
Note : this block should have a contextual filter that correspond to taxonomy term ID (with depth or not)
Then i added this created block in my panel.
That allows to do what you are needing (get alias for URLs), and moreover, you can add things (blocks) to this page and it is a lot more flexible than overriding default taxonomy_term view. I did not this for that puprose, but you can get your goal this way.


How to interpret results of interaction regression in R

Question: Suppose I have the regression in R
lm(formula = income ~ ageQuartile * (numYearsWorking + numHoursPerWeekWorking))

and in R, I get results like:
                                        Estimate
(Intercept)                             12.94520
ageqtile2                                6.63428
ageqtile3                               12.64004
numYearsWorking                          4.25382
numHoursPerWeekWorking                  17.98021
ageqtile2:numYearsWorking                9.98316
ageqtile3:numYearsWorking               12.81078
ageqtile2:numHoursPerWeekWorking        15.35733
ageqtile3:numHoursPerWeekWorking        20.34312

(these numbers are made up and probably dont make sense).
How do I interpret this? Say I wanted to predict the income of someone in the first age quartile; is that just
$12.94520+4.2538*numYearsWorking+17.98021*numHoursPerWeekWorking $
And how does this change as I want to examine different quartiles?
Answer: Yes, that's what you need to do. And for an observation from the second age quartile :
$$12.94520+4.2538∗numYearsWorking+17.98021∗numHoursPerWeekWorking+9.98316*numYearsWorking+15.35733*numHoursPerWeekWorking$$
For interpretation, for example if you want the marginal effect of numHoursPerWeekWorking given age quartile==2, you just have to derivate and it gives : $ 17.98021 + 15.35733  $
Comment: But then where is the information "ageqtile2= 6.63428 " used?
Comment: Yes, I'm sorry, I missed it for the first expression ! It is : 12.94520+ 6.63428 + 4.2538∗numYearsWorking+17.98021∗numHoursPerWeekWorking+9.98316∗numYearsWorking+15.35733∗numHoursPerWeekWorking
Answer: Assuming the results are the default dummy contrasts R uses, then the intercept is the effect of the reference (first) level of ageqtile. Subsequent levels are coded as the additional offset in DV moving from the reference level to that level. So the "ageqtile2= 6.63428" is used in the prediction by adding 6.6 to the prediction for rows where (ageqtile==2).
You can get these coefficients with coef(). You can get the predicted y values with predict(model) and use these to check your work.
It helps to have a running example:
df = mtcars
m1 =lm(mpg ~ factor(am) * disp, data = mtcars)
coef(m1)
# (Intercept)      factor(am)1             disp factor(am)1:disp
# 25.15706407       7.70907298      -0.02758360      -0.03145482

We can generate our own pred column, to check understanding of the coefficients:
b = coef(m1)

df = within(df, {

    pred = (b["(Intercept)"]

     + ((am=="1") * b["factor(am)1"])

     + (disp      * b["disp"])

     + ((am=="1") * disp * b["factor(am)1:disp"]))

})

any(predict(m1)!=df$pred)
# [1] FALSE

This page is helpful for other kinds of contrasts.


How to get the amount of memory my linux 32 machine sees . (Not all the physical memory)

Question: How to get the amount of RAM memory my linux 32 bit sees ?
If I use free / top it reports the whole physical memory which exceeds the 32 bit limitations .
Comment: What makes you think that "free" is lying?  It's completely legitimate for a 32-bit kernel to have more than 4GB of memory available to it (see, eg, http://en.wikipedia.org/wiki/Physical_Address_Extension ).  Each individual process will probably be limited in the amount of memory it can use, but the system as a whole will have access to the full amount for parcelling out to the individual processes.
Comment: its worth noting that 32Bit limits you to 4GB not 2GB for ram, PAE allows you to address (and therefore see) more ram but a single process still cant use more than 4gb of ram
Comment: Anthony: thanks, and I have amended the "4GB" figure above in the light of your comment.
Answer: free and/or top will show how much memory is available to your system. Even if this is > 4GB on a 32-bit system. PAE (http://en.wikipedia.org/wiki/Physical_Address_Extension) will let your 32-bit machine use up to 64GB of physical RAM, each process however will be limited to 4GB or less.
Answer: its worth noting that 32Bit limits you to 4GB not 2GB for ram, PAE allows you to address (and therefore see) more ram but a single process still cant use more than 4gb of ram
free/top is probably not lying here, try running
<code>free -m
</code>
to see the results in MB rather han bytes or Kb and this may look more like what you think you should be seeing


How to do per-page javascript with the Rails asset pipeline

Question: I understand that for performance reasons it is better to let the asset pipeline concatenate and minify all my javascript and send the whole lot with every page request.  That's fair enough
However, a bunch of my javascript is things like binding specific behaviours to specific page elements - stuff like
$('button').click(function(e) { $('input.sel').val(this.name); }

and I would feel more comfortable if I knew that this code was being executed only on that page - not on evey other page which might coincidentally have elements with the same IDs or which matched the same selectors  How do people deal with this?
I would rather not put all this stuff inline in  elements, just because when it gets to be more than about two lines long, keeping javascript correctly indented inside an .html.erb file is more work than it needs to be
Comment: possible duplicate of [Using Rails 3.1, where do you put your "page specific" javascript code?](http://stackoverflow.com/questions/6167805/using-rails-3-1-where-do-you-put-your-page-specific-javascript-code)
Answer: Here is what I do (based on some stackoverflow answers):
application_helper.rb
def body_page_name
  [controller_name.classify.pluralize, action_name.classify].join
end

application.html.haml
  %body{data: {page: body_page_name}}

application.js
$(function() {
  var page = $("body").data("page");
  if("object" === typeof window[page])
    window[page].init();
});

And in appropriate js file there's an object called ControllerAction:
tickets.js
var TicketsShow = new function() {
  var self = this;

  self.init = function() {
    // code which may call other functions in self
  };
};

There's probably better way to do it, but this works for me
Comment: This was #1 in my response. I've actually never seen it done in rails, because most of my mvc work is in PHP, in which i've implemented this, but I'm voting you up for the code snippet! Cheers
Comment: You get an upvote for the module, I think.  That makes it a bit cleaner than my conditionals
Comment: Thank you guys.

@Kristian

it really more like #3 in your response. #1 is not the case with rails' assets pipeline, because all javascripts compiled into single file so including separate file make assets useless.
Answer: I'll describe what I currently do, just in case it gives anyone a better idea
1) I changed the 'body' tag in my application.html.erb to add the current controller and action as data- attributes
<body data-controller="<%= controller.controller_name %>"
  data-action="<%= controller.action_name %>" >

2) I test this at the top of the relevant javascript
$(document).ready(function() {
        if($('body').data('controller')=='stories') {
            $('.story').click(function(e) {
                    var u=$(this).data('url');
                    u && (document.location=u);
            });
        }
    });

I can't decide if I think this is a good idea or not
Comment: Oh hey, I didn't know you could access data attributes like that. That's really good.
Answer: For page specific JavaScript, I typically do something like this:
Application Helper
In the application helper I create a class attribute (though you could just as well use a data attribute instead).
module ApplicationHelper
  def body_attributes
    controller = params[:controller].gsub('/', ' ')
    action = params[:action]
    version = @version ? "version_#{@version}" : nil
    {
      class: ([controller, action, version] - [nil]).join(' ')
    }
  end
end

Note I'm also adding a version string. This helps with Google content experiments, and makes A/B testing a breeze.
Application.html.haml
In my global layout file, I do something like this to insert the attributes on the body tag:
!!! 5
%html
  %head
    ...
  %body{body_attributes}

script.js
Now in my page specific script, I just check for the class attributes, like this:
$(function () {
  if ($('body.pledge.new, body.pledge.create').length > 0) {
    // do work here...
  }
});

The advantage of this method is that getting the body by class is very quick. The script inside the conditional will not be executed at all on any page apart than the ones I choose, so minimal overhead, and I don't need to change my selectors throughout the code.
EDIT
Note that this answer is now 3 years old. You should be using client-side routing with a framework like React instead.
Answer: I'd add a class to the BODY tag, allowing you to identify each page, and therefore each control per page.

JS:
$('.page1 button').click(function(e) { $('input.sel').val(this.name); }

Answer: I've done it and seen it done in several different ways:

Rigging up the mvc to be able to load a particular js file per page, named along the same lines as a controller file. Like: <controller-name>.js
Making a url parser in JS and then setting a global variable to the current page: UrlParams.currentView = 'dashboard'; and then saying if(UrlParams.currentView == 'dashboard') { //do specific js here }
Setting a unique identifier as the page class or ID and then targeting that with your JS selectors. $('#dashboard').xyz();