iptables Tips & Tricks

Question: I'm sure Linux sysadmins are quite familiar with iptables, the userland interface to the netfilter packet-filtering framework.
Now, this "Question" is meant to be a Community Wiki for collecting together various bits-n-pieces of iptables wisdom. Nothing is too common or too obscure. Post anything you know that would help others make the most of iptables.
Answer: Using whitelist and blacklist with iptables
#!/bin/bash

WHITELIST=/whitelist.txt
BLACKLIST=/blacklist.txt

#THIS WILL CLEAR ALL EXISTING RULES!
echo 'Clearing all rules'
iptables -F

#
## Whitelist
#

for x in `grep -v ^# $WHITELIST | awk '{print $1}'`; do
        echo "Permitting $x..."
        $IPTABLES -A INPUT -t filter -s $x -j ACCEPT
done

#
## Blacklist
#

for x in `grep -v ^# $BLACKLIST | awk '{print $1}'`; do
        echo "Denying $x..."
        $IPTABLES -A INPUT -t filter -s $x -j DROP
done

Script to open ports
#!/bin/bash
ALLOWEDTCP="80 3128 3784"
ALLOWEDUDP="3128 3784"

#
## Permitted Ports
#

for port in $ALLOWEDTCP; do
       echo "Accepting port TCP $port..."
       $IPTABLES -A INPUT -t filter -p tcp --dport $port -j ACCEPT
done

for port in $ALLOWEDUDP; do
        echo "Accepting port UDP $port..."
        $IPTABLES -A INPUT -t filter -p udp --dport $port -j ACCEPT
done

Blocking portscan
# Attempt to block portscans
# Anyone who tried to portscan us is locked out for an entire day.
iptables -A INPUT   -m recent --name portscan --rcheck --seconds 86400 -j DROP
iptables -A FORWARD -m recent --name portscan --rcheck --seconds 86400 -j DROP

# Once the day has passed, remove them from the portscan list
iptables -A INPUT   -m recent --name portscan --remove
iptables -A FORWARD -m recent --name portscan --remove

# These rules add scanners to the portscan list, and log the attempt.
iptables -A INPUT   -p tcp -m tcp --dport 139 -m recent --name portscan --set -j LOG --log-prefix "Portscan:"
iptables -A INPUT   -p tcp -m tcp --dport 139 -m recent --name portscan --set -j DROP

iptables -A FORWARD -p tcp -m tcp --dport 139 -m recent --name portscan --set -j LOG --log-prefix "Portscan:"
iptables -A FORWARD -p tcp -m tcp --dport 139 -m recent --name portscan --set -j DROP

Spoofed/Invalid packets
# Reject spoofed packets
# These adresses are mostly used for LAN's, so if these would come to a WAN-only server, drop them.
iptables -A INPUT -s 10.0.0.0/8 -j DROP
iptables -A INPUT -s 169.254.0.0/16 -j DROP
iptables -A INPUT -s 172.16.0.0/12 -j DROP
iptables -A INPUT -s 127.0.0.0/8 -j DROP

#Multicast-adresses.
iptables -A INPUT -s 188.8.131.52/4 -j DROP
iptables -A INPUT -d 18.104.22.168/4 -j DROP
iptables -A INPUT -s 240.0.0.0/5 -j DROP
iptables -A INPUT -d 240.0.0.0/5 -j DROP
iptables -A INPUT -s 0.0.0.0/8 -j DROP
iptables -A INPUT -d 0.0.0.0/8 -j DROP
iptables -A INPUT -d 188.8.131.52/24 -j DROP
iptables -A INPUT -d 255.255.255.255 -j DROP

# Drop all invalid packets
iptables -A INPUT -m state --state INVALID -j DROP
iptables -A FORWARD -m state --state INVALID -j DROP
iptables -A OUTPUT -m state --state INVALID -j DROP

Block Smurf attacks
# Stop smurf attacks
iptables -A INPUT -p icmp -m icmp --icmp-type address-mask-request -j DROP
iptables -A INPUT -p icmp -m icmp --icmp-type timestamp-request -j DROP
iptables -A INPUT -p icmp -m icmp -j DROP

# Drop excessive RST packets to avoid smurf attacks
iptables -A INPUT -p tcp -m tcp --tcp-flags RST RST -m limit --limit 2/second --limit-burst 2 -j ACCEPT

Block ICMP (aka ping)
# Don't allow pings through
iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j DROP

Comment: Consider adding a comment to the "spoofed" comments, so that less experienced users know why the source addresses are considered spoofed (...when arriving on a wan interface).
Comment: Re the whitelist script:  Theres `iptables` on line 8, and then `$IPTABLES` later on.  Is it enough just to use `iptables` everywhere? Otherwise I suppose you'd need to assign something like `IPTABLES=/sbin/iptables` right?
Comment: Good call :-). Done.
Comment: I would not block portscans that way.  Instead have something that will accept tcp/udp connections and looks for multiple related packets.  tcp is easy, just look for established to those ports.  Anything else, others can test for this with spoofed packets and block anything you didn't whitelist, such as your gateway.
Comment: wait. Isn't Block ICMP (aka ping) line redundant to Block smurf attack line: iptables -A INPUT -p icmp -m icmp -j DROP  ?
Comment: also - here is the explanation for many items described here: http://www.newartisans.com/2007/09/neat-tricks-with-iptables.html
Comment: @Andre: Sure, it's redundant. Some people might only want to block ICMP, that's why I mentioned it ;-)
Comment: Correct, @UpTheCreek.
Comment: On blocking portscan: "--dport 139"... copy & paste from the man page, huh? Or, still [netbios paranoia](https://serverfault.com/questions/29065/is-port-139-still-vulnerable)... wait, then one should not have man pages at all ;-)
Comment: @Aaron, would you please expand on your comment with some real examples on blocking portscans?
Comment: I really wish people would stop advising blocking ICMP full stop. Fine filter fragmented ICMP but not ICMP in full. It's meant for network error reporting. And as for Smurf I'm at loss here. It's been a very long time since I've seen the source code but I don't recall it ever using TCP and certainly not RST. Its counterpart Fraggle uses UDP that much I do remember for certain. And yes if private addresses come on the WAN side that's not good but you don't specify this specifically in the rules so it would block all.
Answer: Optimize netfilter's Performance Using ipset
If you write a lot of similar rules based on mere IP, port, or both, consider using ipset to optimize netfilter's performance.
For example:
iptables -s 192.168.1.11 -j ACCEPT
iptables -s 192.168.1.27 -j ACCEPT
iptables -s 192.168.1.44 -j ACCEPT
... hundreds of similar rules ...
iptables -s 192.168.251.177 -j ACCEPT

This means that a packet with the source address of 192.168.251.177 must first traverse hundreds of rules before it can get its verdict of ACCEPT.
Of course, experienced sysadmins will split the rules by subnet. But that still means hundreds of rules.
ipset to the rescue!
First, define an IP Set of ipmap type:
ipset -N Allowed_Hosts ipmap --network 192.168.0.0/16

Then, populate it with the addresses:
for ip in $LIST_OF_ALLOWED_IP; do ipset -A Allowed_Hosts $ip; done

Finally, replace the hundreds of iptables rules above with one rule:
iptables -m set --match-set Allowed_Hosts src -j ACCEPT

When a packet arrives, netfilter will perform a very quick bitmap search for the packet's source (src) IP against the Allowed_Hosts IP Set. All packets coming from 192.168.0.0/16 will experience one rule. And do believe me that searching a bitmap is at least two order of magnitudes faster than performing hundreds of iptables rule-checking.
ipset is not limited to IP addresses. It can also match based on ports, IP-port tuple, network/subnet addresses, IP-MAC tuple, and so on and so forth. And it can match those criteria as source or destination or a mix of both (in the case of tuples).
And finally, with ipset you can automatically put IP addresses in blacklists/whitelists. These blacklists/whitelists can also 'age', thus automatically deleting the IP address after a configurable amount of time has passed.
Please refer to ipset's man page for more details.
VERY IMPORTANT NOTE:
Some Linux distros may not have 'out-of-the-box' support for ipset (e.g. Ubuntu 10.04 had this issue). On these systems one method is to install ipset from source code.
Instead, download ipset's source from its website: http://ipset.netfilter.org/install.html
Alternatively, if you use xtables-addons, ipset is included in its source: http://xtables-addons.sourceforge.net/
Comment: That's a real pity it's not supported by default on Debian and Ubuntu.  I thought you were going to list some obsure distros :/
Comment: @UpTheCreek I've edited my answer... the 'special note' was applicable during time of posting the answer, but no longer applicable now.
Answer: Add comments to your rules:
-m comment --comment "Comments help to read output of iptables -nvL"

Answer: Block Well-Known TCP Attacks
Add the following rules, preferably in -t raw -A PREROUTING
-p tcp --tcp-flags FIN,SYN FIN,SYN -j DROP
-p tcp --tcp-flags SYN,RST SYN,RST -j DROP
-p tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -j DROP
-p tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN -j DROP
-p tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-p tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j DROP

The attacks being blocked are, respectively:

SYN-FIN attack
SYN-RST attack
X-Mas attack
nmap FIN scan
NULLflags attack
ALLflags attack

(feel free to edit the names of the attacks above)
Comment: The `-p tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j DROP` could be removed, as the `-p tcp --tcp-flags FIN,SYN FIN,SYN -j DROP` will catch every packet it could block.
Comment: According to http://security.stackexchange.com/questions/4603/tips-for-a-secure-iptables-config-to-defend-from-attacks-client-side.  "There is no need to drop invalid or malformed packets, all of these attacks are a decade old. The Linux kernel devs are much more up to date than you concerning which kind of packets are valid and which not. "What about future flaws", some might argue. Well, how do you know the future flaw will be in the TCP handler and not in the iptables TCP parser?"
Comment: @VlastimilBurian _theoretically_ they are no longer needed. But adding these set of rules neither slows down the network nor increase CPU Load, so I see no reasons not to add them and forget about them
Comment: For `FIN,SYN,RST,PSH,ACK,URG` you should be able to replace it with just `ALL`.
Answer: Enabling NAT

echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Step 1 sets the kernel parameter to allow for ip forwarding, step 2 sets up an iptables rule that enables NAT on interface eth0.
Comment: That won't be persistent through a reboot, will it?
You need to edit `/etc/sysctl.conf` `net.ipv4.ip_forward = 1`.  (Assuming Red Hat or derivative.)
Answer: Block ICMP attacks
Add the following rules, preferably in -t raw -A PREROUTING
-p icmp -m u32 ! --u32 "4&0x3FFF=0"   -j DROP
-p icmp -m length --length 1492:65535 -j DROP

The first rule blocks all ICMP packets whose "fragmentation flag" is not 0. (ICMP should never be fragmented; they should be carrying small payloads)
The second rule blocks oversized unfragmented ICMP packets.
Comment: Wouldn't that break path MTU Discovery?
Comment: @Matt no, because Path MTU discovery uses packets only as big as the Ethernet payload size, i.e., 1500 bytes, of which 8 bytes are used by the IP header and ICMP header.
Comment: Thanks for the clarification, but 1492-64k?  why is that not 1500-65k.  I can understand 1492 for PPPoE but straight ethernet.
Comment: Is this still applicable?
Comment: @VlastimilBurian I'd say yes. There is still no need for oversized ICMP packets.
Answer: (from my iptables_tricks.txt file, recompiled from a lot of places :P)
Makes iptables wait 15 seconds between new connections from the same IP on port 22 (SSH):
 iptables -A INPUT -p tcp -i eth0 -m state --state NEW --dport 22 -m recent --update --seconds 15 -j DROP
 iptables -A INPUT -p tcp -i eth0 -m state --state NEW --dport 22 -m recent --set -j ACCEPT

Comment: Same, but with counting of attempts: `-A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH`
`-A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 5 --rttl --name SSH -j DROP`
Answer: IPTABLES video tutorials
Episode 1
http://www.youtube.com/watch?v=ldB8kDEtTZA&feature=player_embedded
Episode 2
http://www.youtube.com/watch?v=aDaEXxRHeXY&feature=related
Final episode
http://www.youtube.com/watch?v=JojqHKcSxpo&feature=player_embedded
Will be a good start for any newbie.
Comment: very good tutorial
Answer: using FireHOL - convenient iptables wrapper
I found it much more intuitive than direct iptables commands. Especially for people with past experience with other firewalls:

FireHOL is an iptables firewall
  generator producing stateful iptables
  packet filtering firewalls, on Linux
  hosts and routers with any number of
  network interfaces, any number of
  routes, any number of services served,
  any number of complexity between
  variations of the services (including
  positive and negative expressions).
Comment: I prefer Shorewall which is actively developed, supports IPv4 and IPv6, and can generate firewalls for other iptables systems.
Answer: IP sets revisited
There is already an answer mentioning IP sets. However, it's rather one-dimensional in that it focuses on the performance gains over classic rules and the fact that IP sets mitigate the problem one has with lots of individual IP address that cannot easily be expressed as a subnet in CIDR notation.
Notation used below
For ipset I will use the notation read by ipset restore and written by ipset save.
Correspondingly for iptables (and ip6tables) rules I will use the notation as read by iptables-restore and written by iptables-save. This makes for a shorter notation and it allows me to highlight potential IPv4-only (prefixed -4) or IPv6-only (prefixed -6) rules.
In some examples we'll divert the packet flow into another chain. The chain is assumed to exist at that point, so the lines to create the chains are not produced (nor is the table name mentioned or the commands COMMIT-ted at the end).
Advanced IP sets
IP sets can do a lot more than was mentioned in the other answer and you should definitely read the IP set documentation (ipset(8)) along with iptables-extensions(8) in addition to this brief entry here.
For example I'll mainly focus on three set types: hash:ip, hash:net and list:set, but there are more than those and they all have valid use cases.
You can for example also match port numbers, not just IP addresses.
Saving and restoring IP sets as with iptables-save and iptables-restore
You can create IP set declarations in bulk and import them by piping them into ipset restore. If you want to make your command more resilient against already existing entries, use ipset -exist restore.
If your rules are in a file called default.set you'd use:
ipset -exist restore < default.set

A file like that can contain entries to create sets and to add entries into them. But generally most of the commands from the command line seem to have a corresponding version in the files. Example (creating a set of DNS servers):
create dns4 hash:ip family inet
create dns6 hash:ip family inet6
# Google DNS servers
add dns4 188.8.131.52
add dns4 18.104.22.168
add dns6 2001:4860:4860::8888
add dns6 2001:4860:4860::8844

Here one set is created for IPv4 (dns4) and one for IPv6 (dns6).
Timeouts on IP sets
Timeouts in IP sets can be set as a default per set and also per entry. This is very useful for scenarios where you want to block someone temporarily (e.g. for port-scanning or attempting to brute-force your SSH server).
The way this works is as follows (default during creation of IP sets):
create ssh_loggedon4 hash:ip  family inet  timeout 5400
create ssh_loggedon6 hash:ip  family inet6 timeout 5400
create ssh_dynblock4 hash:ip  family inet  timeout 1800
create ssh_dynblock6 hash:ip  family inet6 timeout 1800

We'll get back to these particular sets below and the rationale as to why they're set the way they are.
If you wanted to set your timeout for a particular IP address, you could simply say:
add ssh_dynblock4 220.127.116.11 timeout 7200

To block IP 188.8.131.52 for two hours instead of the (set) default half hour.
If you were to look at that with ipset save ssh_dynblock4 after a short while, you'd see something along the lines of:
create ssh_dynblock4 hash:ip family inet hashsize 1024 maxelem 65536 timeout 1800
add ssh_dynblock4 220.127.116.11 timeout 6954

Timeout caveats

timeouts are a feature on any given set. If the set was not created with timeout support you'll receive an error (e.g. Kernel error received: Unknown error -1).
timeouts are given in seconds. Use Bash arithmetic expressions to get from minutes to seconds, for example. E.g.: sudo ipset add ssh_dynblock4 184.108.40.206 timeout $((120*60))

Checking whether an entry exists in a given IP set
Inside of your scripts it can be useful to see whether an entry already exists. This can be achieved with ipset test which returns zero if the entry exists and non-zero otherwise. So the usual checks can be applied in a script:
if ipset test dns4 126.96.36.199; then
  echo "Google DNS is in the set"
fi

However, in many cases you'll rather want to use the -exist switch to ipset in order to direct it not to complain about existing entries.
Populating IP sets from iptables rules
This, in my opinion, is one of the killer features of IP sets. Not only can you match against the entries of an IP set, you can also add new entries to an existing IP set.
For example in this answer to this question you have:
-A INPUT -p tcp -i eth0 -m state --state NEW --dport 22 -m recent --update --seconds 15 -j DROP
-A INPUT -p tcp -i eth0 -m state --state NEW --dport 22 -m recent --set -j ACCEPT

... with the intention to rate-limit connection attempts to SSH (TCP port 22). The used module recent keeps track of recent connection attempts. Instead of the state module, I prefer the conntrack module, however.
# Say on your input chain of the filter table you have
   -A INPUT -i eth+ -p tcp --dport ssh -j SSH
# Then inside the SSH chain you can
# 1. create an entry in the recent list on new connections
   -A SSH -m conntrack --ctstate NEW -m recent --set --name tarpit
# 2. check whether 3 connection attempts were made within 2 minutes
#    and if so add or update an entry in the ssh_dynblock4 IP set
-4 -A SSH -m conntrack --ctstate NEW -m recent --rcheck --seconds 120 --hitcount 3 --name tarpit -j SET --add-set ssh_dynblock4 src --exist
-6 -A SSH -m conntrack --ctstate NEW -m recent --rcheck --seconds 120 --hitcount 3 --name tarpit -j SET --add-set ssh_dynblock6 src --exist
# 3. last but not least reject the packets if the source IP is in our
#    IP set
-4 -A SSH -m set --match-set ssh_dynblock4 src -j REJECT
-6 -A SSH -m set --match-set ssh_dynblock6 src -j REJECT

In this case I am redirecting the flow to the SSH chain such that I don't have to repeat myself with -p tcp --dport ssh for every single rule.
To reiterate:

-m set makes iptables aware that we're using switches from the set module (which handles IP sets)
--match-set ssh_dynblock4 src tells iptables to match the source (src) address against the named set (ssh_dynblock4)

this corresponds to sudo ipset test ssh_dynblock4 $IP (where $IP contains the source IP address for the packet)

-j SET --add-set ssh_dynblock4 src --exist adds or updates the source (src) address from the packet into the IP set ssh_dynblock4. If an entry exists (--exist) it will simply be updated.

this corresponds to sudo ipset -exist add ssh_dynblock4 $IP (where $IP contains the source IP address for the packet)

If you wanted to match the target/destination address instead, you'd use dst instead of src. Consult the manual for more options.
Sets of sets
IP sets can contain other sets. Now if you followed the article up to here you'll have wondered whether it's possible to combine sets. And of course it is. For the IP sets from above we can create two joint sets ssh_dynblock and ssh_loggedon respectively to contain the IPv4-only and IPv6-only sets:
create ssh_loggedon4 hash:ip  family inet  timeout 5400
create ssh_loggedon6 hash:ip  family inet6 timeout 5400
create ssh_dynblock4 hash:ip  family inet  timeout 1800
create ssh_dynblock6 hash:ip  family inet6 timeout 1800
# Sets of sets
create ssh_loggedon  list:set
create ssh_dynblock  list:set
# Populate the sets of sets
add ssh_loggedon ssh_loggedon4
add ssh_loggedon ssh_loggedon6
add ssh_dynblock ssh_dynblock4
add ssh_dynblock ssh_dynblock6

And the next question that should pop up in your mind is whether this allows us to match and and manipulate IP sets in an IP version-agnostic fashion.
And the answer to that is a resounding: YES! (alas, this wasn't documented explicitly last time I checked)
Consequently the rules from the previous section can be rewritten to read:
-A INPUT -i eth+ -p tcp --dport ssh -j SSH
-A SSH -m conntrack --ctstate NEW -m recent --set --name tarpit
-A SSH -m conntrack --ctstate NEW -m recent --rcheck --seconds 120 --hitcount 3 --name tarpit -j SET --add-set ssh_dynblock src --exist
-A SSH -m set --match-set ssh_dynblock src -j REJECT

which is a lot more concise. And yes, this is tried and tested and works like a charm.
Putting it all together: SSH brute-force defense
On my servers I have a script run as a cron job which takes a bunch of host names and resolves those to IP addresses, then feeding it into the IP set for "trusted hosts". The idea is that trusted hosts get more attempts to log into the server and aren't necessarily blocked out for as long as anybody else.
Conversely I have whole countries blocked out from connecting to my SSH server, with the (potential) exception of trusted hosts (i.e. order of rules matters).
However, that is left as an exercise for the reader. Here I'd like to add a neat solution that will use the sets contained in the ssh_loggedon set to allow subsequent connection attempts to be handed through and not be subject to the same scrutiny as the other packets.
It is important to remember the default timeouts of 90 minutes for ssh_loggedon and 30 minutes for ssh_dynblock when looking at the following iptables rules:
-A INPUT -i eth+ -p tcp --dport ssh -j SSH
-A SSH -m set --match-set ssh_loggedon src -j ACCEPT
-A SSH -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A SSH -m conntrack --ctstate NEW -m recent --set --name tarpit
-A SSH -m conntrack --ctstate NEW -m recent --rcheck --seconds 120 --hitcount 3 --name tarpit -j SET --add-set ssh_dynblock src --exist
-A SSH -m set --match-set ssh_dynblock src -j REJECT

By now you should ask yourself how the connecting IP address ends up in the ssh_loggedon sub-sets. So read on ...
Bonus: adding the IP you log in with during SSH logon
If you have experimented with sshrc and friends, you'll have learned of its shortcomings. But PAM comes to the rescue. A module named pam_exec.so allows us to invoke a script during SSH logon at a point where we know that the user is admitted in.
In /etc/pam.d/sshd below the pam_env and pam_selinux entries add the following line:
session    optional     pam_exec.so stdout /path/to/your/script

and make sure that your version of the script (/path/to/your/script above) exists and is executable.
PAM uses environment variables to communicate what's going on, so you can use a simple script like this one:
#!/bin/bash
# When called via pam_exec.so ...
SETNAME=ssh_loggedon
if [[ "$PAM_TYPE" == "open_session" ]] && [[ -n "$PAM_RHOST" ]]; then
    [[ "x$PAM_RHOST" != "x${PAM_RHOST//:/}" ]] && SETNAME="${SETNAME}6" || SETNAME="${SETNAME}4"
    ipset -exist add $SETNAME "$PAM_RHOST"
fi

Unfortunately the ipset utility doesn't seem to have the built-in smarts of netfilter. So we need to distinguish between IPv4 and IPv6 IP set when adding our entry. Otherwise ipset will assume we want to add another set to the set of sets, instead of the IP. And of course it's unlikely that there would be a set named after an IP :)
So we check for : in the IP address and append 6 to the set name in such case and 4 otherwise.
The end.
Answer: Another GUI that can be used to configure iptables is Firewall Builder.  It lets users create rule elements as objects in a database and then drag-and-drop these objects into a rules editor to create the desired firewall policy.  The application then generates a script file with all the iptables commands required to implement the rules.
Unlike some other iptables GUI solutions where you can only manage one iptables configuration at a time, with Firewall Builder you can manage a large number of iptables configurations all from a single application.  Firewall Builder runs on Linux, Windows and Mac OS X, has been around for over 10 years and has thousands of active users around the world.
Full Disclosure - I'm the co-founder of NetCitadel which is the company that develops Firewall Builder.
Answer: log outgoing connection with uid
iptables -A OUTPUT -m state --state NEW -m tcp -p tcp -m limit --limit 5/m -j LOG --log-uid --log-prefix="outgoing connection: "

port/connection forwarding:
iptables -A PREROUTING -t nat -i eth1 -p tcp --dport 80 -j DNAT --to 10.0.1.7:80
iptables -A INPUT -p tcp -m state --state NEW --dport 80 -i eth1 -j ACCEPT

Comment: I prefer the `NFLOG` target. It allows a longer line prefix and the user mode daemon can log to databases as well.
Answer: Match several interface names with a wildcard
Example: you have eth0 and eth1 and would like to allow any traffic between them?
iptables -A FORWARD -i eth+ -o eth+ -j ACCEPT

I've used this in the past in order to match the veth<something> interfaces dynamically created and named by LXC. So I can match all at once with veth+.
I also intentionally named some interfaces _<something> in order to match against _+.
Comment: Is this a regexp or is it something else? Ah, in your final sentence it seems that maybe you do though that also sounds like a glob. It doesn't concern any of my boxes but maybe clarifying if it's a regexp would be of use to some?
Answer: Limiting all individual IP addresses to an outbound bandwidth quota with IP Sets
You may please to configure your server to only allow each individual IP 15GiByte of bandwidth usage per month, in hopes to reflect or dodge a bandwidth usage attack, perhaps for your metered bandwidth quota at your ISP. It can be accomplished as the following:
First create the IP sets for IPv4 and IPv6:
ipset create IP_QUOTA_SET_OUT hash:ip timeout 345600 counters
ipset create IP_QUOTA_SET_OUT_INET6 hash:ip timeout 345600 counters family inet6

Now add your iptables rules. The first line will add the IP to the set if it is not already there. The second line will not match if the bytes transferred for the IP in the set is greater than the amount specified. Then the same is done alike for IPv6.
iptables -I OUTPUT -m set ! --match-set IP_QUOTA_SET_OUT dst -j SET --add-set IP_QUOTA_SET_OUT dst --timeout 345600
iptables -I OUTPUT -m set --match-set IP_QUOTA_SET_OUT dst --bytes-gt 16106127360 -j DROP

ip6tables -I OUTPUT -m set ! --match-set IP_QUOTA_SET_OUT_INET6 src -j SET --add-set IP_QUOTA_SET_OUT_INET6 src --timeout 345600
ip6tables -I OUTPUT -m set --match-set IP_QUOTA_SET_OUT_INET6 src --bytes-gt 16106127360 -j DROP

This will prevent attacks such as a user requesting a large file from your webserver for a long period of time, or from any service for that matter. The same can be done for the INPUT chain.
Answer: Block uncommon MSS values
iptables -t mangle -A PREROUTING -p tcp \
-m conntrack --ctstate NEW -m tcpmss ! --mss 536:65535 -j DROP

SYNPROXY target for DDos protection

The purpose of this target is to check does host that sent SYN packet establishes the connection or does nothing after initiating SYN connection. If it does nothing, it discards the packet with minimal effort.

Set syn packets to connection tracking table in the raw table
iptables -t raw -A PREROUTING -p tcp -m tcp --dport 80 --syn -j CT --notrack

Enable synproxy for http server:
iptables -A INPUT -p tcp -m tcp --dport 80 -m conntrack --ctstate INVALID,UNTRACKED \
-j SYNPROXY --sack-perm --timestamp --wscale 7 --mss 1460

Resource :
SYNPROXY target from RHEL blog
Answer: Something I do, mainly because of my ignorance of a more elegant solution, is to manually check my Nginx logs every 4 hours and the mail server logs every 2 minutes for excessive access by individual IP's.  I run a few scripts together that:

Check the access.log and list off the top 10 IP's organized by how many hits they have to the server
Dump the results into a log file
Have another script look at that log file and ban any IP that has hit the server more than X amount of times over the past X hours
Save my iptables.save

Here's what it looks like:
autoBanIPs_mail.sh
#!/bin/bash

# This script checks the last 2 minutes of log entries to see if any
# IP has made over 99 connections

now=$(date +"%m_%d_%Y")

/root/bin/checkBadIPs_mail.sh > /home/ipChecker/ipcheckMAIL_$now.txt
cat /home/ipChecker/ipcheckMAIL_$now.txt | \
    grep " \\(\\([9][9]\\)\\|\\([0-9][0-9][0-9]\\+\\)\\) " | \
    awk '{print $2}' > /home/ipChecker/badMailIPs_$now.sh
sed -i "s/^/\/usr\/local\/sbin\/blockIP /g" /home/ipChecker/badMailIPs_$now.sh
/bin/bash /home/ipChecker/badMailIPs_$now.sh
cat /home/ipChecker/ipcheckMAIL_$now.txt >> /home/ipChecker/ipcheckMAIL_$now.log
rm /home/ipChecker/ipcheckMAIL_$now.txt
rm /home/ipChecker/badMailIPs_$now.sh

checkBadIPs_mail.sh
One thing that is VERY important to note here is that you NEED to setup a whitelist or you are going to start blocking a lot of authentic IP's from servers that you just receive a lot of email from or in the case of other logs, IP's that just hit your server a lot for legitimate reasons.  My whitelist is just built into this script by adding grep pipes  right after | grep ']' | that look something like this "grep -v 127.0 |".
You need to take the time to teach your server which high traffic IP's are legit and which aren't.  For me this meant that I had to spend the first week or so checking my logs manually every couple of hours, looking up high traffic ip's on iplocation.net and then adding the legit ones like amazon, box.com or even my home/office IP ranges to this whitelist.  If you don't you will likely be blocked from your own server or you are going to start blocking legit mail/web servers and cause interruptions in email or traffic.
cat /var/log/mail.log | awk \
    -v d1="$(date --date="-2 min" "+%b %_d %H:%M")" \
    -v d2="$(date "+%b %_d %H:%M")" \
    '$0 > d1 && $0 < d2 || $0 ~ d2' | \
    grep '\[' | grep '\]' | \
    grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | grep -v 127.0 | \
    awk '{print $1}' | sort | uniq -c | sort -n | tail -10

BlockIP
#!/bin/bash
sudo iptables -I INPUT -s $1 -j DROP
sudo bash -c "iptables-save > /etc/network/iptables.save"

Again I know this is crude as hell and there is probably a nice clean efficient protocol that does all of this but I didn't know about it and this thing has been going for a year or two now and keeping the bad guys at bay.  The one thing I would very SERIOUSLY recommend is that you have a proxy or another server in the wings that you can use to access your main server.. The reason being is that if you are doing web development one day out of the blue and you ping your self 2000 times in 5 hours for some testing you could get blocked with no way back in except for a proxy.
You can see that in checkBadIPs.sh I've put grep -v 127.0 and in my actual files I have a ton of ignore rules for my own IP's and other trusted IP ranges but sometimes your IP changes, you forget to update and then you're locked out of your own server.
Anyways, hope that helps.
UPDATE
I have changed things a little bit so that now instead of checking every couple hours I have some logs checked every 2 minutes, mainly my ssh auth log and the mail log as they were getting pounded :(.
I setup specific scripts for each log file although it would be easy enough from the manual script I use myself when wanting to inspect logs.  Looks like this:
#!/bin/bash

log=$1 time=$2

cat /var/log/${log} | awk \
    -v d1="$(date --date="-${time} min" "+%b %_d %H:%M")" \
    -v d2="$(date "+%b %_d %H:%M")" \
    '$0 > d1 && $0 < d2 || $0 ~ d2' | \
    grep '\[' | grep '\]' | \
    grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | \
    sort | uniq -c | sort -n | tail -10

This requires 2 inputs when run, the log file you want to scan and how far back into the past you want to scan.
So if I wanted to check mail.log for the ip counts say 75 minutes into the past I would run:
$ sudo script.sh mail.log 75

Comment: I just read about ipset for the first time the other day and was gleefully delighted to learn about what it does.. I'm a little scared to implement it just because I'll probably muck it up in the beginning and shut the server down but it's on my list of things to learn here.  That being said my iptable chain is probably around 30-40 items and only gets 1 new one every day or two so it's not at a point that I'm overly worried.
Comment: Nice! I'd use `ipset` rather than elongating the iptables chain, but the idea is great and I think I'll apply them to my production servers. Thanks for sharing!
Comment: A quick bit of googling found me this, which seems to do what I described above: http://www.cyberciti.biz/faq/iptables-connection-limits-howto/
Comment: Have you considered fail2ban?
Answer: I agree with the comments about ipsets and tcp flags, but there's still a lot missing:
Use xtables-addons geoip match instead of ipsets for country lists.  Update the geoip data regularly (at least once per month).  The data is more dynamic that a fire-and-forget ipset list.
Consider connection state tracking with tcp flags.  For example, a tcp RST or ACK only makes sense for an established connection.  SYN makes sense only for new and related connections.  A SYN for an established connection means either your SYN+ACK was lost or a hack attempt and should be reset as both sides of the connection don't agree as to state.
Although none, SYN+RST and FIN+RST are illegal combinations, SYN+FIN is valid now under TCP fast-open (TCP option 34), especially for DNS. SYN packets, even with fast-open, should not be fragmented.  I do not consider rules with the PSH and URG flags as useful.  Don't confuse connection tracking state with TCP state:  An RST response to a SYN packet is established for tracking purposes.
SYNPROXY is for forwarded packets and adds nothing for locally delivered ones beyond syncookie support.
ICMP error packets will always be in the related state and of length 48:576 if they are valid.  Length 84:1280 for IPv6. All others should be ignored.  As their maximum size is also the minimum MTU, they should never be fragmented.  ICMP requests (pings, timestamps, etc.) will always be new, and replies established.  Drop ICMP packets in other states.
Like the SSH example with a recent list and accepting only subsequent SYN packets, the same should be done for SMTP, and is similar to "greylisting" on the IP address datum alone.
In the filter table, the first (or second, if accepting established state packets first) rule in the input and output chains should accept everything on the loopback interface.  You should trust your own internal packets.  If you can't, you have bigger problems beyond a firewall solution.
Lastly, don't blindly copy rules unless you really understand what they do.  So many similar lists of rules do that, and most of the time, the result is laughable.
Comment: Your assumption that the definitions for IP sets aren't updated (per-country lists) is one you made up. It's perfectly possible to manage this without geoip. But in general I think the whole scheme with geo-locating the peer is doomed to failure with IPv6 anyway. So no point in using that. Rather be smart how you react to "naughty" behavior ...
Answer: #!/bin/bash
# The following iptables/ip6tables configurations have
# been kindly shared with us from ArckWiki. There are
# a few additions apart from what has been defined.
#
#=================Flush current definitions==============
    iptables -F
    ip6tables -F
    iptables -X
    ip6tables -X

#
#=================Chains=================================
#
#----Define chains for opened ports
    iptables -N TCP
    ip6tables -N TCP
    iptables -N UDP
    ip6tables -N UDP

#
#----Setting up the filter table for NAT
#   iptables -N fw-interfaces
#   ip6tables -N fw-interfaces
#   iptables -N fw-open
#   ip6tables -N fw-open

#
#================Default Chain reactions=================
#
#----Default FORWARD reaction
    iptables -P FORWARD DROP
    ip6tables -P FORWARD DROP

#
#----Default OUTPUT reaction
    iptables -P OUTPUT ACCEPT
    ip6tables -P OUTPUT ACCEPT

#
#----Shellshock
    iptables -A INPUT -m string --algo bm --hex-string '|28 29 20 7B|' -j DROP
    ip6tables -A INPUT -m string --algo bm --hex-string '|28 29 20 7B|' -j DROP

#
#----Default INPUT reaction
    iptables -P INPUT DROP
    ip6tables -P INPUT DROP
#
#----Drop spoofing packets
    iptables -A INPUT -i eth0 -s 127.0.0.0/8 -j DROP
    iptables -A INPUT -i wlan0 -s 127.0.0.0/8 -j DROP
    iptables -A INPUT -i wlan1 -s 127.0.0.0/8 -j DROP
    iptables -A INPUT -s 10.0.0.0/8 -j DROP
    iptables -A INPUT -s 169.254.0.0/16 -j DROP
    iptables -A INPUT -s 172.16.0.0/12 -j DROP
    iptables -A INPUT -s 18.104.22.168/4 -j DROP
    iptables -A INPUT -d 126.96.36.199/4 -j DROP
    iptables -A INPUT -s 240.0.0.0/5 -j DROP
    iptables -A INPUT -d 240.0.0.0/5 -j DROP
    iptables -A INPUT -s 0.0.0.0/8 -j DROP
    iptables -A INPUT -d 0.0.0.0/8 -j DROP
    iptables -A INPUT -d 18.104.22.168/24 -j DROP
    iptables -A INPUT -d 255.255.255.255 -j DROP

#
#================Ping rate limiting globally=============
    iptables -A INPUT -p icmp --icmp-type 8 -m limit --limit 30/min --limit-burst 8 -j ACCEPT
    ip6tables -A INPUT -p icmpv6 --icmpv6-type 8 --match limit --limit-burst 8 -j ACCEPT
    iptables -A INPUT -p icmp --icmp-type 8 -j DROP
    ip6tables -A INPUT -p icmpv6 --icmpv6-type 8 -j DROP

#
#----flooding RST packets, smurf attack Rejection
    iptables -A INPUT -p tcp -m tcp --tcp-flags RST RST -m limit --limit 2/second --limit-burst 2 -j ACCEPT
    ip6tables -A INPUT -p tcp -m tcp --tcp-flags RST RST -m limit --limit 2/second --limit-burst 2 -j ACCEPT

#
#----Bogus packet DROP
    iptables -A INPUT -p tcp -m tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
    ip6tables -A INPUT -p tcp -m tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
    iptables -A INPUT -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j DROP
    ip6tables -A INPUT -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j DROP

#
#================RELATED,ESTABLISHED reaction============
    iptables -A INPUT --match conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
    ip6tables -A INPUT --match conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

#
#================unfetered loopback======================
    iptables -A INPUT -i lo -j ACCEPT
    ip6tables -A INPUT -i lo -j ACCEPT

#
#================INVALID catagory of packets=============
    iptables -A INPUT -p 41 -j ACCEPT
    iptables -A INPUT --match conntrack --ctstate INVALID -j DROP
    ip6tables -A INPUT --match conntrack --ctstate INVALID -j DROP

#
#================IPv6 reactions and definitions==========
    ip6tables -A INPUT -s fe80::/10 -p icmpv6 -j ACCEPT
    ip6tables -t raw -A PREROUTING -p icmpv6 -s fe80::/10 -j ACCEPT
    ip6tables -t raw -A PREROUTING --match rpfilter -j ACCEPT
    ip6tables -t raw -A PREROUTING -j DROP
#
#=======Acceptable INVALIDs and a curteous response======
    iptables -A INPUT -p udp --match conntrack --ctstate NEW -j UDP
    ip6tables -A INPUT -p udp --match conntrack --ctstate NEW -j UDP
    iptables -A INPUT -p tcp --syn --match conntrack --ctstate NEW -j TCP
    ip6tables -A INPUT -p tcp --syn --match conntrack --ctstate NEW -j TCP

#
#================Defining the TCP and UDP chains
#
#########################################################
#            Notes for port open definitions            #
# It is important to note that this should be config-   #
# ured differently if you're providing any routing      #
# activity for any purpose. it is up to you to actively #
# define what suites your needs to get the job done.    #
# In this example, I'm exempting IPv6 from being able   #
# to interact with SSH protocols for two reasons. The   #
# first is because it is generally easier and more com- #
# for internal networks to be deployed with IPv4. The   #
# second reason is, IPv6 can be deployed globally.      #
#########################################################
#
#----SSH configured for eth0
    iptables -A TCP -i eth0 -p tcp --dport ssh -j ACCEPT

#!---Blocking SSH interactions in IPv6
    ip6tables -A TCP -p tcp --dport ssh -j DROP

#!---Leave commented for end service device
#   iptables -A TCP -p tcp --dport 80 -j ACCEPT
#   ip6tables -A TCP -p tcp --dport 80 -j ACCEPT
#   iptables -A TCP -p tcp --dport 443 -j ACCEPT
#   ip6tables -A TCP -p tcp --dport 443 -j ACCEPT
#
#!---Uncomment for remote service to this device
#   iptables -A TCP -p tcp --dport 22 -j ACCEPT
#   ip6tables -A TCP -p tcp --dport 22 -j ACCEPT
#
#!---Uncomment if you're providing routing services
#   iptables -A UDP -p udp 53 -j ACCEPT
#   ip6tables -A UDP -p udp 53 -j ACCEPT
#
#=================Tricking port scanners=================
#
#----SYN scans
    iptables -I TCP -p tcp --match recent --update --seconds 60 --name TCP-PORTSCAN -j DROP
    ip6tables -I TCP -p tcp --match recent --update --seconds 60 --name TCP-PORTSCAN -j DROP
    iptables -A INPUT -p tcp --match recent --set --name TCP-PORTSCAN -j DROP
    ip6tables -A INPUT -p tcp --match recent --set --name TCP-PORTSCAN -j DROP

#
#----UDP scans
    iptables -I UDP -p udp --match recent --update --seconds 60 --name UDP-PORTSCAN -j DROP
    ip6tables -I UDP -p udp --match recent --update --seconds 60 --name UDP-PORTSCAN -j DROP
    iptables -A INPUT -p udp --match recent --set --name UDP-PORTSCAN -j DROP
    ip6tables -A INPUT -p udp --match recent --set --name UDP-PORTSCAN -j DROP

#
#----For SMURF attack protection
    iptables -A INPUT -p icmp -m icmp --icmp-type address-mask-request -j DROP
    iptables -A INPUT -p icmp -m icmp --icmp-type timestamp-request -j DROP
    iptables -A INPUT -p icmp -m limit --limit 2/second --limit-burst 2 -j ACCEPT
    ip6tables -A INPUT -p icmpv6 -m limit --limit 2/second --limit-burst 2 -j ACCEPT

#
#----Ending all other undefined connections
    iptables -A INPUT -j DROP
    ip6tables -A INPUT -j DROP

#
#=======Defining the IN_SSH chain for bruteforce of SSH==
#
#!---I've elected to keep IPv6 out of this realm for
#!---ease of use
    iptables -N IN_SSH
    iptables -A INPUT -p tcp --dport ssh --match conntrack --ctstate NEW -j IN_SSH
    iptables -A IN_SSH --match recent --name sshbf --rttl --rcheck --hitcount 3 --seconds 10 -j DROP
    iptables -A IN_SSH --match recent --name sshbf --rttl --rcheck --hitcount 4 --seconds 1800 -j DROP
    iptables -A IN_SSH --match recent --name sshbf --set -j ACCEPT
    iptables -A INPUT -p tcp --dport ssh -m conntrack --ctstate NEW -j IN_SSH

#
#==================Setting up a NAT gateway==============
#
#########################################################
#                                                       #
# I commented this half out because it's not something  #
# that will apply to all setups. Make note of all par-  #
# tinate interfaces and what exactly is going on.       #
#                                                       #
#########################################################
#
#----Setting up the FORWARD chain
#   iptables -A FORWARD --match conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
#   ip6tables -A FORWARD --match conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
#
#
#----Defining the fw-interfaces/open chains for FORWARD
#   iptables -A FORWARD -j fw-interfaces
#   ip6tables -A FORWARD -j fw-interfaces
#   iptables -A FORWARD -j fw-open
#   ip6tables -A FORWARD -j fw-open
#   iptables -A FORWARD -j DROP # Should be REJECT. But, fuck them
#   ip6tables -A FORWARD -j DROP
#   iptables -P FORWARD DROP
#   ip6tables -P FORWARD DROP
#
#
#----Setting up the nat table
#   iptables -A fw-interfaces -i ### -j ACCEPT
#   ip6tables -A fw-interfaces -i ### -j ACCEPT
#   iptables -t nat -A POSTROUTING -s w.x.y.z/S -o ppp0 -j MASQUERADE
#   ip6tables -t nat -A POSTROUTING -s fe::/10 -o ppp0 -j MASQUERADE
#----The above lines should be repeated specifically for EACH interface
#
#----Setting up the PREROUTING chain
#
#######################################################
#                             #
# The PREROUTING chain will redirect either port      #
# targets to be redirected. This can also redirect    #
# traffic inbound to your network from the gateway    #
# to this machine. This can be useful if you're using #
# a honeypot or have any service within your network  #
# that you want to be pointed to a specific device.   #
#                             #
#######################################################
#
#----SSH honeypot server
#   iptables -A fw-open -d HONEYPOT_IP -p tcp --dport 22 -j ACCEPT
#   ip6tables -A fw-open -d HONEYPOT_IP -p tcp --dport 22 -j ACCEPT
#----With intuition, you can configure the above to also direct specific
#----requests to other devices providing those services. The bellow will
#----be for a squid server
#   iptables -A fw-open -d SQUID_IP -p tcp --dport 80 -j ACCEPT
#   ip6tables -A fw-open -d SQUID_IP -p tcp --dport 80 -j ACCEPT
#   iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 8000 -j DNAT --to SQUID_IP
#   ip6tables -t nat -A PREROUTING -i ppp0 -p tcp --dport 8000 -j DNAT --to SQUID_IP
#
#===============Declare configurations=================
    iptables -nvL
    ip6tables -nvL

Comment: With all the shouting, I can't grasp the message of your post.

iptables Tips & Tricks


Why is $\ln\left(1+t\right)\geq\frac{t}{1+t}$?

Question: Why is $\ln\left(1+t\right)\geq\frac{t}{1+t}$ for every $t>-1$?
I have tried letting $f(x)=\ln(1+t)-\frac{t}{1+t}$ but I'm stuck from here. What do I do next? I am supposed to use the mean value theorem, I think I've done it completely wrong.
Comment: What does the mean value theorem say? something about derivatives? Have you taken the derivative?
Comment: Oops, I goofed. SOrry. I plotted $\log_{10}$.
Answer: If $t\ge 0$, then
$$\log (1+t) = \int_1^{1+t}\frac{dx}x =\int_0^t \frac{du}{1+u} \ge \int_0^t\frac{du}{1+t} = \frac{t}{1+t}.$$
An analogous argument could be used when $-1<t<0$. If it's too messy, I would consider doing $1+t\mapsto z$.
Comment: +1: Nice. ${}{}$
Comment: Very elegant ! (+1)
Answer: To atone for my sins...
Let $f(x) = \ln (1+x)-{x \over 1+x}$, note (meaning prove it) that $\lim_{x \downarrow -1} f(x) = \infty$, $\lim_{x \downarrow 1} f(x) = \infty$ and
$f'(x) = 0$ has exactly one solution at $x= 0$ where $f(x) = 0$. Hence $x=0$ is the minimiser on $x>0$.
Hence $f(x) \ge 0$ for all $x \in (-1,\infty)$.
Answer: Consider the function
$$f(t)=\ln\left(1+t\right)-\frac{t}{1+t}$$
$$f'(t)=\frac{t}{(t+1)^2} >0 \quad \forall t \, > 0$$
So, at least $f(t)$ is an increasing function for $t>0$ and $f(0)=0$ means that
$$\ln\left(1+t\right)\geq \frac{t}{1+t}\quad \forall t \, \geq 0$$ On the other side $t=0$ is an extremum and
$$f''(t)=\frac{1-t}{(t+1)^3}$$ shows that this is a minimum value. So it is true as long as $\ln(1+t)$ is defined in the real domain.
Comment: @nyz. Thanks for the edit **but** there is only **one** logarithm (as far as I now, at least). All other are (again at least to me) just artifacts. Cheers :-)
Answer: The fundamental inequality satisfied by logarithm function is $$\log x\leq x-1\,\forall x>0\tag{1}$$ and it can be proved using any chosen definition of $\log x$. In fact combined with the functional equation $$\log xy=\log x+\log y\, \forall x, y>0\tag{2}$$ the above inequality characterizes the logarithm function uniquely.
Let's replace $x$ by $1/x$ in above inequality $(1)$ to get $$-\log x\leq \frac{1-x}{x}$$ or $$\log x\geq \frac{x-1}{x}$$ Putting $x-1=t$ we get $$\log(1+t)\geq\frac{t}{1+t}$$ and note that $x>0$ implies $t>-1$ so that above inequality is valid for all $t>-1$.


Adding images on specific coordinates (OpenCV, matplotlib)

Question: I have an image with coordinates of specific spots which should be replaced with certain images. The coordinates are represented as 3D numpy array with arbitary shape where 1st dimension is the different classes of the images. Plotting array[1, :,].
How should i approach this?
Comment: One idea is to use color thresholding to identify all teal spots, find contours, for each contour, adjust orientation, resize the template icon to the same dimensions as the contour then use Numpy slicing to copy/paste the icon onto the image
Answer: Please have a look at the below code
import cv2
import numpy as np
from skimage import measure

def resize_image(img,size):
    img = cv2.resize(img,(img.shape[0],size[1]),interpolation = cv2.INTER_AREA)
    img = cv2.resize(img,size,interpolation = cv2.INTER_AREA)
    return img

FINAL_TILE_WIDTH = 8

#loading image
tile_mask = cv2.imread("tile.png")
tile_mask = cv2.cvtColor(tile_mask, cv2.COLOR_BGR2GRAY)
tile_mask = resize_image(tile_mask,(36,FINAL_TILE_WIDTH))

img = cv2.imread("image.png")
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
img[img < 127] = 0
img[img != 0] = 255

#final image will be stored here
final_image = np.zeros(img.shape).astype(np.uint8)

#perform connected components on the image
img = measure.label(img).astype(np.uint8)

labels = np.unique(img)

#removing noise detections
for i in labels:
    if i != 0:
        if len(np.where(img == i)[0]) < 30:
            img[img == i] = 0

img[img != 0] = 255

#extracting the contours of all the rectangles. They might not completely be rectangles.
#but the following operation will make sure everything works fine.
_,cnts,_ = cv2.findContours(img.copy(), cv2.RETR_EXTERNAL,
    cv2.CHAIN_APPROX_SIMPLE)

#iterate trough each contour detected
for cnt in cnts:
    #extract the extreme points, here the assumption of rectangles oriented along axis is taken.
    (minX ,minY)= np.min(cnt,axis = 0)[0]
    (maxX, maxY) = np.max(cnt,axis = 0)[0]
    width = maxX - minX
    width = width - width%2
    height = maxY - minY
    height = height - height%2
    center = (int((maxX+minX)/2),int((maxY+minY)/2))
    #find if width is higher or height is higher. This will be used to orient the tile vertically or horizontally.
    orientation = (width > height)
    temp_tile_mask = []
    tile_width = 0
    tile_height = 0
    #change the orientation of the tile
    if orientation:
        temp_tile_mask = tile_mask
        tile_width = width
        tile_height = FINAL_TILE_WIDTH
        temp_tile_mask = resize_image(temp_tile_mask,(tile_width, tile_height))
    else:
        temp_tile_mask = np.transpose(tile_mask)
        tile_width = FINAL_TILE_WIDTH
        tile_height = height
        temp_tile_mask = resize_image(temp_tile_mask,(tile_width, tile_height))
    #apply the tile to the final image
    final_image[center[1]-int(tile_height/2):center[1]+int(tile_height/2),
                center[0]-int(tile_width/2):center[0]+int(tile_width/2)] = temp_tile_mask[:,:]

cv2.imshow("img", final_image)
cv2.waitKey(0)

I did not really work on any optimization, just getting to the requirement.
Also in the above image, the tiles look very smoothened, some resizing can be changed to resolve this.


Cricket Probability question

Question: My understanding of probability is little weak, and it was since high school I am struggling with it.This question struck me yesterday while watching IPL.
...
In a normal game of cricket, there are 10 fielders, 2 batsman, 1 bowler and 2 empires, (total 15) actively present on the field which can be oval or circular spanning across $90$ mts in radius.
Now, often it happens that when a fielder catches the ball, he throws it up in the air showing enthusiasm or being too happy.
Here is where it struck me, that the ball "can" hit any of the person on the ground while ball descends.
So, if I were to calculate the probability that it will definitely hit someone, will it be $\frac{1}{15}$?
If I am correct, then a fielder standing at the slip (position near batsman) and at long on (position near boundary) both will fall under same probability?
I imagined that there are only 2 fielders present on the ground and one of them throws the ball up in the air.The probability that the ball will land up on others head will be (going by the same logic) 1/2 irrespective of the distance. That means 50 times the fielder will get hit out of 100 throws. Seems weird, because I am not considering the "external factors" like the distance between the fielders, wind speed direction etc.
My question is simple :
1) Does the probability remains the same ($\large\frac{1}{15}$). If not, then what will be the probability that ball will hit some one while it descends!
2) Do we need to include external factors that can (actually) influence the outcome to calculate probability of occurrence of any event.
I need people to enlighten me! :)
Thanks
Comment: first of all probability of ball being hit on any person on field can't be $\frac{1}{15}$. because it means the ball will definitely hit to someone. we can calculate it by like how many area these all person cover and how much is the area of the ground.using this it can be answer
Answer: You're right to think that these predictions are strange! The error you seem to be making is to assume that there is only one probability distribution (the uniform one) for a particular set of possible outcomes. If you want to describe some experiment probabilistically, you need two things. First, you need some set of possible outcomes, say $\{x_i\}$. Then you need a probability distribution $p$, which is a function that assigns to each outcome $x_i$ a non-negative real number $p_i$ such that $\sum_ip_i=1$. These are the probabilities - you say that $p_i$ is the probability of outcome $x_i$ occurring.
In your example, the set of outcomes might be written $\{n,x_1,\ldots,x_{15}\}$, where $n$ represents nobody getting hit and $x_i$ represents player $i$ getting hit. In order to come up with a probability distribution that accurately describes the situation, you need to create some kind of mathematical model. A (very rough) first approximation might be to divide the field into squares with the same area as a player's head, and assume that the ball is equally likely to fall in any of these squares, and that if a player is standing in the square the ball hits, he or she will be struck. If there are $N$ such squares, then the probability of a particular player (say player $i$) being hit will be $p_i=1/N$.
This model is of course far too simple to be very accurate. You could next start to think about the distribution of angles and speeds of tosses in order to figure out the probabilities of the ball going various distances in different directions. Wind speeds could deflect the ball (and might depend on height off the ground). Players might also react to seeing the ball in the air, running towards or away from it. There's not really a limit to the number of factors you could include, and as you include more and more, your model will increase in complexity!
Comment: Your explanation is thorough and to the point. My followup question to you is : why don't we consider external factors in other situations like tossing a coin, rolling a dice? Is it like some situation can be explained with classical approach and others cannot?
Comment: We could. In principle (assuming physics is deterministic, etc. etc.) the outcome of a coin toss or die roll is completely determined by the way the object is thrown, but you're never going to have perfect knowledge of the way that the coin was thrown. But for instance, if you hold the coin just above the ground with the heads side up, it will be almost certain to land heads up, so. But if you're tossing a coin to make some decision, or rolling a die in a game, you typically do it in such a way that it's hard to predict what will happen, and so that all outcomes are roughly equally likely.
Comment: What you're calling the "classical approach" is just a particular choice of probability distribution (the uniform one). If you know nothing about the problem whatsoever, this is a good distribution to choose for certain applications (you can look up Shannon entropy to understand why).
Answer: You're considering the classical theory of probability where every event is considered equally likely, try reading axiomatic approach to probability. If you provide more data such as ball speed air speed and direction along with other useful things (like if the players are considered point sized or occupy some area.), it is easy to calculate using the probability as a fraction of favorable outcomes upon total outcomes. Try reading Mathematical Modelling too.
Comment: So we do need to consider external factors while calculating probability?
Comment: @MonK yes, obviously (unless the problem stated to you explicity refers to ignore any such effect)
Answer: There really is a lot of factors that affect this. In general, if a ball is thrown into the air "at random", then the probability that it will hit a person on the field is much less than 1/15.
What you're thinking is that if the ball DOES hit a random person, there is a 1/15 chance that it would be a particular person.
So let's say that a ball is thrown vertically up into the air. The position of the ball's origin will determine how dense that part of the pitch is. For example, if the ball was thrown from the middle of the pitch, the immediate area is densely populated by the umpire, the wicket keeper, the batsmen, etc. There is more chance that someone will be hit than if the ball was caught on the boundary.
For the purpose of a random throw though, we can also assume that the ball can be thrown from anywhere in the pitch, with equal probability of landing anywhere else in the pitch (discounting all the throws that will land outside the pitch) - for this, we map out the area of the pitch that is covered by players.
If we give a generous estimate, assuming most players are flailing about, celebrating their catch, each player can take up to 1 square meter of space, out of the 17000 or so square meters of a cricket ground with a 75 meter radius. This means that there's some 15/17000 chance that the ball will land on someone's area!
However, if the ball was travelling at a flat enough angle, it could cross a larger portion of the pitch at player level, increasing the chance of hitting someone in the face or body.
So there, I hope that clears it up. In short, the origin of the throw, the angle of the throw and the randomness of the throw will affect this probability, but however you calculate it, the probability of a random person getting hit would be much lower than 1/15.

Cricket Probability question


How to deal with outlier in the experiment

Question: My question is related to this one. I'm working on an experiment involving transferring data between different locations all over the world via the Internet. As the Internet can be busy, sometimes it takes longer to transfer the same amount of data than the other times. The difference can be significant, e.g. for the 1mb of data, it normally takes 2 seconds to transfer, but occasionally (around 1 out of 20 times), it can take up to 20 seconds or more.
I am comparing different setups for the same problem. For each one, I perform multiple times and get the average result, i.e. time to solve the problem. However, as outlier can happen at one setup but not on another, my comparison is affected.
So, at the moment, for each setup, I perform M times but only get the average of N of the smallest results (N < M). I also plan to state and justify my method in the paper as well.
So, my question is whether my method is acceptable and if it makes my paper less reliable. I'm also interested in how experimental outlier is handled in general. I'm working in CS but answers from other fields are welcome.
Comment: What you have done seems related to a "trimmed mean" (but not exactly the same). It's OK but it might be easier to justify if you would remove observations at both ends (e.g. remove the 5% smallest and 5% largest times).
Answer: Think of this from the position of a reader who wants to use your research to deside between these setups. She will want to know that one setup is more likely to occationally need substantial longer times even if a "typical" session will be faster. I would report say the 50th percentile (median) and the 95th percentile (your 1 in 20 outlier). That way you give a fair account of both the typical time it takes to solve a problem and what to expect in a "bad run".


Angularjs MMMM dd, yyyy ng-pattern returns error

Question: I have a field that is set for date to be entered as a value but AngularJs returns error on the field whenever I enter a date.
This is how the pattern looks
<code>ng-pattern='/^(?:(((Jan(uary)?|Ma(r(ch)?|y)|Jul(y)?|Aug(ust)?|Oct(ober)?|Dec(ember)?)\ 31)|((Jan(uary)?|Ma(r(ch)?|y)|Apr(il)?|Ju((ly?)|(ne?))|Aug(ust)?|Oct(ober)?|(Sept|Nov|Dec)(ember)?)\ (0?[1-9]|([12]\d)|30))|(Feb(ruary)?\ (0?[1-9]|1\d|2[0-8]|(29(?=,\ ((1[6-9]|[2-9]\d)(0[48]|[2468][048]|[13579][26])|((16|[2468][048]|[3579][26])00)))))))\,\ ((1[6-9]|[2-9]\d)\d{2}))/'
</code>
I've tested the patter on regexr and it's working. The date I'm testing with is June 30, 2016
Comment: Why do you allow matching "Mary" (see `Ma(r(ch)?|y)`)?
Comment: Your regex looks terrible despite it compiles well, you can look at [Debuggex Demo](https://www.debuggex.com/r/zXqxzmY0J6ClQoaP). I would use a different way to test it, you could use a method to validate the date and pass it to ng-pattern like `ng-pattern='myValidation()'`
Answer: I went over your regex (it's very well thought out), and ended up re-writing it based off your concepts.
Here's what I ended up with:
<code>((Jan(uary)?|Mar(ch)?|July?|Aug(ust)?|Oct(ober)?|Dec(ember)?) 31|(Jan(uary)?|Mar(ch)?|Apr(il)?|July?|June?|Aug(ust)?|Oct(ober)?|Sep|(Sept|Nov|Dec)(ember)?) (0?[1-9]|([12]\d)|30)|Feb(ruary)? (0?[1-9]|1\d|2[0-8]|29(?=, ((1[6-9]|[2-9]\d)(0[48]|[2468][048]|[13579][26])|(16|[2468][048]|[3579][26])00)))), (1[6-9]|[2-9]\d)\d{2}
</code>
Here's an example: https://regex101.com/r/gS0xB4/1
And here it is broken up a little:
<code>( // Begin get month and a day
    (Jan(uary)?|Mar(ch)?|July?|Aug(ust)?|Oct(ober)?|Dec(ember)?) 31 // Months with 31 days, and a 31
    | // OR
    (Jan(uary)?|Mar(ch)?|Apr(il)?|July?|June?|Aug(ust)?|Oct(ober)?|Sep|(Sept|Nov|Dec)(ember)?) // Every month but Feb, and...
    (0?[1-9]|([12]\d)|30) // A number from 1-30, optionally padded to 2 digits
    | // OR
    Feb(ruary)? // Feburary, and...
        (0?[1-9]|1\d|2[0-8] // A number from 1-28, optionally padded to 2 digits
        | // OR
        29(?=, ( // A 29, IF it is followed by a comma and...
            (1[6-9]|[2-9]\d)(0[48]|[2468][048]|[13579][26]) // A year which is a multiple of 4, but not a multiple of 100
            | // OR
            (16|[2468][048]|[3579][26])00) // A year which is a multiple of 400
        ) // End 29 conditional
    ) // End Feb
) // End month and day
, (1[6-9]|[2-9]\d)\d{2} // A comma, and a year between 1600 and 9999
</code>
I don't have any capturing groups for breaking it up, since I was just trying to get it to match - but they should be fairly easy to add in. I also removed all the escapes, so you might have to add them back in if the engine demands them.


How do I run commands on a remote machine with admin privilege using powershell

Question: How do I run commands on a remote machine with admin privilege (example: running as Run as Administrator) using PowerShell.
So far I have tried
start-process powershell.exe -verb runas -argumentlists ".\test.ps1"

This is failing.
Comment: In what way is it failing? Error message please.
Comment: have you configured winrm?
Comment: @tony - yes i have
Answer: Executing a Remote Command
Use the Invoke-Command cmdlet to run a command on a remote machine. The syntax is as follows:
Invoke-Command -ScriptBlock <COMMAND> -ComputerName <COMPUTER> -credential <USERNAME>

COMMAND is the command you want to run, COMPUTER is the remote machine's hostname, and USERNAME is the username of the account under which you want to run the command. You will be prompted for the password.
Starting a Remote Session
Use the Enter-PSSession cmdlet to start a remote PowerShell session in which you can run multiple commands using the Session parameter of Invoke-Command:
Enter-PSSession -ComputerName <COMPUTER> -Credential <USERNAME>

Source:  http://www.howtogeek.com/117192/how-to-run-powershell-commands-on-remote-computers/
Answer: Invoke-Command cannot be used in instances where elevation is required.  Additionally, I found the accepted answer to be a bit incomplete.  In my case, however, I wanted to run an .exe file instead  of a script, but I believe my solution can be adapted to handle scripts as well.
The script below will:

Start a PowerShell session
Run the command specified in the $EXE variable
Wait for it to finish
Terminate the session

Don't terminate the session before the job is complete or you will kill the process you started.  To run this in parallel, remove the -Wait argument from Start-Process and watch the jobs before terminating the sessions.
# Enter or already have some administrative-level credentials here
$Cred=Get-Credential
$EXE='"<path to exe>"'
$Arguments=@(<array of arguments>)
$ScriptString="Start-Process -FilePath $EXE -ArgumentList @('$(Arguments -join "','")') -WindowStyle Hidden -Verb RunAs -Wait"
$ScriptBlock=[System.Management.Automation.ScriptBlock]::Create($ScriptString)
$Session=New-PSSession -ComputerName <target computer> -EnableNetworkAccess -Name <name of session> -Credential $Cred
$Job=Invoke-Command -Session $Session -ScriptBlock $ScriptBlock -AsJob
Wait-Job -Job $Job > $null
Remove-PSSession -Session $Session


How to score study for composition

Question: Could somebody tell, step by step, what is the most effective way to study a score, if my goal is to copy a composer's "language" as an exercise: to create a similar piece and to learn from the process?
Should I start by figuring out the harmonies? How about the form?
I ask this because I know that writing in the style of a given composer is a common exercise, but I don't know how to actually do it.
Comment: Thanks for the comment, Im mostly interested in small scales of orchestration for now, such as solo piano or a piano trios.
Comment: What _scale_ of orchestration, etc? Piano alone? Piano accompanying solo instrument? String quartet? ... ?
Comment: One small comment about a not-so-analytical approach, which I've found helpful, for solo piano, is just to look at the appearance of the music on the page. Contrast Bach, Mozart, Beethoven, Chopin, Brahms, Rachmaninoff. I'd wager that I could tell the author of a score (from among those six, for example) just from looking for a minute at the score. It can be made tangible, too: Brahm's piano music has more "open" voicings than much of Beethoven's, and so on. To make such distinctions "formal" is a somewhat separate task, I think. :)
Answer: If all you want is to copy a score's texture and instrumentation and you do not care for the form at all, figure out the harmonies, melodies, use of motives, and instrumentation and do not bother analyzing the form. (I did that for one of my arrangements, "The Wayfarer and the Setting Star" - an arrangement of one Kirby-series musical theme in the style of another Kirby-series theme, the latter of which I was fortunate enough to find a full transcription of all instruments of half of it.)
If you care more about the form and phrase structure and care less about the instrumentation, analyze the form and phrase structure (and quite possibly harmonies and motive use) instead of the instrumentation. This tends to be the tactic I use when I want to compose piano pieces that are in the style of existing orchestral pieces (e.g. "Monarch's Glory", a march in the style of Edward Elgar's major-key Pomp and Circumstance marches).
If you want to copy everything from the instrumentation and texture through the harmonies and motive use to the form and phrase structure...well, you've gotta figure out everything.
Answer:
...Should I start by figuring out the harmonies? How about the form?

Yes to both.
But, you would also look at rhythm, melody, metrical treatment, scale degrees, counterpoint and texture, accompaniment styles, instrumental genres, and on and on...
But to do this well goes beyond a step by step guide. If it were that easy, then all the great master composer works would amount to mere formula. At the very least you can't lump "composer" together. You wouldn't have one step by step method that would work for two composers like, for example, Handel and Bartok, or Bach and Satie.
You should also ask yourself how authentically you are trying to emulate a given composer. You can probably convince friends and family that something is "in the style of Mozart" by playing cadential harmonies with Alberti bass accompaniment and simple even rhythms... but you probably won't convince someone that way if they can tell the difference between Haydn and Mozart.

...I ask this because I know that writing in the style of a given composer is a common exercise...

I think it's more common to use a piece by a master composer as a template to practice composition, but that isn't necessarily an exercise in mimicking that composer's style. I think the important thing is to be aware of the difference.
For example, you could take a Haydn sonata, make a harmonic reduction, note where the cadences and structural sections begin and end, then fill in that skeleton with your own rhythms and melodic contours. The harmonic skeleton gives you a sort of "safe" formal structure to cover a certain amount of space - sort like the lines of a paint by number picture - your add rhythm and melody to make it unique. You started with Haydn, but the end result won't necessarily sound like Haydn.
Comment: Oddly, I found that the same step-by-step method worked for composing marches in the style of both Edward Elgar's major-key Pomp and Circumstance Marches and Eric Coates' orchestral marches (or at least the Dambusters/Calling All Workers/Knightsbridge/Oxford Street slew I picked). I composed both marches for the piano but analyzed tempos, keys, forms, phrase structure of sections, rhythms used, average note lengths of sections, rough accompaniment patterns, degree of chromaticism, chord progressions, melodies, and more of both sets of sources.
Comment: Each point for deep analysis can be done in the same order, step by step. Key and tempo analysis were done first and second, for example, because they were among the easiest to analyze from 1-2 listens. Form was always done earlier than phrase structure because key changes are easier to detect than phrase lengths are to count for me.
Comment: I'm not reading a step by step method in your comment or answer. After "...but analyzed tempos..." you've listed points for deep analysis. My point is those are two different things.
Comment: Yes, but _what_ steps? You have written them out. If it's as easy as step by step, then list it all out. I don't believe that is a simple thing to do.
Comment: You're right, it's not simple - several steps were semi-consciously absorbed into other steps both times. I consciously had to analyze key -> tempo -> form -> phrase structure, but by then, I think I'd already absorbed the melodies, average note lengths of those melodies in each form section, rhythms used, chromaticism (also by section), and use of countermelody (obvious in Coates). Rough accompaniment patterns were analyzed dead last (due to unreliability of Coates sheet music), to the point of often constructing the melody before that, and chord progressions were only incompletely analyzed.

How to score study for composition


Is "QUEUE=" in /etc/sysconfig/sendmail the same as the "-q" switch in sendmail?

Question: CentOS 5.x, SendMail 8.14.4
My Sendmail server is currently configured to retry deferred messages every hour.  In /etc/sysconfig/sendmail I have the following:
DAEMON=yes
QUEUE=1h

I don't see this file mentioned in the Batbook.  If I change the queue interval here (to say, QUEUE=15m) and restart sendmail services, would that effectively be the same as running /usr/sbin/sendmail -q15m?
Comment: Could you name OS/distribution you use?  AFAIK you refer to non sendmail.org  "extras".
Answer: Short answer is yes. To be absolutely accurate the final command would probably be sendmail -bd -q15m.
Assuming a dist based on init scripts you can always have a look at the scripts in /etc/init.d to see what can be set in the sysconfig file and how the commands are built up.


How to do simultaneous formControl validation

Question: I have a parent component with a form that has 2 date-time-picker components, which are formControls. These represent the startDate and endDate.
I also have 2 custom validator directives that I am using on these 2 form controls. One is checking if given input Date is less than the control's date. The other checks if given input Date is greater than control's date.
These 2 validators allow me to validate the following conditions for the startDate and endDate:

startDate and endDate must be in the future(later than now)
startDate must be before endDate

So, my issue is that when I update one of the dates, the validation only happens on the date I updated. For example:

I set startDate first to a valid future date.
I set endDate to before startDate. This makes endDate invalid.
I change startDate to a valid future date before endDate.
I expect endDate to become valid even though it was startDate that changed.

My question is this: How do I elegantly make validity run on another control?
Template of parent component:
<form #projectForm="ngForm" novalidate class="row">

      Start Date
      <date-time-picker #startDate="ngModel" name="startDate" [(ngModel)]="project.startDate" [disabled]="isReadOnly" [dateLessThan]="project.endDate" [dateGreaterThan]="now"></date-time-picker>

Start date should be before end date

Date should be in the future

      End Date
      <date-time-picker #endDate="ngModel" name="endDate" [(ngModel)]="project.endDate" [disabled]="isReadOnly" [dateGreaterThan]="maxDate()"></date-time-picker>

End date is too early

date-less-that-validator.directive.ts:
import { Directive, forwardRef, Input } from '@angular/core';
import { NG_VALIDATORS, FormControl } from '@angular/forms';
import * as moment from 'moment';

/**
 * Directive to validate whether FormControl's date is less than input date
 */
@Directive({
  selector: '[dateLessThan][ngModel],[dateLessThan][formControl]',
  providers: [
    { provide: NG_VALIDATORS, useExisting: forwardRef(() => DateLessThanValidatorDirective), multi: true }
  ]
})
export class DateLessThanValidatorDirective {

  validator: Function;

  constructor() { //If validating onEndDate, reverse the otherDate
      this.validator = this.dateLessThan();
  }

  @Input('dateLessThan') inputDate: Date; // Date comparing against.

  validate(c: FormControl) {
    return this.validator(c);
  }

  /**
 * Factory method that creates a function that accepts a form control.
 * Returns null if form is valid. Returns an object that contains error message if invalid.
 */
  dateLessThan() {
    return (c: FormControl) => {

      let controlDate = c.value;

      if (controlDate && this.inputDate) { //Only if both dates are set do we do validation
        if (moment(controlDate).diff(this.inputDate) > 0) {
          return {
            dateLessThan: 'Controls date is greater than given date'
          };
        }
      }
      return null;
    };
  }
}

date-greater-than-validator.directive.ts:
import { Directive, forwardRef, Input } from '@angular/core';
import { NG_VALIDATORS, FormControl } from '@angular/forms';
import * as moment from 'moment';

/**
 * Directive to validate whether FormControl's date is greater than input date
 */
@Directive({
  selector: '[dateGreaterThan][ngModel],[dateGreaterThan][formControl]',
  providers: [
    { provide: NG_VALIDATORS, useExisting: forwardRef(() => DateGreaterThanValidatorDirective), multi: true }
  ]
})
export class DateGreaterThanValidatorDirective {

  validator: Function;

  constructor() { //If validating onEndDate, reverse the otherDate
      this.validator = this.dateGreaterThan();
  }

  @Input('dateGreaterThan') inputDate: Date; // Date comparing against.

  validate(c: FormControl) {
    return this.validator(c);
  }

  /**
 * Factory method that creates a function that accepts a form control.
 * Returns null if form is valid. Returns an object that contains error message if invalid.
 */
  dateGreaterThan() {
    return (c: FormControl) => {

      let controlDate = c.value;

      if (controlDate && this.inputDate) { //Only if both dates are set do we do validation
        if (moment(controlDate).diff(this.inputDate) < 0) {
          return {
            dateGreaterThan: 'Controls date is less than given date'
          };
        }
      }

      return null;
    };
  }
}

Answer: The behavior which you are observing is quite expected. Validation is triggered only for the controls which are being changed by user interaction.
You can approach the scenario in the following way:

create custom validator for startDate|endDate. It will only validate whether current formControl value is in the future.
create custom validator for the formGroup which wraps startDate|endDate. This way when any of the fields is changed you will be able to check whether start < end date.

Implementing two validators separates nicely the responsibilities and encapsulates domain logic in each validator.
I hope this makes sense to you.
Comment: That makes sense, But when I am passing in variables outside of the control into the validator, shouldn't angular look for updates/changes to those variables as well?
Comment: I don't think so. The validator is associated with only one date time picker at a time.
If you apply both validators for a group, by using `ngModelGroup`, both will run when changing either of the fields.
Answer: My approach worked:
  private MaxCompareValidator(maxformControl: FormControl): ValidatorFn {
    let subscribe = false;
    return (control: AbstractControl): { [key: string]: boolean } | null => {
        if (!subscribe) {

            subscribe = true;
            maxformControl.valueChanges.subscribe(() => {
                control.updateValueAndValidity();
            });
        }

        if (!maxformControl || !maxformControl.value)
            return { dateCompareInvalid: false };
        if (control.value && new Date(control.value) > new Date(maxformControl.value)) {
            return { dateCompareInvalid: true };
        }
        return { dateCompareInvalid: false };
    };
}

 var newServiceStartTimeCtrl = new FormControl();
                            newServiceStartTimeCtrl.setValue(activity.startDate.toISOString());

                            var newServiceEndTimeCtrl = new FormControl();
                            newServiceEndTimeCtrl.setValue(activity.endDate.toISOString());

                            newServiceStartTimeCtrl.setValidators([Validators.required, this.MaxCompareValidator(newServiceEndTimeCtrl)]);
                            newServiceEndTimeCtrl.setValidators([Validators.required]);


EclipseLink and GlassFish: Predeployment fails: NullPointerException

Question: When I try to deploy my EJB-application to glassfish (3.1.2), I randomly get "Predeployment Errors". Some times it works, sometimes it doesn't, with a 50%-chance.
I searched the internet for this problem, but it seems that I am the only one with this problem. I am not able figure out what the problem is by analyzing the stack trace, either.
Does anybody have an idea what's the problem here? This is really killing me.
SCHWERWIEGEND: Exception while invoking class org.glassfish.persistence.jpa.JPADeployer prepare method
SCHWERWIEGEND: Exception while invoking class org.glassfish.javaee.full.deployment.EarDeployer prepare method
SCHWERWIEGEND: Exception while preparing the app
SCHWERWIEGEND: Exception [EclipseLink-28018] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Predeployment of PersistenceUnit [SeatingPlannerEJB] failed.
Internal Exception: java.lang.NullPointerException
org.glassfish.deployment.common.DeploymentException: Exception [EclipseLink-28018] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Predeployment of PersistenceUnit [SeatingPlannerEJB] failed.
Internal Exception: java.lang.NullPointerException
    at org.glassfish.javaee.full.deployment.EarDeployer.prepare(EarDeployer.java:166)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.prepareModule(ApplicationLifecycle.java:871)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:410)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
    at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:389)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1085)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
    at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
    at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
    at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
    at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
    at com.sun.enterprise.v3.services.impl.ContainerMapper$Hk2DispatcherCallable.call(ContainerMapper.java:354)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
    at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
    at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
    at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
    at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
    at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
    at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
    at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
    at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
    at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
    at java.lang.Thread.run(Thread.java:662)
Caused by: javax.persistence.PersistenceException: Exception [EclipseLink-28018] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Predeployment of PersistenceUnit [SeatingPlannerEJB] failed.
Internal Exception: java.lang.NullPointerException
    at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.predeploy(EntityManagerSetupImpl.java:1402)
    at org.eclipse.persistence.jpa.PersistenceProvider.createContainerEntityManagerFactory(PersistenceProvider.java:208)
    at org.glassfish.persistence.jpa.PersistenceUnitLoader.loadPU(PersistenceUnitLoader.java:206)
    at org.glassfish.persistence.jpa.PersistenceUnitLoader.(PersistenceUnitLoader.java:120)
    at org.glassfish.persistence.jpa.JPADeployer$1.visitPUD(JPADeployer.java:224)
    at org.glassfish.persistence.jpa.JPADeployer$PersistenceUnitDescriptorIterator.iteratePUDs(JPADeployer.java:495)
    at org.glassfish.persistence.jpa.JPADeployer.createEMFs(JPADeployer.java:233)
    at org.glassfish.persistence.jpa.JPADeployer.prepare(JPADeployer.java:168)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.prepareModule(ApplicationLifecycle.java:871)
    at org.glassfish.javaee.full.deployment.EarDeployer.prepareBundle(EarDeployer.java:290)
    at org.glassfish.javaee.full.deployment.EarDeployer.access$200(EarDeployer.java:86)
    at org.glassfish.javaee.full.deployment.EarDeployer$1.doBundle(EarDeployer.java:141)
    at org.glassfish.javaee.full.deployment.EarDeployer$1.doBundle(EarDeployer.java:138)
    at org.glassfish.javaee.full.deployment.EarDeployer.doOnBundles(EarDeployer.java:215)
    at org.glassfish.javaee.full.deployment.EarDeployer.doOnAllTypedBundles(EarDeployer.java:224)
    at org.glassfish.javaee.full.deployment.EarDeployer.doOnAllBundles(EarDeployer.java:250)
    at org.glassfish.javaee.full.deployment.EarDeployer.prepare(EarDeployer.java:138)
    ... 30 more
Caused by: Exception [EclipseLink-28018] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Predeployment of PersistenceUnit [SeatingPlannerEJB] failed.
Internal Exception: java.lang.NullPointerException
    at org.eclipse.persistence.exceptions.EntityManagerSetupException.predeployFailed(EntityManagerSetupException.java:221)
    ... 47 more
Caused by: java.lang.NullPointerException
    at org.eclipse.persistence.internal.jpa.metadata.accessors.mappings.OneToOneAccessor.process(OneToOneAccessor.java:116)
    at org.eclipse.persistence.internal.jpa.metadata.MetadataProject.processNonOwningRelationshipAccessors(MetadataProject.java:1436)
    at org.eclipse.persistence.internal.jpa.metadata.MetadataProject.processStage3(MetadataProject.java:1707)
    at org.eclipse.persistence.internal.jpa.metadata.MetadataProcessor.processORMMetadata(MetadataProcessor.java:535)
    at org.eclipse.persistence.internal.jpa.deployment.PersistenceUnitProcessor.processORMetadata(PersistenceUnitProcessor.java:526)
    at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.predeploy(EntityManagerSetupImpl.java:1337)
    ... 46 more

SCHWERWIEGEND: Exception while preparing the app : Exception [EclipseLink-28018] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Predeployment of PersistenceUnit [SeatingPlannerEJB] failed.
Internal Exception: java.lang.NullPointerException
javax.persistence.PersistenceException: Exception [EclipseLink-28018] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Predeployment of PersistenceUnit [SeatingPlannerEJB] failed.
Internal Exception: java.lang.NullPointerException
    at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.predeploy(EntityManagerSetupImpl.java:1402)
    at org.eclipse.persistence.jpa.PersistenceProvider.createContainerEntityManagerFactory(PersistenceProvider.java:208)
    at org.glassfish.persistence.jpa.PersistenceUnitLoader.loadPU(PersistenceUnitLoader.java:206)
    at org.glassfish.persistence.jpa.PersistenceUnitLoader.(PersistenceUnitLoader.java:120)
    at org.glassfish.persistence.jpa.JPADeployer$1.visitPUD(JPADeployer.java:224)
    at org.glassfish.persistence.jpa.JPADeployer$PersistenceUnitDescriptorIterator.iteratePUDs(JPADeployer.java:495)
    at org.glassfish.persistence.jpa.JPADeployer.createEMFs(JPADeployer.java:233)
    at org.glassfish.persistence.jpa.JPADeployer.prepare(JPADeployer.java:168)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.prepareModule(ApplicationLifecycle.java:871)
    at org.glassfish.javaee.full.deployment.EarDeployer.prepareBundle(EarDeployer.java:290)
    at org.glassfish.javaee.full.deployment.EarDeployer.access$200(EarDeployer.java:86)
    at org.glassfish.javaee.full.deployment.EarDeployer$1.doBundle(EarDeployer.java:141)
    at org.glassfish.javaee.full.deployment.EarDeployer$1.doBundle(EarDeployer.java:138)
    at org.glassfish.javaee.full.deployment.EarDeployer.doOnBundles(EarDeployer.java:215)
    at org.glassfish.javaee.full.deployment.EarDeployer.doOnAllTypedBundles(EarDeployer.java:224)
    at org.glassfish.javaee.full.deployment.EarDeployer.doOnAllBundles(EarDeployer.java:250)
    at org.glassfish.javaee.full.deployment.EarDeployer.prepare(EarDeployer.java:138)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.prepareModule(ApplicationLifecycle.java:871)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:410)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
    at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:389)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1085)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
    at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
    at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
    at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
    at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
    at com.sun.enterprise.v3.services.impl.ContainerMapper$Hk2DispatcherCallable.call(ContainerMapper.java:354)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
    at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
    at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
    at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
    at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
    at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
    at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
    at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
    at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
    at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
    at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
    at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
    at java.lang.Thread.run(Thread.java:662)
Caused by: Exception [EclipseLink-28018] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Predeployment of PersistenceUnit [SeatingPlannerEJB] failed.
Internal Exception: java.lang.NullPointerException
    at org.eclipse.persistence.exceptions.EntityManagerSetupException.predeployFailed(EntityManagerSetupException.java:221)
    ... 47 more
Caused by: java.lang.NullPointerException
    at org.eclipse.persistence.internal.jpa.metadata.accessors.mappings.OneToOneAccessor.process(OneToOneAccessor.java:116)
    at org.eclipse.persistence.internal.jpa.metadata.MetadataProject.processNonOwningRelationshipAccessors(MetadataProject.java:1436)
    at org.eclipse.persistence.internal.jpa.metadata.MetadataProject.processStage3(MetadataProject.java:1707)
    at org.eclipse.persistence.internal.jpa.metadata.MetadataProcessor.processORMMetadata(MetadataProcessor.java:535)
    at org.eclipse.persistence.internal.jpa.deployment.PersistenceUnitProcessor.processORMetadata(PersistenceUnitProcessor.java:526)
    at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.predeploy(EntityManagerSetupImpl.java:1337)
    ... 46 more
Answer: The error is occurring in, OneToOneAccessor, so is related to one of your @OneToOne relationships.  My guess is that it is using a mappedBy but the target relationship name is not correct.
Try to determine the mapping causing the issue and include the code for it.
Also try the latest EclipseLink release, from the source code it seem like it will throw a proper exception at least in the case of a missing mappedBy.
Comment: You were right. There was an "illegal" OneToOne-Relationship between a Class and it's superclass. Once I removed it, everything worked fine.


Prove the order of finite simple group is not perfect and even number.

Question: Prove the order of finite simple group is not  perfect and even number.
From elementary number theory, all even perfect
numbers are of the form
$$
2^{p-1}\left(2^{p}-1\right)
$$
where $p$ is a prime number and $2^{p}-1$ is also a prime-
number.
Suppose the order of a group $G$ is $2^{p-1}\left(2^{p}-1\right),$ with prime $p, 2^{p}-1$
Then the number $n_{2^{p}-1}$ of Sylow $\left(2^{p}-1\right)$ -subgroup satisfies
$n_{2^{p}-1} \equiv 1 \quad\left(\bmod 2^{p}-1\right)$ and $n_{2^{p}-1} | 2^{p-1}$
These force that $n_{2^{p}-1}=1$ Therefore the group $G$ contains the unique normal
Sylow $\left(2^{p}-1\right)$ -subgroup, hence $G$ is not simple. But is this proof unique ? I want proof without use sylow subgroup.
Comment: Please provide some context.  Where did you encounter this problem?  What are your thoughts on the problem?  What have you tried so far?
Comment: Are you allowed to use Cauchy's Theorem, which says that if the prime $q$ divides $|G|$ then $G$ has an element of order $q$? If so, let $R$ be the subgroup of order $q:=2^p-1$ generated by such an element, let $g \in G$ and let $S=g^{-1}Rg$. Then we must have $g^{-1}Rg=R$ (and hence $R$ is normal in $G$), because otherwise the order of $|RS|$ would be $q^2$, which is greater than $|G|$.
Answer: Any even perfect number is of the form $2^{n-1} \cdot (2^n-1)$ for some $n$ such that $2^n-1$ is prime. By Sylow's Theorems, the number of $(2^n-1)$-Sylow groups divides $2^{n-1}$ and is congruent to $1$ modulo $2^n-1$. This implies that the number of $(2^n-1)$-Sylows is $1$, and hence the $(2^n-1)$-Sylow is normal.
Comment: But four minutes after you posted this good solution to the question the OP added your good proof to the question without any acknowledgement and said they didn't want to use Sylow. Makes on wonder why one helps, doesn't it?


Transparent Proxy to Docker Network Means TCP is Broken

Question: My logging setup is a single Docker host with UDP 514 exposed for syslog.  An nginx container has its port published so when you send logs to 10.1.1.100 (in the image below) it first hits nginx, whose config for transparent load balancing to Logstash containers is:
user root;
events {worker_connections 32768;}
stream {
 upstream logstash_servers {
  server logstash-collector-01:514;
  server logstash-collector-02:514;
  server logstash-collector-03:514;
 }
 listen 514 udp;
 proxy_pass logstash_servers;
 proxy_bind $remote_addr transparent;
 }
}

This works fine.  However, TCP 514 (or anything TCP, for that matter) doesn't.  Even when I add the right listener and configs, I believe the TCP handshake does not complete because with nginx doing transparent load balancing, its proxy_bind passes along, e.g. 10.1.1.5 as a source IP to e.g. 172.18.0.4 (a Logstash instance).  That instance then tries to complete the handshake but 10.1.1.5 (and any routers along the way) does not know how to route to the Docker network of 172.18.0.0/16.
Is there a solution here to be able to use TCP for logging?
Comment: Why do you want to use TCP?
Comment: @EEAA To support all logging protocols, such as GELF and syslog-ng over TCP
Answer: It sounds like nginx might not be the best tool for the job here - either haproxy (if you want simple TCP/UDP load-balancing), or rsyslog (if you want something slightly more application-aware) would seem better fits, and would fulfill the same function as nginx currently is.
Looking at your setup, you could even have a front-end logstash listener on 514, that could forward to another logstash instance for actual processing.
As logstash doesn't AFAIK do load-balancing, you could instead 'route' events based on some tags/types you put into place.
Checking https://www.nginx.com/resources/admin-guide/tcp-load-balancing/, in order to have both TCP and UDP, you would need something like:
stream {
    upstream logstash_servers {
        server logstash-collector-01:514;
        server logstash-collector-02:514;
        server logstash-collector-03:514;
    }
    server {
        listen 514;
        ...
    }
    server {
        listen 514 udp;
        ...
    }
}

I am not completely clear on whether using the logstash_servers in the UDP server block will cause nginx to speak UDP or TCP to them, so you might need to make both available.
I would be a little doubtful that the issue is down to the nginx node using the 'wrong' IP as source when connecting to the logstash backends - typically, your nginx front end will use whatever the 'right' IP is to do that, which in your case, would mean an address within 172.18/16.
This is similar to how nginx is used when reverse proxying to localhost: in that situation, nginx will communicate with the localhost application using a source IP of 127.0.0.1 (or ::1), not the IP that nginx saw the request coming from.
And there is also the fact that you seem to be saying this all works for UDP - which would tend to say routing is not an issue as such, but that something in your TCP setup might be.