The idea is that local security/SIEM solutions want to populate their database with information gathered from different security feeds. This is done through importing and monitoring .txt files (for some reason).The idea behind this, is that I want each feed to have their own function, to easier be able to filter out, or add new feeds. Not to mention that almost every feed has it's own format.The output of all this, needs to be in the current format, which is:IP [FEED NAME]DOMAIN [FEED NAME]I am sure there are better ways, especially to generate the .txt file outputs.The idea is that the script is to be run by itself, without any input, in a cron file, and if something has to be changed, the script itself should be edited.All feeds are directly available for everyone, so if you are curious about any formats, the URLs are in the script, with the exception of the first feed (OTX), which needs you to sign up, if needed I can paste a output from this feed.So, you can copy and paste this script in and run it out of the box, except you might want to comment out get_alienvault.Questions:Due to that every feed has a different format, I am guessing there would be no way for me to have one function that goes through all feeds, or does someone see a way?I am sure the output can be heavily modified, at a later date, I was thinking of either outputting to .txt, or use the CSV lib to writerows the same way I do now with the txt files.Do I need the main method? Or is there a better way to call all the different functions? Maybe with the possibility to ignore something in the future? Without commenting out the function call?The systems that will be importing this are very sensitive to extra whitespaces and so, currently using lstrip/rstrip for this, and splitlines(). Any better way?Script:import requestsimport refrom OTXv2 import OTXv2from collections import defaultdictfrom datetime import datetime, timedelta# Defines the list that is going to populate the different feedsd = defaultdict(list)# Your API key for OTX. If you do not have any, please comment out# get_alienvault in the main function.otx = OTXv2(XXXXX)class GetFeeds(object):        Main class to gather and output feed results        def download_file(self, url):                Download the feeds specified        :param url: The location of the source to download        :return The content of the request                r = requests.get(url)        return r.content    def ipgrabber(self, results):                :param reults: The results that should be filtered        :return: Only the IP addresses from the object it filtered out                ip = re.findall(r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}', results)        return ip    def get_alienvault(self, days):                Gets the information from Alienvault OTX        :param days: How many days the reslts from the feed can be        :return: The IP addresses and domains returned from OTX                mtime = (datetime.now() - timedelta(days=days)).isoformat()        pulses = otx.getsince(mtime)        # Populate the list with the returned results from OTX        for index, feeds in enumerate(pulses):            for pulse in pulses[index]['indicators']:                if pulse['type'] == 'IPv4':                    d['alienvaultip'].append(pulse['indicator'])                elif pulse['type'] == 'URL':                    d['alienvaulturl'].append(pulse['indicator'])                elif pulse['type'] == 'domain':                    d['alienvaultdomain'].append(pulse['indicator'])        return d    def get_malc0de(self):                Gets the information from the Malc0de Feed        :return: The IP addresses returned from Malc0de                url = 'http://malc0de.com/bl/IP_Blacklist.txt'        results = self.download_file(url).splitlines()        for result in results:            if len(result) > 0:                ip = self.ipgrabber(result.lstrip().rstrip())                if ip:                    d['malc0deip'].append(result)        return d    def get_talosintel(self):                Gets the information from the TalosIntel Feed        :return: The IP addresses returned from TalosIntel                url = 'http://www.talosintelligence.com/feeds/ip-filter.blf'        results = self.download_file(url).splitlines()        for result in results:            if len(result) > 0:                ip = self.ipgrabber(result.lstrip().rstrip())                if ip:                    d['talosintelip'].append(result)        return d    def get_malwaredomainlist(self):                Gets the information from the MDL Feed        :return: The IP addresses and domains returned from MDL                url = 'http://www.malwaredomainlist.com/hostslist/ip.txt'        domains = 'http://www.malwaredomainlist.com/hostslist/hosts.txt'        results = self.download_file(url).splitlines()        results2 = self.download_file(domains).splitlines()        for result in results:            if len(result) > 0:                ip = self.ipgrabber(result.lstrip().rstrip())                if ip:                    d['malwaredomainlistip'].append(result)        for result in results2:            result = result.lstrip('127.0.0.1  , ').rstrip()            if not result.startswith('#') \                    and not result.startswith('localhost') and len(result) > 0:                d['malwaredomainlistdomain'].append(result)        return d    def get_tornodes(self):                Gets the information from the Torproject overview        :return: The IP addresses of the current TOR exit nodes                url = 'https://check.torproject.org/exit-addresses'        results = self.download_file(url).splitlines()        for result in results:            if len(results) > 0:                ip = self.ipgrabber(result.lstrip().rstrip())                if ip:                    d['tornodeip'].append(ip)        return d    def get_ransomwaretracker(self):                Gets the information from the RansomewareTracker Feed        :return: The IP addresses returned from RansomewareTracker                url = 'https://ransomwaretracker.abuse.ch/feeds/csv/'        results = self.download_file(url).splitlines()        for result in results:            if len(result) > 0:                ip = self.ipgrabber(result.lstrip().rstrip())                for singleip in ip:                    d['ransomwaretrackerip'].append(singleip)        for result in results:            if re.search(r'\online\', result):                for url in re.findall(r',\\w+\.\w+\', result):                    url = re.sub(',', '\', url)                    url = re.sub('', '', url)                    d['ransomewaretrackerdomain'].append(url)        return d    def get_bambenek(self):                Gets the information from the RansomewareTracker Feed        :return: The IP addresses and domains returned from Bambenek                url = 'https://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt'        domains = 'http://osint.bambenekconsulting.com/feeds/c2' \                  '-dommasterlist.txt'        results = self.download_file(url).splitlines()        results2 = self.download_file(domains).splitlines()        for result in results:            if len(result) > 0:                ip = self.ipgrabber(result.lstrip().rstrip())                if ip:                    d['bambenekip'].append(ip)        for result in results2:            if not result.startswith('#') \                    and not result.startswith('localhost') and len(result) > 0:                url = result.split(,)[0]                d['bambenekdomain'].append(url)        return d    # Executes the downloading and parsing of feeds    def main(self):        self.get_malc0de()        self.get_alienvault(10)        self.get_talosintel()        self.get_malwaredomainlist()        self.get_tornodes()        self.get_bambenek()        self.get_ransomwaretracker()        # Export IP Addresses to txt file, tagged with their sources        f = open(iplist.txt, w)        iplist = d['ransomwaretrackerip']        f.write(            .join({}\t[Ransomewaretracker]\n.format(t) for t in iplist))        iplist = d['bambenekip']        f.write(.join({}\t[Bambenek]\n.format(t) for t in iplist))        iplist = d['alienvaultip']        f.write(.join({}\t[AlienVault]\n.format(t) for t in iplist))        iplist = d['malc0deip']        f.write(.join({}\t[Malc0de]\n.format(t) for t in iplist))        iplist = d['talosintelip']        f.write(.join({}\t[TalosIntel]\n.format(t) for t in iplist))        iplist = d['malwaredomainlistip']        f.write(.join({}\t[MalwareDomainList]\n.format(t) for t in iplist))        f.close()        # Export domains to txt file, tagged with their sources        f = open(domainlist.txt, w)        domainlist = d['ransomwaretrackerdomain']        f.write(.join(            {}\t[Ransomewaretracker]\n.format(t) for t in domainlist))        domainlist = d['bambenekdomain']        f.write(.join({}\t[Bambenek]\n.format(t) for t in domainlist))        domainlist = d['alienvaultdomain']        f.write(.join({}\t[AlienVault]\n.format(t) for t in domainlist))        domainlist = d['malwaredomainlistdomain']        f.write(            .join({}\t[MalwareDomainList]\n.format(t) for t in domainlist))        f.close()if __name__ == __main__:    o = GetFeeds()    o.main()As output was requested, here is one output per feed:This is one entry for get_alienvault, there will be many arrays of industries:[  {    u'industries': [    ],    u'TLP': u'green',    u'description': u'Analysis of HackingTeam Android malware',    u'created': u'2016-11-15T03:10:51.515000',    u'tags': [      u'android',      u'hackingteam',      u'hacking team',      u'apk',      u'mobile',      u'spyware'    ],    u'modified': u'2016-11-15T03:10:51.515000',    u'author_name': u'AlienVault',    u'public': True,    u'references': [      u'http://rednaga.io/2016/11/14/hackingteam_back_for_your_androids/'    ],    u'targeted_countries': [    ],    u'indicators': [      {        u'indicator': u'e362a037e70517565d28ab85959e6c9d231b2baf0c2df3b87dfaa1451278e80c',        u'description': u'',        u'title': u'',        u'created': u'2016-11-15T03:10:53',        u'content': u'',        u'type': u'FileHash-SHA256',        u'id': 1398126      },      {        u'indicator': u'07278c56973d609caa5f9eb2393d9b1eb41964d24e7e9e7a7e7f9fdfb2bb4c31',        u'description': u'',        u'title': u'',        u'created': u'2016-11-15T03:10:53',        u'content': u'',        u'type': u'FileHash-SHA256',        u'id': 1398127      },      {        u'indicator': u'ed33b83be3af715d3fd8ba6ac8b2b551a16697c5a37a9fcebfc40a024cc9b818',        u'description': u'',        u'title': u'',        u'created': u'2016-11-15T03:10:53',        u'content': u'',        u'type': u'FileHash-SHA256',        u'id': 1398128      },      {        u'indicator': u'4308286905ee3d3dd1f4171da15050d1672dcc63bfc65e012b18938275e96085',        u'description': u'',        u'title': u'',        u'created': u'2016-11-15T03:10:53',        u'content': u'',        u'type': u'FileHash-SHA256',        u'id': 1398129      },      {        u'indicator': u'87efe6a1cbf4d4481c6fa6e2c70a26a0b50a460557a1ad876af9966a571f8a4c',        u'description': u'',        u'title': u'',        u'created': u'2016-11-15T03:10:53',        u'content': u'',        u'type': u'FileHash-SHA256',        u'id': 1398130      },      {        u'indicator': u'68.233.237.11',        u'description': u'',        u'title': u'',        u'created': u'2016-11-15T03:10:53',        u'content': u'',        u'type': u'IPv4',        u'id': 1398131      },      {        u'indicator': u'66.232.100.221',        u'description': u'',        u'title': u'',        u'created': u'2016-11-15T03:10:53',        u'content': u'',        u'type': u'IPv4',        u'id': 1398132      }    ],    u'revision': 1.0,    u'adversary': u'HackingTeam',    u'id': u'582a7cbbc9eef92d31ad7c90',    u'name': u'HackingTeam back for your Androids'  },  {Format for get_malc0deMalc0detxt:// This file will be automatically updated daily and populated with the last 30 days of malicious IP addresses.// Last updated 2016-11-15    103.234.36.7559.45.79.75176.103.55.73108.41.24.14052.84.125.22854.230.193.23754.230.193.52198.143.138.15559.188.68.200222.163.80.69122.225.98.19758.218.211.25061.147.204.4895.131.28.24445.79.194.109193.238.153.44153.196.238.158209.90.88.13194.23.26.10279.127.127.67193.238.153.90189.113.175.24946.28.67.156193.238.153.106Format for get_talosintelTalostxt:119.29.132.1141.169.224.242151.97.190.239188.247.135.58216.176.100.240109.127.8.242188.219.154.228216.215.112.149188.247.135.74188.247.135.53188.247.135.9983.69.233.121203.170.193.23109.229.36.65Format for get_malwaredomainlist:IP addressesMDLtxtfile103.14.120.121103.19.89.55103.224.212.222103.31.186.207103.31.186.29103.4.16.91103.4.218.22103.6.196.156103.8.127.189103.8.127.205104.152.215.90104.245.239.7Domains:MDLdomainstxt#               MalwareDomainList.com Hosts List           ##   http://www.malwaredomainlist.com/hostslist/hosts.txt   ##         Last updated: Fri, 11 Nov 16 06:02:27 +0000      #127.0.0.1  localhost127.0.0.1  0koryu0.easter.ne.jp127.0.0.1  109-204-26-16.netconnexion.managedbroadband.co.uk127.0.0.1  11.lamarianella.info127.0.0.1  1866809.securefastserver.com127.0.0.1  2amsports.com127.0.0.1  3.bluepointmortgage.com127.0.0.1  3.coolerpillow.com127.0.0.1  4.androidislamic.com127.0.0.1  4.collecorvino.org127.0.0.1  4.dlevo.com127.0.0.1  4.e-why.net127.0.0.1  4.newenergydata.biz127.0.0.1  4.newenergydata.info127.0.0.1  4.periziavela.com127.0.0.1  4.pianetapollo.com127.0.0.1  4.whereinitaly.com127.0.0.1  4.whereinlazio.com127.0.0.1  4.whereinliguria.comFormat for get_tornodes:TornodesExitNode 0011BD2485AD45D984EC4159C88FC066E5E3300EPublished 2016-11-15 07:22:42LastStatus 2016-11-15 08:03:23ExitAddress 162.247.72.201 2016-11-15 08:09:55ExitNode 004FDAC2B5B966F53361FA34D7E6E48D512FE869Published 2016-11-14 21:58:30LastStatus 2016-11-14 23:02:24ExitAddress 103.14.68.50 2016-11-14 23:06:35ExitNode 006CC1DD17754582618DE2539DAAFE0A96962583Published 2016-11-14 23:19:25LastStatus 2016-11-15 00:03:24ExitAddress 198.50.159.155 2016-11-15 00:03:42ExitNode 0077BCBA7244DB3E6A5ED2746E86170066684887Published 2016-11-14 23:28:32Format for get_ransomewaretrackerRWTcsvfile:############################################################################### Ransomware Tracker CSV Feed                                                ## Generated on 2016-11-15 15:26:56 UTC                                       ##                                                                            ## For questions please refer to:                                             ## https://ransomwaretracker.abuse.ch/feeds/                                  ################################################################################# Firstseen (UTC),Threat,Malware,Host,URL,Status,Registrar,IP address(es),ASN(s),Country2016-11-15 11:53:22,C2,Locky,185.82.217.29,http://185.82.217.29/message.php,offline,,185.82.217.29,59729,BG2016-11-15 10:07:59,Payment Site,TorrentLocker,ojmekzw4mujvqeju.minitili.at,http://ojmekzw4mujvqeju.minitili.at/,online,,95.163.118.67,207027|12695,RU2016-11-15 07:39:38,Payment Site,Cerber,ffoqr3ug7m726zou.ndnmuk.top,http://ffoqr3ug7m726zou.ndnmuk.top,offline,Eranet International Limited,,34702,2016-11-15 02:48:56,C2,Locky,107.181.174.34,http://107.181.174.34/message.php,offline,,107.181.174.34,46562,US2016-11-14 19:15:48,C2,Locky,93.170.123.119,http://93.170.123.119/message.php,offline,,93.170.123.119,48666,CZ2016-11-14 11:15:34,Payment Site,Cerber,ffoqr3ug7m726zou.rssh3l.bid,http://ffoqr3ug7m726zou.rssh3l.bid,online,,104.238.215.11,36114,US2016-11-14 09:27:25,Payment Site,Cerber,ffoqr3ug7m726zou.2rl0pv.top,http://ffoqr3ug7m726zou.2rl0pv.top,offline,Eranet International Limited,,36114,2016-11-14 06:45:30,Payment Site,Cerber,lfdachijzuwx4bc4.vrgdrs.top,http://lfdachijzuwx4bc4.vrgdrs.top,online,Eranet International Limited,104.238.215.11|104.238.215.11|46.22.220.37,36114|34702,US|EE2016-11-14 04:11:26,Payment Site,Cerber,ffoqr3ug7m726zou.iocvou.top,http://ffoqr3ug7m726zou.iocvou.top,offline,Eranet International Limited,,36114|34702,2016-11-14 01:16:46,Payment Site,Cerber,ahuqfrqk54v3vnzj.8kiec2.top,http://ahuqfrqk54v3vnzj.8kiec2.top,offline,Eranet International Limited,,36114|34702,2016-11-14 00:58:48,Payment Site,Cerber,ffoqr3ug7m726zou.bwei9h.top,http://ffoqr3ug7m726zou.bwei9h.top,offline,EraneFormat for get_bambenekbanbenek:############################################################################### Ransomware Tracker CSV Feed                                                ## Generated on 2016-11-15 15:26:56 UTC                                       ##                                                                            ## For questions please refer to:                                             ## https://ransomwaretracker.abuse.ch/feeds/                                  ################################################################################# Firstseen (UTC),Threat,Malware,Host,URL,Status,Registrar,IP address(es),ASN(s),Country2016-11-15 11:53:22,C2,Locky,185.82.217.29,http://185.82.217.29/message.php,offline,,185.82.217.29,59729,BG2016-11-15 10:07:59,Payment Site,TorrentLocker,ojmekzw4mujvqeju.minitili.at,http://ojmekzw4mujvqeju.minitili.at/,online,,95.163.118.67,207027|12695,RU2016-11-15 07:39:38,Payment Site,Cerber,ffoqr3ug7m726zou.ndnmuk.top,http://ffoqr3ug7m726zou.ndnmuk.top,offline,Eranet International Limited,,34702,2016-11-15 02:48:56,C2,Locky,107.181.174.34,http://107.181.174.34/message.php,offline,,107.181.174.34,46562,US2016-11-14 19:15:48,C2,Locky,93.170.123.119,http://93.170.123.119/message.php,offline,,93.170.123.119,48666,CZ2016-11-14 11:15:34,Payment Site,Cerber,ffoqr3ug7m726zou.rssh3l.bid,http://ffoqr3ug7m726zou.rssh3l.bid,online,,104.238.215.11,36114,US2016-11-14 09:27:25,Payment Site,Cerber,ffoqr3ug7m726zou.2rl0pv.top,http://ffoqr3ug7m726zou.2rl0pv.top,offline,Eranet International Limited,,36114,2016-11-14 06:45:30,Payment Site,Cerber,lfdachijzuwx4bc4.vrgdrs.top,http://lfdachijzuwx4bc4.vrgdrs.top,online,Eranet International Limited,104.238.215.11|104.238.215.11|46.22.220.37,36114|34702,US|EE2016-11-14 04:11:26,Payment Site,Cerber,ffoqr3ug7m726zou.iocvou.top,http://ffoqr3ug7m726zou.iocvou.top,offline,Eranet International Limited,,36114|34702,2016-11-14 01:16:46,Payment Site,Cerber,ahuqfrqk54v3vnzj.8kiec2.top,http://ahuqfrqk54v3vnzj.8kiec2.top,offline,Eranet International Limited,,36114|34702,2016-11-14 00:58:48,Payment Site,Cerber,ffoqr3ug7m726zou.bwei9h.top,http://ffoqr3ug7m726zou.bwei9h.top,offline,EraneDomains:Bambenekdomains############################################################# Master Feed of known, active and non-sinkholed C&Cs domain## names## ## Feed generated at: 2016-11-15 15:16 #### Feed Provided By: John Bambenek of Bambenek Consulting## jcb@bambenekconsulting.com // http://bambenekconsulting.com## Use of this feed is governed by the license here: ## http://osint.bambenekconsulting.com/license.txt #### For more information on this feed go to: ## http://osint.bambenekconsulting.com/manual/c2-dommasterlist.txt## ## All times are in UTC#############################################################ns1.backdates10.com,Domain used by beebone,2016-11-15 15:09,http://osint.bambenekconsulting.com/manual/beebone.txtns1.backdates2.com,Domain used by beebone,2016-11-15 15:09,http://osint.bambenekconsulting.com/manual/beebone.txtns1.backdates3.com,Domain used by beebone,2016-11-15 15:09,http://osint.bambenekconsulting.com/manual/beebone.txtns1.dnsfor15.com,Domain used by beebone,2016-11-15 15:09,http://osint.bambenekconsulting.com/manual/beebone.txtns1.dnsfor7.com,Domain used by beebone,2016-11-15 15:09,http://osint.bambenekconsulting.com/manual/beebone.txtns1.dnsfor8.com,Domain used by beebone,2016-11-15 15:09,http://osint.bambenekconsulting.com/manual/beebone.txtg0spgts27johy2q.ddns.net,Domain used by corebot,2016-11-15 15:08,http://osint.bambenekconsulting.com/manual/corebot.txtk016u4wdspqb10a.ddns.net,Domain used by corebot,2016-11-15 15:08,http://osint.bambenekconsulting.com/manual/corebot.txto8ehahitsbw0s07.ddns.net,Domain used by corebot,2016-11-15 15:08,http://osint.bambenekconsulting.com/manual/corebot.txtjyalcixnmcjafecuk.net,Domain used by fobber,2016-11-15 15:09,http://osint.bambenekconsulting.com/manual/fobber-iplist.txtsuogzzicvq.com,Domain used by fobber,2016-11-15 15:09,http://osint.bambenekconsulting.com/manual/fobber-iplist.txt
Get malware security feeds, and output to file
python;python 2.7
Regarding your questions:See below, the code is very repetitive, so if you notice a case likethat there's almost always a way to reduce the repetition.I'm not sure what you are concerned about, if you need to disablesomething temporarily consider argparse if you drive it from thecommandline with feature flags.That seems sensible, normalising the input by stripping out whitespaceis pretty common.Other remarks about the code:Use with with open so that files are always closed.Using a class is fine I guess to keep things together, but seems a bitpointless given that it's mostly just procedural code.I'd probably remove all the temporary variables and inline things tosave a few lines.The fixed URLs might be better off as constants grouped at the top.d is a global.  That's ... not good.  The one thing that reallyshould be a member variable is not.  Consider moving that into theconstructor if you keep it as a class.Even if the structure isn't exactly the same, at least extract somecommon code into separate methods, e.g.:def get_source(self, url, key):    result = d[key]    for line in self.download_file(url).splitlines():        if len(line) > 0:            if self.ipgrabber(line.lstrip().rstrip()):                result.append(line)or something along those lines.  I've put result at the startbecause it's a minor optimisation, it could be left in the inner loopin case you find it too verbose.The main method has what, about ten duplicate loops?  Again, extractthe shared code with the name of the list as the parameter, e.g.:def write_list(self, f, key, name):    f.write(.join({}\t[{}]\n.format(t, name) for t in d[key]))...# in main    self.write_list(f, 'ransomwaretrackerdomain', 'Ransomewaretracker')Typo in get_tornodes, if len(results) > 0: should use result.The methods are all returning d for no reason.In get_alienvault the loop could be a bit simplified with adefinition like the following (or with a try/catch for themapping[t] access in case the key wasn't defined in mapping,but I dislike that pattern):mappings = {'IPv4': 'alienvaultip', 'URL': 'alienvaulturl', 'domain': 'alienvaultdomain'}for index, feeds in enumerate(pulses):    for pulse in pulses[index]['indicators']:        t = pulse['type']        if t in mappings:            d[mapping[t]].append(pulse['indicator'])