I am building a script that checks a large network of sites for invalid links. The idea is to flag links that continuously show as not available so that they can be cleaned by the administration team.The basic process is this:Grab a set of pages - approximately 30K per day, not the same 30K each day - and extract all links from the pagesInsert Unique URLs into the Links TableInsert records into LinksInPage containing the anchor text for each link as well as the associated LinkID for each linkCheck the status of each link utilizing HEAD/GET methods and inserting results into the LinkResults tableI am currently planning on building this database structure in an relational database.Tables:LinksLinkID     (auto inc)URL    (varchar)LinksInPageLinkID (foreign key to Links.LinkID)AnchorText (varchar)Page URL   (varchar)LinkResultsLinkID (foreign key to Links.LinkID)DateOfCheck (date)StatusCode (int)NumFailures (int)My question:How can I efficiently handle non-unique links in this situation (I'm open to completely changing how the database is laid out)? For example, on day one if I check example.com, I insert a record into the Links table. On day two, if I have another page that also links example.com, I don't wanta duplicate entry in the Links table, but I do need a record in LinksInPage that points to the same LinkID. At, assuming at least one link per page, 30K links daily I don't think querying for a LinkID at each insert is going to be efficient.Storing everything in memory also seems like it will lead to problem in the future as the number of links gets larger. I also need to be able to handle the LinkID when inserting link results. Again, looking up the LinkID at each insert seems inefficient.The plan is to build this utilizing Python. 
Efficient way to handle foreign keys
design;design patterns;python