mike klaas xdihqxzrsgmailcom 22apr08 216 pm martin v löwis wrote program needs examine contents documentsfeedswhatever web needs deal incorrectlyspecified encodings thats true programs need examine contents web page dont need guess encoding programs encoding hardcoded declared encoding correct programs know page webscraping else couldnt extract information want get certainly agree target set documents small enough possible handcode encoding many applications however need examine content arbitrary least nonsmall set web documents name applications web search engines translation software documentbookmark management systems kinds document analysis market research seo etc feeds give examples incorrectly encoded one dont ever use feeds honestly dont know whether typically encoded incorrectly ive heard often xml case strongly doubt incorrectly encoded also dont much experience feeds statement based fact chardet tool cited thread written specifically use authors feed parsing package whatever give specific examples substantiate documents feeds covers lot webi trying make point web whenever encoding specified specified incorrectly significant chunk exemplars sadly rather common set programs programs need functionality probably set needs beautifulsoupi think set larger browsers give specific examples web browsers programs needing beautifulsoup may still need encoding guessing since still might able hardcode encoding web page want process indeed one site pretty easy work around main use python processing analyzing hundreds millions web documents pretty easy see applications listed think libraries like mark pilgrims feedparser beautifulsoup possible consumers guessing well case im skeptical general guess encoding module would meaningful thing applied incorrectly encoded html pages well wish could easily provide data often necessary whole web would somewhat difficult generate say much important able parse different kinds encoding _specification_ web contenttypecontentencodingmeta httpequiv tags etc malformed cases also think good arguments excluding encoding detection maintenance reasons every case algorithm guessing wrong bug needs fixed stdlib unbounded commitment mike _______________________________________________ pythondev mailing list zvlllneumpythonorg httpmailpythonorgmailmanlistinfopythondev unsubscribe httpmailpythonorgmailmanoptionspythondevpythondev40tangomucom thu 07 aug 2008 162133 0700 pythondev encoding detection standard library