Q:

Is it okay to add 'www' to any domains?

I modified a web scraping code from https://impythonist.wordpress.com/2015/01/06/ultimate-guide-for-scraping-javascript-rendered-web-pages/.
from PyQt4.QtCore import QUrl
from PyQt4.QtGui import QApplication
from PyQt4.QtWebKit import QWebPage
from sys import argv
from bs4 import BeautifulSoup

# Use result of rendering.
class Render(QWebPage):
  def __init__(self, url):
    self.app = QApplication(argv)
    QWebPage.__init__(self)
    self.loadFinished.connect(self._loadFinished)
    self.mainFrame().load(QUrl(url))
    self.app.exec_()

  def _loadFinished(self, result):
    self.frame = self.mainFrame()
    self.app.quit()

r = Render(url)
result = unicode(r.frame.toHtml().toUtf8(), encoding="UTF-8")

soup = BeautifulSoup(result, 'html.parser')

for script in soup(["script", "style"]):
    script.extract()
text = soup.get_text().encode("utf-8")

With this code, I found that "nate.com" doesn't return any text, but "www.nate.com" does. So, I am trying to add "www" for all domains.

Are there some websites that I shouldn't add "www" to their domains?

(like this)
if "www" in url:
    url = url.split("www")[1]
url = "www" + url

(optional) Why "nate.com" doesn't return any texts but "www.nate.com" does? I found it redirects to "www.nate.com" with chrome.

Any comments are welcome.

A:

Are there some websites that I shouldn't add "www" to their domains?

Yes. For example huji.ac.il:
$ http http://huji.ac.il
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 94
Cache-Control: max-age=300
Connection: Keep-Alive
Content-Length: 173
Content-Type: text/html
Date: Fri, 25 Aug 2017 01:16:23 GMT
Expires: Fri, 25 Aug 2017 01:19:49 GMT
Server: Apache/2.2.15 (Red Hat)

<HTML>
<HEAD>
<meta http-equiv="refresh" content="0; URL=http://new.huji.ac.il">
</HEAD>
<BODY>
<a href="http://new.huji.ac.il">click here</a> jumping ....
</BODY>
</HTML>

Ok, now lets try www.huji.ac.il:
$ http http://www.huji.ac.il
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=300
Connection: close
Content-Length: 173
Content-Type: text/html
Date: Fri, 25 Aug 2017 01:16:31 GMT
Expires: Fri, 25 Aug 2017 01:21:31 GMT
Server: Apache/2.2.15 (Red Hat)

<HTML>
<HEAD>
<meta http-equiv="refresh" content="0; URL=http://new.huji.ac.il">
</HEAD>
<BODY>
<a href="http://new.huji.ac.il">click here</a> jumping ....
</BODY>
</HTML>

In any case it redirects to new.huji.ac.il, lets try it with www:
$ http http://www.new.huji.ac.il

http: error: ConnectionError: HTTPConnectionPool(host='www.new.huji.ac.il', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f54f57fdd90>: Failed to establish a new connection: [Errno -2] Name or service not known',)) while doing GET request to URL: http://www.new.huji.ac.il/

So www in this example caused to error.

Why "nate.com" doesn't return any texts but "www.nate.com" does? I found it redirects to "www.nate.com" with chrome.

Because "nate.com" do redirection with JavaScript
$ http http://nate.com
HTTP/1.1 200 OK
Cache-Control: no-store, no-cache, must-revalidate
Connection: close
Content-Encoding: gzip
Content-Language: ko
Content-Length: 88
Content-Type: text/html; charset=utf-8
Date: Fri, 25 Aug 2017 01:13:34 GMT
Pragma: no-cache
Server: Apache
Vary: Accept-Encoding

<script type='text/javascript'>location.href='http://www.nate.com';</script>

As pointed in comments: you should add a feature to your code to follow redirects.

