Q:

Using Python Requests to simulate clicking a 'show more' button

I am not sure what code to use for clicking the show more button. I want to get a list of university who are doing certain topic. below is one of the websites    
http://www.sciencedirect.com/science/article/
your helps will be true appreciated 
Thanks 

A:

You shouldn't have to simulate, in Python, an actual "click" of the "show more" button to accomplish web-scraping.
"Show more" buttons in websites are usually tied to some JavaScript that either reveals a hidden element already in the HTML (see Bootstrap's collapse class for a typical example) or fires off a request to some web service (e.g. a REST API) for information to insert in the DOM.
Either way, you can scrape that data. For the former, find the hidden element in the DOM (view the page's source [Ctrl + U] and search the HTML [Ctrl + F]), and use your typical webscraping tools. For the latter, use something like Google Dev Tools' Network tab to inspect the API request when you click "show more" and then try to replicate that request with Python.
In the specific example you've given, it appears the data you want is stored in an HTML <script> tag as a JSON object. Search the HTML for the word "affiliation".

