Q:

Rails, RabbitMQ and Sidekiq Architecture

I have a very trivial - I guess - situation, and several implementation choices. I have a Rails app (call it Core) that, upon user request, crawls data from the internet. The crawler doesn't have any business logic - at least for now - however fetching and persisting data is a job that takes time and I would'n want it to be blocking. 
I've considered different architectures, however I am not sure which one will work the best when things scale up. Options:

Sidekiq: Every time a user requests data to be crawled, a background job in Sidekiq gets born and crawls data, persists them and job is done. Possible drawback here is that I will not pay for Sidekiq Pro edition and I am not sure whether the OSS version will suit my needs.
RabbitMQ: Develop a microservice, let's call it Fetcher, in Rails again, which will consume from RabbitMQ and fetch and persist data. I haven't not understood whether in this case I need background processing as well, so again Sidekiq (or Sneakers). If I do, what's the advantage of using a message broker (in this case) instead of having Sidekiq do the background processing in the same application?

A:

Your architectural proposal is moving in the right direction.

Sidekiq OSS: I truly belive that sidekiq OSS version will properly fit your needs. It has several features to support background jobs supported by Redis. I recommend that you to read its documentation to understand best practices and what should be avoided.
Microservices is a good architectural choice to achieve scalability as it promotes functional decomposition and you can make specific decisions on how to scale each service individually. However, it will increase the complexity of your system in several ways, requiring you to highly automate your infrastructure. I suggest you to read this arcitle from Martin Fowler webpage. He discusses the trade-offs related to the adoption of micro-services.
The advantanges of using RabbitMQ are to enable reliable asynchronous communication between multiple services, support scalability, and provide flexible means for communication such as pub/sub and RPC. If you keep the idea of microservices this is the right way to go. Otherwise, a monolithic Rails application will only require the use of the sidekiq.
If you use RabbitMQ, you should use a proper RabbitMQ client. For ruby, use Bunny in both the publisher and the consumer services since it will provide a good API to publish data on RabbitMQ and will also support background processing for receiving data. Thus, there is no need to use sidekiq to consume data in Fetcher service.
As you are going to do asynchronous processing, your clients will not know the final result of their request with your HTTP responses. So it would be interesting to use a mechanism that allows your application to notify them when your app finish processing the tasks asynchronously (i.e., through a status update on the user page). For this goal you could use Websockets, which is already supported in Rails through Action Cable.

My final tip is: if you're still prototyping your application and experimenting with ideas, I suggest you follow the path of monolithic architecture and be more concerned with validating your proposal. But if you already have a well-structured proposal and know the expected demand for the application, the microservice architecture can be a good choice to develop a scalable and reliable system.

