<jupyter_start><jupyter_text>Working with Kafka dataDuring a simulation, the producer and the marketplace are constantly logging sales and the activity on the market to Kafka. These information are organised in topics. In order to estimate customer demand and predict good prices, merchants can use the Kafka API to access this data.The merchants gets the data in form of a pandas DataFrame.If you want to try the following examples, make sure that the Pricewars plattform is running.Either by deploying them individually or by using the docker setup. The following step is specific for this notebook.It is not necessary if your merchant is in the repository root.<jupyter_code>import sys
sys.path.append('../')<jupyter_output><empty_output><jupyter_text>Initialize Kafka API You need a merchant token to use the Kafka API.To get one, register the merchant at the marketplace.<jupyter_code>from api import Marketplace
marketplace = Marketplace()
registration = marketplace.register(
    'http://nobody:55000/',
    merchant_name='kafka_notebook_merchant',
    algorithm_name='human')

registration<jupyter_output><empty_output><jupyter_text>It was not possible to connect to the marketplace if you got the following error:```ConnectionError: HTTPConnectionPool(host='marketplace', port=8080)```In that case, make sure that the marketplace is running and host and port are correct.If host or port are wrong, you can change it by creating a marketplace object with the host argument:```marketplace = Marketplace(host='www.another_host.com:1234')```Same is true for the upcoming Kafka API  Next, initialize the Kafka API:<jupyter_code>from api import Kafka
kafka = Kafka(token=registration.merchant_token)<jupyter_output><empty_output><jupyter_text>Request topicYou can request data for specific topics. The most important topics are `buyOffer` which contains your own sales and `marketSituation` which contains a history of market situations.The call will return the data in form of a pandas DataFrame.Depending on how active the simulation is and how much data is logged, this can take some time.<jupyter_code>sales_data = kafka.download_topic_data('buyOffer')
sales_data.head()<jupyter_output><empty_output><jupyter_text>This method may return `None` if it was not possible to obtain the data. For example, this happens if the merchant doesn't have any sales.<jupyter_code>len(sales_data)
market_situations = kafka.download_topic_data('marketSituation')
print(len(market_situations))
market_situations.head()<jupyter_output>5
{'fraction_non_alphanumeric': 0.06534653465346535, 'fraction_numerical': 0.005544554455445544, 'mean_word_length': 6.407624633431085, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '163313', 'license': 'MIT', 'n_tokens_mistral': 707, 'n_tokens_neox': 663, 'n_words': 288, 'path': 'docs/Working with Kafka data.ipynb', 'repo_name': 'cmfcmf/pricewars-merchant', 'size': '2525'}