<jupyter_start><jupyter_text>Factors1. Macro fiscal and monetary balances have improved2. Government debt costs have trended lower and foreign reserves have risen3. Overall cost of capital has declined4. Corporate valuations have improved relative to global peers5. Real asset ZAR returns have compared favourably6. China and African trade rise has largely offset European trade decline7. Disposable income of South Africans has risen8. The rise of the black middle class has led to a structural boost in spending9. Wage inflation and government grants have supported this trend10. Per unit labour productivity has improved ---  Ores and metals imports (% of merchandise imports) - South AfricaOres and metals imports (% of merchandise imports) - South Africa.https://www.quandl.com/data/WWDI/ZAF_TM_VAL_MMTL_ZS_UN-Ores-and-metals-imports-of-merchandise-imports-South-Africa<jupyter_code>data = pd.read_csv('ores_metal_imports.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Gold Price: London FixingGold Price: London Fixings, London Bullion Market Association (LBMA). Fixing levels are set per troy ounce. The London Gold Fixing Companies set the prices for gold that are globally considered as the international standard for pricing of gold. The Gold price in London is set twice a day by five LBMA Market Makers who comprise the London Gold Market Fixing Limited (LGMFL). The process starts with the announcement from the Chairman of the LGMFL to the other members of the LBMA Market Makers, then relayed to the dealing rooms where customers can express their interest as buyers or sellers and also the quantity they wish to trade. The gold fixing price is then set by collating bids and offers until the supply and demand are matched. At this point the price is announced as the 'Fixed' price for gold and all business is conducted on the basis of that price.https://www.quandl.com/data/LBMA/GOLD-Gold-Price-London-Fixing<jupyter_code>data = pd.read_csv('gold.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Crop production (Wheat)Crop production depends on the availability of arable land and is affected in particular by yields, macroeconomic uncertainty, as well as consumption patterns; it also has a great incidence on agricultural commodities' prices. The importance of crop production is related to harvested areas, returns per hectare (yields) and quantities produced. Crop yields are the harvested production per unit of harvested area for crop products. In most of the cases yield data are not recorded, but are obtained by dividing the production data by the data on area harvested. The actual yield that is captured on farm depends on several factors such as the crop's genetic potential, the amount of sunlight, water and nutrients absorbed by the crop, the presence of weeds and pests. This indicator is presented for wheat, maize, rice and soybean. Crop production is measured in tonnes per hectare, in thousand hectares and thousand tonnes.https://data.oecd.org/agroutput/crop-production.htm<jupyter_code>data = pd.read_csv('weat.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Share pricesShare price indices are calculated from the prices of common shares of companies traded on national or foreign stock exchanges. They are usually determined by the stock exchange, using the closing daily values for the monthly data, and normally expressed as simple arithmetic averages of the daily data. A share price index measures how the value of the stocks in the index is changing, a share return index tells the investor what their “return” is, meaning how much money they would make as a result of investing in that basket of shares. A price index measures changes in the market capitalisation of the basket of shares in the index whereas a return index adds on to the price index the value of dividend payments, assuming they are re-invested in the same stocks. Occasionally agencies such as central banks will compile share indices.https://data.oecd.org/price/share-prices.htm<jupyter_code>data = pd.read_csv('share_prices.csv')
data.head()
data = data[data['LOCATION'] == 'ZAF'][['TIME', 'Value']]
data.columns = ['time', 'share_prices']
data.to_csv('share_prices.csv', index=False)<jupyter_output><empty_output><jupyter_text>Investment (GFCF)Gross fixed capital formation (GFCF) is defined as the acquisition (including purchases of new or second-hand assets) and creation of assets by producers for their own use, minus disposals of produced fixed assets. The relevant assets relate to products that are intended for use in the production of other goods and services for a period of more than a year. The term "produced assets" means that only those assets that come into existence as a result of a production process recognised in the national accounts are included. This indicator is in million USD at current prices and PPPs, and in annual growth rates. All OECD countries compile their data according to the 2008 System of National Accounts (SNA).https://data.oecd.org/gdp/investment-gfcf.htm<jupyter_code>data = pd.read_csv('investment_gfcf.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Gross domestic product (GDP)Gross domestic product (GDP) at market prices is the expenditure on final goods and services minus imports: final consumption expenditures, gross capital formation, and exports less imports. "Gross" signifies that no deduction has been made for the depreciation of machinery, buildings and other capital products used in production. "Domestic" means that it is production by the resident institutional units of the country. The products refer to final goods and services, that is, those that are purchased, imputed or otherwise, as: final consumption of households, non-profit institutions serving households and government; fixed assets; and exports (minus imports). GDP per capita data are measured in US dollars at current prices and PPPs. GDP data are measured in million US dollars at current prices and PPPs. Data are internationally comparable across countries for any single year. All OECD countries compile their data according to the 2008 System of National Accounts (SNA).https://data.oecd.org/gdp/gross-domestic-product-gdp.htm<jupyter_code>data = pd.read_csv('gdp.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Current account balanceThe current account balance of payments is a record of a country's international transactions with the rest of the world. The current account includes all the transactions (other than those in financial items) that involve economic values and occur between resident and non-resident entities. Also covered are offsets to current economic values provided or acquired without a quid pro quo. This indicator is measured in million USD and percentage of GDP.https://data.oecd.org/trade/current-account-balance.htm<jupyter_code>data = pd.read_csv('current_account_balance.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Exchange Rates (USD/ZAR)Exchange rates are defined as the price of one country's' currency in relation to another. Exchange rates may be expressed as the average rate for a period of time or as the rate at the end of the period. Exchange rates are classified by the International Monetary Fund in three broad categories, reflecting the role of the authorities in the determination of the exchange rates and/or the multiplicity of exchange rates in a country: the market rate, in which the rate "floats" and is largely set by market forces; the official rate, in which the rate is "fixed" by a country's authorities; and arrangements falling between the two, in which the rate holds a stable value against another currency or a composite of currencies. This indicator is measured in terms of national currency per US dollar.https://data.oecd.org/conversion/exchange-rates.htm<jupyter_code>data = pd.read_csv('exchange_rate_2.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Prime EnergyPrimary energy supply is defined as energy production plus energy imports, minus energy exports, minus international bunkers, then plus or minus stock changes. The International Energy Agency (IEA) energy balance methodology is based on the calorific content of the energy commodities and a common unit of account: tonne of oil equivalent (toe). Toe is defined as 107 kilocalories (41.868 gigajoules). This quantity of energy is, within a few per cent, equal to the net heat content of one tonne of crude oil. The difference between the “net” and the “gross” calorific value for each fuel is the latent heat of vaporisation of the water produced during combustion of the fuel. For coal and oil, net calorific value is about 5% less than gross, for most forms of natural and manufactured gas the difference is 9-10%, while for electricity the concept of calorific has no meaning. The IEA calculates balances using the physical energy content method to find the primary energy equivalent. This indicator is measured in million toe and in toe per 1 000 USD.https://data.oecd.org/energy/primary-energy-supply.htm<jupyter_code>data = pd.read_csv('prime_energy.csv')
data.head()<jupyter_output><empty_output><jupyter_text>CPI (Inflation)Inflation measured by consumer price index (CPI) is defined as the change in the prices of a basket of goods and services that are typically purchased by specific groups of households. Inflation is measured in terms of the annual growth rate and in index, 2015 base year with a breakdown for food, energy and total excluding food and energy. Inflation measures the erosion of living standards. A consumer price index is estimated as a series of summary measures of the period-to-period proportional change in the prices of a fixed set of consumer goods and services of constant quantity and characteristics, acquired, used or paid for by the reference population. Each summary measure is constructed as a weighted average of a large number of elementary aggregate indices. Each of the elementary aggregate indices is estimated using a sample of prices for a defined set of goods and services obtained in, or by residents of, a specific region from a given set of outlets or other sources of consumption goods and services.https://data.oecd.org/price/inflation-cpi.htm<jupyter_code>data = pd.read_csv('cpi.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Long-term interest ratesLong-term interest rates refer to government bonds maturing in ten years. Rates are mainly determined by the price charged by the lender, the risk from the borrower and the fall in the capital value. Long-term interest rates are generally averages of daily rates, measured as a percentage. These interest rates are implied by the prices at which the government bonds are traded on financial markets, not the interest rates at which the loans were issued. In all cases, they refer to bonds whose capital repayment is guaranteed by governments. Long-term interest rates are one of the determinants of business investment. Low long-term interest rates encourage investment in new equipment and high interest rates discourage it. Investment is, in turn, a major source of economic growth.https://data.oecd.org/interest/long-term-interest-rates.htmindicator-chart<jupyter_code>data = pd.read_csv('long_term_interest.csv')
data.head()<jupyter_output><empty_output><jupyter_text>M3 Money SupplyBroad money (M3) includes currency, deposits with an agreed maturity of up to two years, deposits redeemable at notice of up to three months and repurchase agreements, money market fund shares/units and debt securities up to two years. M3 is measured as a seasonally adjusted index based on 2015=100https://data.oecd.org/money/broad-money-m3.htm<jupyter_code>data = pd.read_csv('m3_supply.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Unemployment, total (% of total labor force) (modeled ILO estimate)Unemployment refers to the share of the labor force that is without work but available for and seeking employment.Source: https://databank.worldbank.org/data/reports.aspx?source=2&series=SL.UEM.TOTL.ZS<jupyter_code>unemployment = pd.read_csv('unemployment.csv')
unemployment.head()<jupyter_output><empty_output><jupyter_text>--- Concatenate data<jupyter_code>import os
file_names = ['cpi.csv',
 'ores_metal_imports.csv',
 'gold.csv',
 'prime_energy.csv',
 'exchange_rate.csv',
 'share_prices.csv',
 'disposable_income.csv',
 'long_term_interest.csv',
 'm3_supply.csv',
 'unemployment.csv',
 'investment_gfcf.csv',
 'current_account_balance.csv',
 'gdp.csv']
store = []
for name in file_names:
    store.append(pd.read_csv(name, index_col=0))
data = pd.concat(store, axis=1)
# Subset by cutting out data with NaNs
data = data.iloc[3:-1, :]
data = data.drop('disposable_income', axis=1)
# Back fill data
data = data.bfill()
data<jupyter_output><empty_output><jupyter_text>Take the log returns of the following features:* Gold* Exchange Rate* Share Prices* Long term interest rates* M3 Money supply* Unemployment* GDPApply Min Max Scaling to these:* CPI* Ores & metals* Prime Energy* Investment_gfcf* Current_account_balance<jupyter_code>log_names = [' gold', 'exchange_rate', 'share_prices', 'long_term_interest_rate', 'm3', 'unemployment','gdp']
other_names = ['current_account_balance', 'cpi', 'ores_metal_imports', 'prime_energy', 'investment_gfcf', 'current_account_balance']
logged_data = np.log(data[log_names]) - np.log(data[log_names].shift(1))
# Apply min max scaling
min_max_data = pd.DataFrame(pre.minmax_scale(data[other_names], axis=0), index=data[other_names].index)
# Add column names
min_max_data.columns = other_names
# Append data
final_data = pd.concat([logged_data, min_max_data], axis=1).iloc[1:, :]
# Save DF
final_data.to_csv('final_data.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06412114186349804, 'fraction_numerical': 0.004709462396754093, 'mean_word_length': 5.623320537428023, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 13, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '453245', 'license': 'MIT', 'n_tokens_mistral': 3727, 'n_tokens_neox': 3519, 'n_words': 1723, 'path': 'Notebooks/Getting and Cleaning Data.ipynb', 'repo_name': 'Jackal08/sa_risk_management', 'size': '13802'}