<jupyter_start><jupyter_text>Ease of Replication StrategyPython, sourcing data, ETL constrains and logic.  Nuclear & Hydroelectric Energy Dummy Variables Country Sorting AlgorithmIn our ETL process (Extract, Transform, Load) we are working on the premise that certain countries have better policies, cultural attitude towards alternative energy that make them more likely to have a turning point in the EKC.  If the goal is CO2 emmisions, other countries will seek proof of the existence of such policies to effectively enable such turning point, as every nation wishes to both maximize wealth and minize emmissions.  Thus, in spite of the potential critisism that this is an exercise in "fitting the data to the model", we feel it has value in a proper test of the existence of the EKC, as most academic literature to this date has failed to do this, either to poor selection of explanatory variables, or simply because the data available was too sparse or inaccurate.   Data on wind power generation in Megawatts is available from Eurostat from 1990 to 2018, but only for very few countries. For some countries the data from early years is strangely replicated identically across years, almost as if a Windmill was built and remained the only unit in operation, then suddenly there is a ramp up. This is a suspicius pattern indeed, for example:`1991,"France","1.000"1992,"France","1.000"1993,"France","3.000"1994,"France","3.000"1995,"France","3.000"`We suspect that someone would really require historical domain expertise to enterprit such numbers as errouneous or valid. In analyzing EKC relationships this data may prove useful, especially if it can be tracked in Kilowatt per Capita by dividing yearly capacity by population for each country.  The difficulty, however, lies in the fact that this data is highly suspect.   This section will outline what problems exist, and various potential ways to overcome it. I have experimented exstensively with data cleanup and manipulation in Stata, Excel with Visual Basic, R, and Python.  This section is meant to provide instructions to anyone who wishes to replicate this paper for future validation as more wind power data continues to become availabe in the future. This paper is specifically biased towards the immense power and flexibility of Python to deal with this type of data cleaning process, while pointing out some perculiar drawbacks of it's nearest rival R.  Technical Preliminariest an integrated set of data structures andtools providing this functionality. Processing raw data from EurostatRaw wind data from Eurostat can be downloaded from all European countries in the following format:<jupyter_code>from pandas import read_csv
df = read_csv('data/nrg_inf_epcrw_1_Data.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Drop Useless Columns & Rename Columns using PipelinesSee [this article from Nov 2019](https://towardsdatascience.com/https-medium-com-tirthajyoti-build-pipelines-with-pandas-using-pdpipe-cade6128cd31)`pipeline = pdp.ColDrop(‘Avg. Area House Age’)pipeline+= pdp.OneHotEncode(‘House_size’)df3 = pipeline(df)`Net maximum electrical capacity 	Megawatt 	is repeated but should be in the metadata.<jupyter_code>def price_tag(x):
    if x>250000:
        return 'keep'
    else:
        return 'drop'

    
#value_map = {"Germany (until 1990 former territory of the FRG)": 'Germany'}
#! pip install pdpipe
import pdpipe as pdp
#  Creat some tasks without executing them
pipeline = pdp.ColDrop(['SIEC', 'PLANT_TEC', 'UNIT'])
pipeline+=pdp.RowDrop({'Value': lambda x: x ==':'})
pipeline+=pdp.RowDrop({'GEO': lambda x: str(x).find('Euro')==0}) # Find method finds the first Occurance place in string
pipeline+=pdp.RegexReplace('GEO', r'\(.*\)', "") # Remove parenthese and everything in between
#pipeline+=pdp.MapColVals('GEO', value_map)
# Now send add this tasks in a pipeline
df = pipeline(df)

#data = data.drop(columns=['SIEC', 'PLANT_TEC', 'UNIT'])
#data.pd.rename(columns = {'test':'TEST'}, inplace = True) 
df.head(300)<jupyter_output><empty_output><jupyter_text>By immediate visual inspection it's obvious that much cleanup is required to make it useful for regression analysis.   Purge Useless DataGeEurostat has certain oddities that need to be removed:`"Germany (until 1990 former territory of the FRG)""Kosovo (under United Nations Security Council Resolution 1244/99)"`should both be chaned to "Germany" and "Kosovo".  It's odd that Eurostat does not include international ISO country codes in the data, as this is essential to merge  panel data sets various sources using universal country codes and years.<jupyter_code>df = (
df
# clean column names
.rename(columns={'TIME': 'Year', 'GEO': 'Country', 'Value':'Megawatts'})
.dropna()
)
df.head()
x = "European String"

print(x.find('European String'))<jupyter_output>0
{'fraction_non_alphanumeric': 0.07550300767475628, 'fraction_numerical': 0.017838622692387472, 'mean_word_length': 5.455153949129853, 'pattern_counts': {'":': 1, '<': 12, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '57801', 'license': 'MIT', 'n_tokens_mistral': 1381, 'n_tokens_neox': 1267, 'n_words': 613, 'path': 'EKC-CountrySort.ipynb', 'repo_name': 'pjakobsen/ekc-stats', 'size': '4821'}