<jupyter_start><jupyter_text>Table of ContentsSimulation SetupTOUPV system (deterministic)Outdoor Temperature (deterministic)Formulation of RL and ANN-Based Home Energy Management[State]WM (Washing Machine)AC (air conditioner)ESS (energy storage system)[Action]WM (Washing Machine)AC (air conditioner)ESS (energy storage system)[Reward]WM (Washing Machine)AC (air conditioner)ESS (energy storage system)Environment codeAgent codeVisulaizationAgent learnAC - [Figure 9] RL-based day-ahead energy consumption schedule of air conditioner through indoor temperature prediction using:(a) equivalent thermal parameters (ETP) modelWM - [Figure 8] Impact of different preferred operating time interval [$\omega^{pref}_s, \omega^{pref}_f$] for WM scheduling on energy consumption ($E_{a,t}$).(a) (06:00 a.m., 10:00 p.m.)(b) (12:00 a.m., 10:00 p.m.)ESS - [Figure] charging and discharging of energy storage system (ESS);Reference<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Simulation Setup  TOU<jupyter_code>TOU = np.array([0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 
                0.06, 0.06, 0.12, 0.14, 0.14, 0.12,
                0.14, 0.14, 0.14, 0.14, 0.12, 0.12,
                0.12, 0.12, 0.12, 0.12, 0.06, 0.06,                      
                ])
plt.plot(TOU, 'r-o')

plt.xlim(0,23)
plt.ylim(0, 0.2)

plt.xticks(np.arange(24))

plt.title("time-of-use (TOU) price")
plt.xlabel("Time (hour)")
plt.ylabel("Price ($/Wh)")

plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>PV system (deterministic)<jupyter_code>pv_system = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 250, 
                      500, 1000, 1400, 1750, 2000, 1750,
                      1400, 1000, 500, 250, 0.0, 0.0,
                      0.0, 0.0, 0.0, 0.0, 0.0, 0.0                    
                     ])
plt.bar(np.arange(24), pv_system)

plt.xlim(0,23)
plt.ylim(0, 2250)

plt.xticks(np.arange(24))

plt.title("solar photovoltaic (PV)")
plt.xlabel("Time (hour)")
plt.ylabel("Energy Generation (Wh)")

plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>Outdoor Temperature (deterministic)<jupyter_code>outdoor_temperature = np.array([23, 23, 23, 23, 24, 24, 
                                27, 28, 28, 31, 31, 32,
                                33, 33, 32, 32, 30, 30,
                                29, 27, 27, 26, 25, 24                    
                                ])
plt.plot(outdoor_temperature, 'b-o')

plt.xlim(0,23)
plt.ylim(21, 36)

plt.xticks(np.arange(24))

plt.title("outdoor temperature")
plt.xlabel("Time (hour)")
plt.ylabel("Temperature ($C^{\cdot}$)")

plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>Formulation of RL and ANN-Based Home Energy Management  [State]  WM (Washing Machine) $$ \mathcal{S}^{\mathrm{WM}}=\left\{E_{t}^{\mathrm{WM}}\right\} , \quad \forall t=1, \ldots, 24 $$   AC (air conditioner) $$ \mathcal{S}^{\mathrm{AC}}=\left\{E_{t}^{\mathrm{AC}}\right\} , \quad \forall t=1, \ldots, 24 $$   ESS (energy storage system) $$\mathcal{S}^{\mathrm{ESS}}=\left\{S O E_{t}^{\mathrm{ESS}}\right\}, \quad \forall t=1, \ldots, 24$$  [Action]  WM (Washing Machine) $$\begin{aligned}\mathcal{A}^{\mathrm{WM}} &=\{\mathrm{On}, \mathrm{Off}\}\end{aligned}$$  AC (air conditioner) $$\begin{aligned}\mathcal{A}^{\mathrm{AC}} &=\left\{0, \Delta E^{\mathrm{AC}}, 2 \Delta E^{\mathrm{AC}}, \ldots, 8 \Delta E^{\mathrm{AC}}, 9 \Delta E^{\mathrm{AC}}\right\}\end{aligned}$$  ESS (energy storage system) $$\begin{aligned}\mathcal{A}^{\mathrm{ESS}} &=\left\{-4 \Delta E^{\mathrm{ESS}},-3 \Delta E^{\mathrm{ESS}},-2 \Delta E^{\mathrm{ESS}},-1 \Delta E^{\mathrm{ESS}}, 0,1 \Delta E^{\mathrm{ESS}}, 2 \Delta E^{\mathrm{ESS}}, 3 \Delta E^{\mathrm{ESS}}, 4 \Delta E^{\mathrm{ESS}}\right\}\end{aligned}$$  [Reward] $$ r^{\text{Total}} = r_t^{\text{WM}} + r_t^{\text{AC}} + r_t^{\text{ESS}}$$  WM (Washing Machine)- $\pi_{t}$ : TOU price at time slot $t$- $E_{t}^{WM}$ : the energy consumption of the WM at time slot $t$- $\omega_{s}^{\text{pref}}$ : the consumer preferred starting times of WM- $\omega_{f}^{\text{pref}}$ : the consumer preferred  finishing times of WM- $\overline{\delta}$ : the penalties for early operation of WM- $\underline{\delta}$ : the penalties for late operation of WM$$ r_{t}^{\mathrm{WM}}=\left\{\begin{array}{ll}-\left[\pi_{t} E_{t}^{\mathrm{WM}}+\bar{\delta}\left(\omega_{s}^{\text {pref}}-t\right)\right], & \text { if } t<\omega_{s}^{\text {pref}} \\-\left[\pi_{t} E_{t}^{\mathrm{WM}}+\underline{\delta}\left(t-\omega_{f}^{\text {pref}}\right)\right], & \text { if } t>\omega_{f}^{\text {pref}} \\-\pi_{t} E_{t}^{\mathrm{WM}}, & \text { otherwise }\end{array}\right. \tag{26}\label{26}$$  AC (air conditioner)- $\pi_{t}$ : TOU price at time slot $t$- $E_{t}^{AC}$ : the energy consumption of the AC at time slot $t$- $T^{min}$ : the consumer preferred min temperature : (23)- $T^{max}$ : the consumer preferred max temperature : (25)- $T_{t}^{in}$ : in-door temperature- $\kappa$ : the penalty for the consumer thermal discomfort : (10, 50, 100)$$r_{t}^{\mathrm{AC}}=\left\{\begin{array}{ll}-\left[\pi_{t} E_{t}^{\mathrm{AC}}+\kappa\left(T^{\min }-T_{t}^{\mathrm{in}}\right)\right], & \text { if } T_{t}^{\mathrm{in}}<T^{\mathrm{min}} \\-\left[\pi_{t} E_{t}^{\mathrm{AC}}+\kappa\left(T_{t}^{\mathrm{in}}-T^{\max }\right)\right], & \text { if } T_{t}^{\mathrm{in}}>T^{\mathrm{max}} \\-\pi_{t} E_{t}^{\mathrm{AC}}, & \text { otherwise }\end{array}\right. \tag{27}\label{27}$$  ESS (energy storage system)- $\pi_{t}$ : TOU price at time slot $t$- $E_{t}^{ESS}$ : the energy consumption of the ESS at time slot $t$- $SOE_t$ : State of energy of ESS at time slot $t$- $SOE^{max}$ : Maximum state of energy of ESS- $SOE^{min}$ : Minimum state of energy of ESS- $\overline{\tau}$ : the penalties for overcharging of ESS- $\underline{\tau}$ : the penalties for undercharging of ESS$$r_{t}^{\mathrm{ESS}}=\left\{\begin{array}{ll}-\left[\pi_{t} E_{t}^{\mathrm{ESS}}+\bar{\tau}\left(S O E_{t}-S O E^{\max }\right)\right], & \text { if } S O E_{t}>S O E^{\max } \\-\left[\pi_{t} E_{t}^{\mathrm{ESS}}+\underline{\tau}\left(S O E^{\mathrm{min}}-S O E_{t}\right)\right], & \text { if } S O E_{t}<S O E^{\mathrm{min}} \\-\pi_{l} E_{t}^{\mathrm{ESS}}, & \text { otherwise }\end{array}\right. \tag{28}\label{28}$$  Environment code<jupyter_code># A task/simulation which needs to be solved by the Agent.
class Environment():
    """
    가전기기 환경
    """
    
    def __init__(self, state_dim=24):
        # State Space (the energy consumption of time unit)
        self.state_dim = state_dim
        self.AC_states = {i:0 for i in range(self.state_dim)}
        self.WM_states = {i:0 for i in range(self.state_dim)}
        self.ESS_states = {i:2400 if i==0 else 0 for i in range(self.state_dim)}
        
        # TOU
        self.TOU = np.array([0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 
                             0.06, 0.06, 0.12, 0.14, 0.14, 0.12,
                             0.14, 0.14, 0.14, 0.14, 0.12, 0.12,
                             0.12, 0.12, 0.12, 0.12, 0.06, 0.06,                      
                             ])

        # PV
        self.pv_system = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 250, 
                                   500, 1000, 1400, 1750, 2000, 1750,
                                   1400, 1000, 500, 250, 0.0, 0.0,
                                   0.0, 0.0, 0.0, 0.0, 0.0, 0.0                    
                                  ])
        
        # outdoor temperature
        self.outdoor_temperature = np.array([23, 23, 23, 23, 24, 24, 
                                             27, 28, 28, 31, 31, 32,
                                             33, 33, 32, 32, 30, 30,
                                             29, 27, 27, 26, 25, 24                    
                                             ])
        
        # indoor temperature
        self.indoor_temperature = np.zeros(24)
        self.indoor_temperature[-1] = 22    # 실내온도 초기값 (임의로 설정) arbitrary initial value
        
        # 소비 전력에 따른 온도 변화 (가정)
        # self.temp_change_by_power_consumption = {i:round((i/1390*4700)/(1.3*0.242*800),2) for i in np.arange(10)*40}

    def reset(self):
        """ Environment states reset
        Args:
            None
        Returns:
            state
        """
        self.AC_states = {i:0 for i in range(self.state_dim)}
        self.WM_states = {i:0 for i in range(self.state_dim)}
        self.ESS_states = {i:2400 if i==0 else 0 for i in range(self.state_dim)}
        
        self.indoor_temperature = np.zeros(24)
        self.indoor_temperature[-1] = 22
        
        return list(env.AC_states.items())[0], list(env.WM_states.items())[0], list(env.ESS_states.items())[0]
        
        
    def AC_step(self, time, action, temp_pref=[23, 25]):
        """
        Args:
            time : 시간
            action : 행동
            temp_pref :선호 온도 [최저, 최대]
            
        Returns:
            next_state, reward, done
        """
        # the penalty for the consumer thermal discomfort
        kappa = 50
        
        alpha = 0.8
        beta = -0.02
        
        self.AC_states[time] = action
        
        self.indoor_temperature[time] = (self.indoor_temperature[time-1] + 
                                         alpha * (self.outdoor_temperature[time-1] - self.indoor_temperature[time-1]) + 
                                         beta * action) # - self.temp_change_by_power_consumption[action])
        
        # reward
        if self.indoor_temperature[time] < temp_pref[0]:
            reward = - (self.TOU[time]*action + kappa*(temp_pref[0]-self.indoor_temperature[time]))
        
        elif self.indoor_temperature[time] > temp_pref[1]:
            reward = - (self.TOU[time]*action + kappa*(self.indoor_temperature[time]-temp_pref[1]))
        else :
            reward = self.TOU[time]*action

        # next_state, done
        if time == 23:
            done = True
            next_state = None
        else :
            done = False
            next_state = list(env.AC_states.items())[time+1]
        
        return next_state, reward, done
    
    
    def WM_step(self, time, action, WM_pref = [6, 22]):
        """
        Args:
            time : 시간
            action : 행동
            WM_pref :선호 시간 [시작, 종료]
            
        Returns:
            reward
        """
        # 세타긱 소비전력 (e.g. 500W)
        WM_power = 500
        
        # penalty
        early_penalty = 50
        late_penalty = 50
        discontinuous_penalty = 100
        
        start_time = time
        end_time = time + 1
        
        reward = 0
        
        
        if action == 1:
            
            self.WM_states[time] = WM_power
            
            # distance = (time - [i for i in [i for i in self.WM_states.values()]].index(500))
            
            if start_time < WM_pref[0]:
                reward = -(self.TOU[start_time] * WM_power + early_penalty*(WM_pref[0]-start_time))
            elif end_time > WM_pref[1]:
                # reward = -(self.TOU[start_time] * WM_power + late_penalty*(start_time - WM_pref[1]))
                reward = -(self.TOU[start_time] * WM_power + late_penalty*(end_time - WM_pref[1]))
            else:
                reward = -(self.TOU[start_time] * WM_power)
            
            # next_state, done
            if time == 23:
                done = True
                next_state = None
            else :
                done = False
                next_state = list(env.WM_states.items())[time+1]   

            return next_state, reward, done
            
        else :
            
            self.WM_states[time] = 0
            
            if [i for i in [i for i in self.WM_states.values()]][:time].count(500) > 2:
                reward = - 50
            elif [i for i in [i for i in self.WM_states.values()]][:time].count(500) < 2:
                reward = - 50
            else:
                reward = 0
            
            # next_state, done
            if time == 23:
                done = True
                next_state = None
            else :
                done = False
                next_state = list(env.WM_states.items())[time+1]      

            return next_state, reward, done
    
    
    def ESS_step(self, time, action, min_max_SOE = [800, 4000]):
        """
        Args:
            time : 시간
            action : 행동
            min_max_ESS : minimum, and maximum SOE values [최소, 최대]
            
        Returns:
            reward
        """
        # the penalties for the ESS overcharging and undercharging
        overcharging_penalty = 50
        undercharging_penalty = 50
        
        # Q1) ESS 충전 및 사용 시점  
        self.ESS_states[time+1] = (self.ESS_states[time] + self.pv_system[time] + action
                                   -self.AC_states[time] - self.WM_states[time])
        
        temp_SOE = self.ESS_states[time]
        
        if temp_SOE > min_max_SOE[1]:
            reward = - (self.TOU[time]*action + overcharging_penalty*(temp_SOE-min_max_SOE[1]))
        
        elif temp_SOE < min_max_SOE[0]:
            reward = - (self.TOU[time]*action + undercharging_penalty*(min_max_SOE[0]-temp_SOE))
            
        else :
            reward = self.TOU[time]*action
        
        # next_state, done
        if time == 23:
            done = True
            next_state = (24, 0)
        else :
            done = False
            next_state = list(env.ESS_states.items())[time+1]
            
        return next_state, reward, done<jupyter_output><empty_output><jupyter_text>Agent code<jupyter_code>class QLearning_Agent():
    """ 가전기기 Agent
    Args:
        state_dim: states
            e.g.: 24
        action_sapce: actions
            e.g.: 
            AC_actions = np.arange(10)*40     # delta E = 40 Wh
            WM_actions = np.array([0, 1])     # o : off, 1 : on
            ESS_actions = np.arange(-4,5)*150 # delta E = 150 Wh         
    """
    
    def __init__(self, state_dim=24, action_sapce=np.arange(10)*40):
        self.epsilon = 0.1
        self.learning_rate = 0.1
        self.discount_factor = 0.9
        
        # Q Table
        self.q_table = np.zeros([state_dim, len(action_sapce)])

        # Action Space
        self.action = action_sapce
    
    # <s, a, r, s'> 샘플로부터 큐함수를 업데이트
    def learn(self, state, action, reward, next_state):
        """ q table update of agent
        Args:
            state: time
            action: 
            reward: 
            next_state: time + 1   
        """
        action = self.action.tolist().index(action)
        
        q_value = self.q_table[state][action]
        
        # 벨만 최적 방정식을 사용한 큐함수의 업데이트
        if next_state == 23:
            q_new = reward
        else:
            q_new = reward + self.discount_factor * max(self.q_table[next_state])
        
        self.q_table[state][action] = (1-self.learning_rate)*q_value + self.learning_rate*q_new
    
    # 큐함수에 대한 입실론 탐욕 정책에 따라 행동을 반환
    def get_action_with_e_greedy(self, state):
        """ random action according to epsilon
        Args:
            state: time
        Retruns:
            action
        """
        p = np.random.random()  # 0 ~ 1의 범위를 가지는 uniform distribution에서 p 추출
        if p < self.epsilon:
            # 무작위 행동 반환
            action = np.random.choice(self.action)
            return action
        else:
            
            # 큐함수에 따른 행동 반환
            state_action = self.q_table[state]
            action = np.argmax(state_action)
            action = self.action[action]
            return action<jupyter_output><empty_output><jupyter_text>Visulaization  Agent learn<jupyter_code>env = Environment()

# AC agent (delta E = 40 Wh)
AC_agent = QLearning_Agent(state_dim = 24, action_sapce = np.arange(10)*40)
# WM agent (o : off, 1 : on)
WM_agent = QLearning_Agent(state_dim = 24, action_sapce = np.array([0, 1]))
# ESS agent (# delta E = 150 Wh)
ESS_agent = QLearning_Agent(state_dim = 24, action_sapce = np.arange(-4,5)*150)

for episode in range(2000):
    # 환경과 상태를 초기화
    AC_state, WM_state, ESS_state = env.reset() # ((0, 0), (0, 0), (0, 2400))
    
    AC_state = AC_state[0]
    WM_state = WM_state[0]
    ESS_state = ESS_state[0]
            
    # 0 ~ 23 까지 순차적으로 step 진행
    while True:
        
        # 현재 상태에 대한 행동 선택
        AC_action = AC_agent.get_action_with_e_greedy(state = AC_state)
        WM_action = WM_agent.get_action_with_e_greedy(state = WM_state)
        ESS_action = ESS_agent.get_action_with_e_greedy(state = ESS_state)
        
        # 행동을 취한 후 환경에서 한 타임스텝 진행
        AC_next_state, AC_reward, AC_done = env.AC_step(time = AC_state, action = AC_action)
        WM_next_state, WM_reward, WM_done = env.WM_step(time = WM_state, action = WM_action)
        ESS_next_state, ESS_reward, ESS_done = env.ESS_step(time = ESS_state, action = ESS_action)
        
        reward = AC_reward + WM_reward + ESS_reward
        
        if AC_done & WM_done & ESS_done:
            break

        # <s, a, r, s'> 로 큐함수를 업데이트
        AC_agent.learn(state=AC_state, action=AC_action, reward=reward, next_state=AC_next_state[0])
        WM_agent.learn(state=WM_state, action=WM_action, reward=reward, next_state=WM_next_state[0])
        ESS_agent.learn(state=ESS_state, action=ESS_action, reward=reward, next_state=ESS_next_state[0])
            
        AC_state = AC_next_state[0]
        WM_state = WM_next_state[0]
        ESS_state = ESS_next_state[0]
f, ax1 = plt.subplots(figsize=(10,7))
f.patch.set_facecolor('w')

ax1.bar(range(24), [AC_agent.action[i] for i in np.argmax(AC_agent.q_table, axis=1)], color='b', label='Energy Comsumption')
ax1.set_xlabel("Time(hour)")
ax1.set_ylabel("Energy consumption (Wh)")
ax1.set_ylim(0,500)

ax2 = ax1.twinx()
ax2.bar(0, 0, color='b', label='Energy Comsumption')
ax2.plot(env.outdoor_temperature, 'r--s', label='Outdoor Temp')
ax2.plot(env.indoor_temperature, 'g-o', label='Indoor Temp')
ax2.plot([23]*24, 'r:')
ax2.plot([25]*24, 'r:', label='Consumer Preferred Temp Range')


ax2.set_xlim(-1,24)
ax2.set_ylim(min(env.indoor_temperature)-2,max(env.outdoor_temperature)+5)
ax2.set_xticks(np.arange(24))
ax2.set_xlabel("Time (hour)")
ax2.set_ylabel("Temperature ($^{\cdot}C$)")

plt.grid(True)
plt.legend(loc='upper left')
plt.show()<jupyter_output><empty_output><jupyter_text>AC - [Figure 9] RL-based day-ahead energy consumption schedule of air conditioner through indoor temperature prediction using:  (a) equivalent thermal parameters (ETP) model<jupyter_code>env = Environment()

# AC agent (delta E = 40 Wh)
AC_agent = QLearning_Agent(state_dim = 24, action_sapce = np.arange(10)*40)

for episode in range(8000):
    # 환경과 상태를 초기화
    AC_state, WM_state, ESS_state = env.reset() # ((0, 0), (0, 0), (0, 2400))
    AC_state = AC_state[0]
    
    # 0 ~ 23 까지 순차적으로 step 진행
    while True:
        
        # 현재 상태에 대한 행동 선택
        AC_action = AC_agent.get_action_with_e_greedy(state = AC_state)
        
        # 행동을 취한 후 환경에서 한 타임스텝 진행
        AC_next_state, AC_reward, AC_done = env.AC_step(time = AC_state, action = AC_action)
        
        reward = AC_reward
        
        if AC_done:
            break

        # <s, a, r, s'> 로 큐함수를 업데이트
        AC_agent.learn(state=AC_state, action=AC_action, reward=reward, next_state=AC_next_state[0])
            
        AC_state = AC_next_state[0]
f, ax1 = plt.subplots(figsize=(10,7))
f.patch.set_facecolor('w')

ax1.bar(range(24), [AC_agent.action[i] for i in np.argmax(AC_agent.q_table, axis=1)], color='b', label='Energy Comsumption')
ax1.set_xlabel("Time(hour)")
ax1.set_ylabel("Energy consumption (Wh)")
ax1.set_ylim(0,500)

ax2 = ax1.twinx()
ax2.bar(0, 0, color='b', label='Energy Comsumption')
ax2.plot(env.outdoor_temperature, 'r--s', label='Outdoor Temp')
ax2.plot(env.indoor_temperature, 'g-o', label='Indoor Temp')
ax2.plot([23]*24, 'r:')
ax2.plot([25]*24, 'r:', label='Consumer Preferred Temp Range')


ax2.set_xlim(-1,24)
ax2.set_ylim(min(env.indoor_temperature)-2,max(env.outdoor_temperature)+5)
ax2.set_xticks(np.arange(24))
ax2.set_xlabel("Time (hour)")
ax2.set_ylabel("Temperature ($^{\cdot}C$)")

plt.grid(True)
plt.legend(loc='upper left')
plt.show()<jupyter_output><empty_output><jupyter_text>WM - [Figure 8] Impact of different preferred operating time interval [$\omega^{pref}_s, \omega^{pref}_f$] for WM scheduling on energy consumption ($E_{a,t}$).  (a) (06:00 a.m., 10:00 p.m.)<jupyter_code>env = Environment()

# AC agent (delta E = 40 Wh)
WM_agent = QLearning_Agent(state_dim = 24, action_sapce = np.arange(2))

for episode in range(4000):
    # 환경과 상태를 초기화
    AC_state, WM_state, ESS_state = env.reset() # ((0, 0), (0, 0), (0, 2400))
    
    WM_state = WM_state[0]
            
    # 0 ~ 23 까지 순차적으로 step 진행
    while True:
        
        # 현재 상태에 대한 행동 선택
        WM_action = WM_agent.get_action_with_e_greedy(state = AC_state)
        
        # 행동을 취한 후 환경에서 한 타임스텝 진행
        WM_next_state, WM_reward, WM_done = env.WM_step(time = WM_state, action = WM_action)
        
        reward = WM_reward
        
        if WM_done:
            break
        
        # <s, a, r, s'> 로 큐함수를 업데이트
        WM_agent.learn(state=WM_state, action=WM_action, reward=reward, next_state=WM_next_state[0])
            
        WM_state = WM_next_state[0]
f, ax1 = plt.subplots(figsize=(10,7))
f.patch.set_facecolor('w')

ax1.bar(range(24), [WM_agent.action[i]*400 for i in np.argmax(WM_agent.q_table, axis=1)], color='g', label='Energy Comsumption of Washing Machine')
ax1.plot([], 'r-o', label='TOU Price')
ax1.set_xlim(-1,24)
ax1.set_ylim(0,600)
ax1.set_xticks(np.arange(24))
ax1.set_xlabel("Time (hour)")
ax1.set_ylabel("Energy Comsumption (Wh)")
ax1.legend(loc='upper right')

ax2 = ax1.twinx()
ax2.plot(env.TOU, 'r-o', label='TOU Price')
ax2.set_ylabel("Price ($/Wh)")
ax2.set_ylim(0,0.2)

plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>(b) (12:00 a.m., 10:00 p.m.)<jupyter_code>env = Environment()

# AC agent (delta E = 40 Wh)
WM_agent = QLearning_Agent(state_dim = 24, action_sapce = np.arange(2))

for episode in range(4000):
    # 환경과 상태를 초기화
    AC_state, WM_state, ESS_state = env.reset() # ((0, 0), (0, 0), (0, 2400))
    
    WM_state = WM_state[0]
            
    # 0 ~ 23 까지 순차적으로 step 진행
    while True:
        
        # 현재 상태에 대한 행동 선택
        WM_action = WM_agent.get_action_with_e_greedy(state = AC_state)
        
        # 행동을 취한 후 환경에서 한 타임스텝 진행
        WM_next_state, WM_reward, WM_done = env.WM_step(time = WM_state, action = WM_action)
        
        reward = WM_reward
        
        if WM_done:
            break
        
        # <s, a, r, s'> 로 큐함수를 업데이트
        WM_agent.learn(state=WM_state, action=WM_action, reward=reward, next_state=WM_next_state[0])
            
        WM_state = WM_next_state[0]
f, ax1 = plt.subplots(figsize=(10,7))
f.patch.set_facecolor('w')

ax1.bar(range(24), [WM_agent.action[i]*400 for i in np.argmax(WM_agent.q_table, axis=1)], color='g', label='Energy Comsumption of Washing Machine')
ax1.plot([], 'r-o', label='TOU Price')
ax1.set_xlim(-1,24)
ax1.set_ylim(0,600)
ax1.set_xticks(np.arange(24))
ax1.set_xlabel("Time (hour)")
ax1.set_ylabel("Energy Comsumption (Wh)")
ax1.legend(loc='upper right')

ax2 = ax1.twinx()
ax2.plot(env.TOU, 'r-o', label='TOU Price')
ax2.set_ylabel("Price ($/Wh)")
ax2.set_ylim(0,0.2)

plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>ESS - [Figure] charging and discharging of energy storage system (ESS);<jupyter_code>f, ax1 = plt.subplots(figsize=(10,7))
f.patch.set_facecolor('w')

ax1.bar(range(24), [ESS_agent.action[i] for i in np.argmax(ESS_agent.q_table, axis=1)], color='tan', label='Charging/Discarging of ESS')
ax1.plot([], 'r-o', label='TOU Price')
ax1.set_xlim(-1,24)
# ax1.set_ylim(0,4000)
ax1.set_xticks(np.arange(24))
ax1.set_xlabel("Time (hour)")
ax1.set_ylabel("Energy Comsumption (Wh)")
ax1.legend(loc='upper right')

ax2 = ax1.twinx()
ax2.plot(env.TOU, 'r-o', label='TOU Price')
ax2.set_ylim(0,0.2)

plt.grid(True)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.151183970856102, 'fraction_numerical': 0.04040404040404041, 'mean_word_length': 2.347235693501455, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 35, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '625154', 'license': 'MIT', 'n_tokens_mistral': 9604, 'n_tokens_neox': 9088, 'n_words': 1870, 'path': '[SGU 2021] reinforcement learning/Term Project/Home Energy Management via Q-Learning.ipynb', 'repo_name': 'DeepHaeJoong/reinforcement-learning', 'size': '24156'}