2016-01 Modifications to TOP and IRO Standards | IRO-002-5 and TOP-001-4
2016-01 Modifications to TOP and IRO Standards IRO-002-5 IN 1 ST 2016-01 Modifications to TOP and IRO Standards IRO-002-5 06/20/2016 07/19/2016 07/25/2016 08/03/2016
2016-01 Modifications to TOP and IRO Standards TOP-001-4 IN 1 ST 2016-01 Modifications to TOP and IRO Standards TOP-001-4 06/20/2016 07/19/2016 07/25/2016 08/03/2016
2016-01 Modifications to TOP and IRO Standards IRO-002-5 Non-binding Poll IN 1 NB 2016-01 Modifications to TOP and IRO Standards IRO-002-5 Non-binding Poll 06/20/2016 07/19/2016 07/25/2016 08/03/2016
2016-01 Modifications to TOP and IRO Standards TOP-001-4 Non-binding Poll IN 1 NB 2016-01 Modifications to TOP and IRO Standards TOP-001-4 Non-binding Poll 06/20/2016 07/19/2016 07/25/2016 08/03/2016
1. The SDT has developed TOP-001-4 Requirement R10 to address directives for TOP monitoring of non-BES facilities necessary for reliability. Do you agree with the proposed requirement? If you do not agree, or if you agree but have comments or suggestions for the proposed requirement provide your recommendation and explanation.
Under the plain reading of TOP-001-4 R10, when a non-BES Facility that adversely impacts reliability is identified, it essentially becomes a BES Facility ("Converted non-BES Facility", are term for purposes of these comments). As a Converted non-BES Facility, it falls within the applicability of CIP-002-5.1 (See Applicability Sec. 4.2.2.). The concern is R10 has the effect of drawing the Facility into CIP-002, which does not provide guidance as how Converted non-BES Facilities are to be characterized—High, Medium, Low Impact Cyber Assets. While an entity may be able to "fit" the Converted non-BES Facility within CIP-002 criteria to assign an impact rating, it is not ideal. The scenario muddles an entity's compliance obligation under both Standards.
Additionally, CIP-002-5.1 Applicability creates double impact criteria—where a cyber asset affects a facility and that facility affects the reliable operation of the BES. Under Project 2016-02, Modifications to CIP Standards, the SDT will address and clarify the double impact criteria issue which, in turn, will impact how Converted non-BES Facilities will be characterized.
While we can accept the TOP in R10 making the determinations and identifications, it is our belief that the Standard would better align with the objectives of other Standards by having the RC designate a non-BES facility with a capability to adversely impact the BES; pulling it into scope; and, the RC having a process to bring that facility into scope for Real Time Monitoring and Analysis.
We would respectfully ask the SDT consider the compliance implications under CIP-002, and other applicable Standards, when identifying a non-BES Facility as adversely impacting reliability, converting it to a BES Facility.
Douglas Webb, 8/3/2016
PNMR agrees with most of the proposed requirement in R10. However for those requirements with the "…identified as necessary by the Transmission Operator…" consider altering the language to "…identified as necessary by the Transmission Operator to determine System Operating Limit (SOL) exceedances…". While it might be clear when looking at the main and sub-requirement together, the sub-requirement itself is less clear and separated from the main requirement by other sub-requirments that do not have an "… as necessary…" qualifier. The proposed language change clarifies to what extent is it considered necessary and reminders the reader of the purpose of the main requirement.
Laurie Williams, 8/3/2016
Jack Stamper, Clark Public Utilities, 3, 7/14/2016
AEP recognizes FERC's concerns regarding identification of non-BES facilities, however, there would be far more flux involved in their identification and real-time monitoring (as suggested by the SAR) than may be widely understood or appreciated. This subset of non-BES facilities would change quite frequently, and creating obligations to govern such frequently changing identification and real-time monitoring would likely require much effort, with little to no improvement in reliability. Rather than developing additional requirements which would not likely be beneficial, we continue to believe a more prudent approach would be to focus on the desired end state itself. We believe the argument can still be made that our existing obligations, when considered as a whole, could collectively appease FERC's concerns.
Thomas Foltz, AEP, 5, 7/19/2016
What defines the list of facilities that are required to be telemetered and used ?
Justin Wilderness, 7/22/2016
There is already a mechnism via the BESnet tool to submit non-BES elements for inclusion. For elements that have a long term impact on the Reliability of the BES, this is the correct way to address it, not blur the lines between BES and non-BES without far more detailed guidelines to protect entities from well meaning auditors. Entities already have an obligation to respond to requests from the RC/PC/BA, this new requirement will not add any reliability that isn't already addressed.
Jim Nail, 7/25/2016
NERC already makes provision for the modification of BES Facilities through the Inclusions and Exclusions spelled out in the NERC definition of Bulk Electric System therefore Vectren believes the Requirements R10.3 and R10.6 are redundant and unnecessary. An entity may choose to monitor a non-BES facility but it shouldn't fall under a NERC requirement if it wasn't previously identified in the BES Inclusion.
Brad Lisembee, 7/25/2016
We understand that these changes are to address a FERC Directive. This is basically a fill in the blank requirement.
RF offers the following comment and modification for the SDTs consideration.
Requirement R10
The term "identified as necessary" is ambiguous and can lead to confusion in industry. For example, as written, there is no requirement for the TOP to identify "non-BES facilities" that are "necessary". In the rational section, it alludes to the fact that the TOP identifies these "necessary facilities" by performing planning and operating studies such as the Operational Planning Analysis required by TOP-002-4 Requirement R1 and IRO-008-2 Requirement R1. RF suggests replacing all the Requirement R10 sub-part language containing the phrase "identified as necessary" with the following language "identified as a result of performing planning and operating studies required by TOP-002-4 Requirement R1 and IRO-008-2 Requirement R1".
Anthony Jablonski, ReliabilityFirst , 10, 7/27/2016
John Williams, 7/28/2016
Joshua Smith, 7/29/2016
Oliver Burke, Entergy - Entergy Services, Inc., 1, 7/29/2016
David Kiguel, 8/1/2016
Dominion - RCS, Segment(s) 1, 6, 3, 5, 4/6/2015
How do you have an effective date of a procedure prior to the implementation of the system change?
Colorado Springs Utilities, Segment(s) 1, 6, 3, 5, 9/14/2015
Jaclyn Massey, 8/1/2016
Sunflower is signing on in support of ACES comments.
Paul Mehlhaff, 8/1/2016
ATC is concerned regarding requirements 10.3 and 10.6 as there is a perceived disconnect between the TOP requirement to monitor without a corresponding requirement for non-registered entities to provide requested data needed for monitoring. The standard as written requires the TOP to monitor non-BES facilities within its Transmission Operator Area. In one specific case in ATC's system, the entity who owns the facilities and thus manages the model and real time data is not a registered TOP, BA, GO, GOP, LSE, TO, or DP so they have no compliance obligation to provide the data. As good utility practice we believe they should provide the data but that's no guarantee that they will. If ATC, as the TOP, does not have the correct operating parameters, whether impedances, charging values or ratings, or we do not have the correct real-time telemetry, we cannot properly monitor the operating state of their facilities and the resulting impacts on our system. If we cannot monitor, we cannot be compliant.
Consider amending R10.3 to read as follows:
Monitor non-BES facilities within its Transmission Operator Area identified as necessary by the Transmission Operator. In those cases where sufficient modeling and real time data is not available from the facility owner and the facility owner is not required to provide said data then monitoring is not feasible and not required.
Andrew Pusztai, 8/1/2016
Michelle Amarantos, APS - Arizona Public Service Co., 5, 8/1/2016
Scott Langston, Tallahassee Electric (City of Tallahassee, FL), 1, 8/2/2016
Part 10.3 leave the question as to who shall determine the necessity to monitor non-BES facilities. Which Transmission Operator? SRP recommends address this ambiguity by adjusting the verbiage to be "Monitor non-BES facilities within its Transmission Operator Area it has identified as necessary." SRP recommends similar adjustments to parts 10.4, 10.5, and 10.6 for consistency.
There seems to be some ambiguity as to why a TOP would monitor non-BES facilities necessary for reliability versus including the less than 100 kV element as a BES element through the exception process. If the < 100 kV facility has a significant impact on the BES system it seems logical that the non-BES facility would be added to the list of BES elements for the TOP. The only reason we can surmise that a <100 kV facility would be monitored instead of added as an exception would be if the facility was outside of the TOP area, such as a generator on the distribution system or a neighboring TOP line that has a significant impact on the TOPs system. For these examples, the TOP would not have the ability to designate the <100 kV facility as BES and therefore they would only be able to monitor it in a similar manner to BES facilities. We recommend the drafting team revise the language in order to remove some ambiguity as to when a non-BES would be monitored versus added as a BES element.
Johnny Anderson, 8/2/2016
There are 2 typos:
(…)in order to perform its Real-time monitoring and Real-time Assessments as specified in the requirement
(…) in order to perform its Real-time monitoring and analysis functions as specified in the requirement.
We suggest that when you provide the rationale to TOP-001-3 at the end of the standard, you indicate the correspondence with the new (TOP-001-4) numbering of the requirements. Thus, the last paragraph would read:
Rationale for Requirements R19 and R20 (Correspond to R19, R20, R22 and R23 in TOP-001-4)
Tyson Archie, Platte River Power Authority, 5, 8/2/2016
Sergio Banuelos, On Behalf of: Tri-State G and T Association, Inc., MRO, WECC, Segments 1, 3, 5
The phrase "...identified as necessary by the Transmission Operator" leaves a large amount of latitude in determining whether non-BES facilities should be identified. More specificity on this point would improve clarity and reduce the reisk of noncompliance by TOPs.
Catrina Martin, 8/3/2016
Southern Company, Segment(s) 1, 3, 5, 6, 8/3/2016
Quintin Lee, Eversource Energy, 1, 8/3/2016
RSC, Segment(s) 1, 0, 2, 4, 5, 6, 7, 3, 8/3/2016
ITC concurs with the comments and position provided by SPP.
Stephanie Burns, 8/3/2016
Gregory DAnnibale, On Behalf of: PSEG - PSEG Energy Resources and Trade LLC, NPCC, Segments NA - Not Applicable
We understand that these changes are in direct response to a FERC Directive and neither agree nor disagree.
ISO/RTO Council Standards Review Committee, Segment(s) 2, 8/3/2016
WAPA agrees with monitoring certain identified Non-BES facilities per engineering judgement and neighbor input (especially under prior outage conditions) with the caveat that this could greatly increase the scope and workload of the TOPs and RC.
ERCOT joins the comments submitted by the IRC Standards Review Committee (SRC).
Elizabeth Axson, 8/3/2016
Texas RE is concerned there is no guidance provided for the phrase "identified as necessary" (in TOP-001-4, parts 10.3-10.6) which will result in inconsistencies by Transmission Operators in the identification of data needed for determining SOL exceedances. Texas RE recommends setting thresholds, such as an outage distribution factor for including non-BES facilities or facilities outside the TOP Area. A threshold for distribution factors for contingency outages would create a concrete target for registered entities.
Texas RE is also concerned there is no guidance for the terms "neighboring" and "adjacent", as well as no requirements for TOPs who may designate something within its own TOP Area that may affect a neighboring/adjacent TOP's Area SOL exceedance(s) (i.e., no communication requirement, no coordination requirement).
CenterPoint Energy does not agree with the language in R10.3 and recommends it be modified to more closely resemble language used in R10.6. Strictly monitoring non-BES facilities within a TOP Area will not assist in determining SOL exceedences. In order to determine SOL exceedences, information from non-BES facilities must be utilized to determine how non-BES facilities will affect SOLs. CenterPoint Energy recommends the following language:
R10.3 Utilize status, voltages, and flow data for non-BES facilities within its Transmission Operator Area identified as necessary by the Transmission Operator.
David Bueche, On Behalf of: CenterPoint Energy Houston Electric, LLC, Texas RE, Segments 1
We request the SDT to provide some rationale or guidance to what is expected of a TOP in 'identifying non-BES facilities' as being necessary. What is considered a sufficient identification process? We are not looking for a prescriptive requirement. We just request guidance. Any revisions to the rationale should also be reflected in the 'Note to Auditor' section(s) of the RSAWs.
SPP Standards Review Group, Segment(s) 0, 1, 3, 5, 8/3/2016
Matthew Beilfuss, On Behalf of: WEC Energy Group, Inc., MRO, RF, Segments 1, 3, 4, 6
We believe the SDT has significantly deviated from the expectations identified within the FERC directive, which asks for the real-time monitoring of non-BES facilities necessary to determine SOL exceedances. The guidance provided by the SDT references Operational Planning Analyses and various other requirements that are independent of this standard. The SDT has provided no defined criteria for determining what is "necessary," leaving its interpretation subjective by an Auditor. We believe it should be up to the TOP to develop its own methodology to determine what is necessary, including which non-BES facilities should be monitored and included in the pre-Contingency analyses of its Real-time Assessments; this should be reflective within the RSAW. Hence, we ask the SDT to consider using this alternative language in its place: "Monitor non-BES facilities located within its Transmission Operator Area necessary to complete pre-Contingency analyses for Real-time Assessments."
TOP-001-4 R10 is not applicable to Hydro One Networks Inc.
Oshani Pathirane, 8/3/2016
We understand that these changes are to address a FERC Directive. This is basically a fill in the blank requirement. However, clarification of "as necessary" would be appreciated.
Santee Cooper , Segment(s) 1, 3, 6, 5, 8/3/2016
The NERC Standard Drafting Team (SDT) has not completed work on the definition of the System Operating Limit (SOL) which is the cornerstone for the TOP-001. The industry has to have clear definition of SOL in order to be able to comply with the TOP-001. The industry needs the SOL definition from the SDT and before voting for TOP-001 and the additional impact of including non-BES elements.
The criteria for monitoring non-BES facilities within the TOP area is defined vaguely by using wording "identified as necessary by the Transmission Operator" . This vague definition opens a large space for interpretations and ambiguity. The criteria for monitoring non-BES facilities needs to be clearly defined. It may be inappropriate to apply a BES process to a non-BES facility, or at a minimum NERC standards need to include corresponding language for non-BES facilities that are being monitored in operations, otherwise why have the BES exception process of including non-traditional BES facilities as BES facilities.
Terry Harbour, Berkshire Hathaway Energy - MidAmerican Energy Co., 1, 8/3/2016
TOPs currently decide which Facilities need monitoring. Introducing the language "as necessary" needs to be defined if it is a change from current practice.
Sandra Shaffer, 8/3/2016
NVE has concerns that the wording of the phrase "identified as necessary by the Transmission Operator" is too vague. There is no requirement for the TOP to identify non-BES facilities as necessary or criteria for determining which non-BES facilities should be studied. The rationale section mentions that these facilities could be identified by planning and operational studies such as the Operational Planning Analysis required by TOP-002-4 Requirement 1. Based on this requirement, NVE is also concerned that the subset of non-BES facilities could change quite frequently based on the Operational Planning Analysis, creating much effort to identify and monitor frequently changing non-BES facilities. NVE feels that some guidance should be given to help identify which non-BES facilities should be monitored.
Jeffrey Watkins, 8/3/2016
2. The SDT has developed IRO-002-5 Requirement R2 and TOP-001-4 Requirements R20 and R23 to address directives for redundancy and diverse routing of RC, TOP, and BA data exchange capabilities. Do you agree with the proposed requirements? If you do not agree, or if you agree but have comments or suggestions for the proposed requirements provide your recommendation and explanation.
While we generally understand what the requirement is moving to address, there is additional clarification needed in order to understand what is representative of performance.
The reference to "within the Control Center" is specific to ensure there is no single point of failure in the data transfer supporting the BES and to ensure its availability in continuous (availability in the context of the CIA Security Triad). In addition, there is subjectivity in the exact data exchanges intended for the associated obligations.
With respect to communications and data exchanges between the RC, TOP and BA, there are relationships to many different Standards currently in force, as well as those in development. It would greatly benefit industry and the regulatory process to consider everything in flight and delineate the desired end-state for the total reliability objective in an effort to allocate the elements of the desired outcome to the appropriate places either in existing standards or development.
KCP&L agrees that the definition of critical data and validation that the appropriate data is available should be required, though, with additional clarification to what the SDT has proposed. We recommend adding these clarifications to the proposed drafted requirements and specific expectation that availability is the goal (if that is the case).
While PNMR agrees with the intent of the SDT for R20 and R23, the language needs more specificity. First, if the standard is to only apply to the primary Control Center then replace the language "…within the [Transmission Operator's | Balancing Authority's ] Control Center…" with the following language "…within the primary Control Center of a [Transmission Operator | Balancing Authority ] …." If the standard is to apply to any Control Center either primary or backup then replace the word "primary" is the suggest text with "any." In addition consider further scoping "…redundant and diversely routed data exchange infrastructure…" to include where it starts and where it ends. Does it start at the data exchange device (e.g. ICCP server, mailbox RTU) within the Control Center? Or does it start from where those devices get their data, typically an EMS or SCADA server? Or does it start from the collection of field telemetry data and thus redundant and diversely routed include the data exchange infrastructure used for field telemetry? If a beginning is not defined then it will make the standard difficult to consistently audit from Region to Region. In addition an end needs to be defined. This could be the point where the data exchange capabilities leave the Control Center. For Telco circuits this point could be defined as the demarcation (aka demarc) for the circuit.
As written, the current requrements for R20 are ambiguous as to what redundancy and diverse routing actually means. Does the redundancy and diverse routing apply equally at the Primary Contol Center and the Backup Control Center. If a utility uses its Backup Control Center as the location of its redundant and diversely routed data exchange capabilities and it is capable of tranfering system operations from its Primary Contol Center to its Backup Control Center within 2 hours as required in EOP-008, why would that not meet the FERC's directive to have "redundancy and diverse routing as stated in paragraph 47?" Instead, R20 states that such redundancy and diversity must be accomplished by infrastructure within the TOP's control center. This seems to limit the means to achieve redundancy and diversity to the specific location of the control center irrespective of other locations (i.e. backup control center) where redundancy and diversity may be acheived and done so in a more reliable manner since it exists at a facility that is geographically separate. Redundancy and diversity at one facility is not useful if that facility is not useable. That is why EOP-008 requires TOPs to have a Primary Contol Center and a Backup Control Center. The SDT for this project should not fail to take advantage of referencing redundancy and diverse routing that may have already been achieved by the implementaion of of a Backup Control Center as requried in EOP-008.
What is the definition of Redundant and Diversely routed data exchange ?
We understand that these changes are to address a FERC Directive.
We have issue with the term "diversly routed" within the Control Center.
A "backup" communication/ICCP server in a Back Up Control Center, would not meet the requirements of this standard as written. It will require a second set of "infrastructure" in the Primary Control Center.
The potential for "scope adjustment", is quite troublesome.
Does this mean cabling should be in separate cable trays or pass through separate floor penetrations to get to the exterior physical boundary of the Control Center?
If TAL puts in two of everything, but have them in the same rack, not good enough.
If we put them in separate racks on opposite sides of the room, getting there.
If we put them in separate racks on opposite sides of the room, powered by two different sources (one via UPS, the other house power) even better.
At what point do we have to have a separate room to house the alternate equipment? We do not require it for the core SCADA/EMS platforms.
While the standard leaves it up to the entity to determine what they want to do to accomplish compliance, it will ALWAYS be interpreted by an auditor that is it "diversely routed 'within' the Control Center.
The proposed requirements are changing the regulations to be a "best practice" which was not supposed to happen. We have plans and processes in place for when the RC or TOP/BA cannot monitor the equipment necessary to determine if an SOL is being exceeded, or if it is an IROL.
TAL recommends "Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure (less cabling) within the Transmission Operator's Control Center, for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities identified for data needs in order for it to perform its Real-time monitoring and Real-time Assessments."
The standard requirements say that the RC, TOp and BA must have redundant and diversely routed data exchange infrastructure "within the control center." The standard seems to mean that as soon as the data path enters the walls of the control center building then it must be on fully redundant and diversely routed path. If data is received from individual RTUs from the TOp, each of those RTUs would be required to have a redundant path into the control center. Also, it is unclear if one of those paths were to be unavailable for a certain amount of time, would the RC, TOp, or BA be non-compliant, because the redundancy is no longer available? It seems the standard should somehow account for data communicated over RTUs and not necessarily require each be fully redundant especially since the loss of one doesn't necessarily mean any significant loss of system visibility.
Does the standard require redundant and diversely routed data exchange infrastructure for all data communications or just data communication between RC, TOP, BA control centers?
In addition to the TOP-001-4 requirements included in Q2 above, Dominion believes that Requirements 19 and 22 also address the FERC directive for redunancyand diverse routing capabilities.
defer to comments by Oliver Burke of Entergy.
Agree for TOP-001-4. No comments on the IRO-002-5 standard as it does not apply to ATC directly.
2016-01_TOP-001-4_Draft-1_Question-2.docx
SRP feels these requirements could be more apporpriately addressed in a separate COM Standard.
The standard requirements say that the RC, TOP and BA must have redundant and diversely routed data exchange infrastructure "within the control center." The standard seems to mean that as soon as the data path enters the walls of the control center building then it must be on fully redundant and diversely routed paths. If data is received from individual RTUs from the TOP, each of those RTUs would be required to have a redundant path into the control center. It seems the standard should account for data communicated over RTUs and not require each be fully redundant especially since the loss of one doesn't necessarily mean any significant loss of system visibility.
Please explain in the Rationale, the difference between the redundant infrastructure in R2 and that in R6. If it is the same infrastructure, then we suggest removing the reference to the redundant infrastructure in R6.
In the R20 language, "have data exchange capabilities, with redundant and diversely routed data exchange infrastructure within the Transmission Operator's Control Center", the words "within the Transmission Operator's Control Center" are ambiguous.
The NERC Glossary states: a Control Center is, "One or more facilities hosting operating personnel that monitor and control the Bulk Electric System (BES) in real-time to perform the reliability task".
The R20 language could be interpreted to imply that each individual Control Center must have redundant and diverse data exchange routes.
Or, The R20 language could be interpreted, along with the definition, to imply that one or more Control Center facilities together must have redundant and diverse data exchange routes.
The intent is for the Transmission Operator to continue exchanging Real-time data in the event that a data route is lost. The intent is not to ensure the Transmission Operator's Control Center has a specific number of connections. To be complaint, an entity must demonstrate that the loss of a data route does not affect the exchange of Real-time data.
Platte River is suggesting that the Drafting Team update the language as follows:
R20. Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure, for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments.
It would be nice if redundant and diversely routed where defined terms.
From other comments, it seems there is stakeholder confusion about what exactly "diversely routed" means and what is expected of the applicable entities. FERC acknowledged the ambiguity in their NOPR proposing to approve the revisions to the TOP and IRO standards, and seems to favor the approach taken in developing COM-001-2 to resolve the confusion.
From Paragraph 73.
"…it is not clear whether redundancy and diverse routing of data exchange capabilities (or an equally effective alternative that eliminates the ambiguity of "redundancy" and "diversely routed") are adequately addressed in proposed Reliability Standards TOP-001-3 and IRO-002-4 for the reliability coordinator, transmission operator, and balancing authority."
FMPA believes clarity is needed either in the requirements themselves or in a defined term so that applicable entities know exactly what is expected.
Replace the phrase 'diversely routed' with another word like 'separated' or phrase like 'redundancy designed to avoid a single point of failure'.
Replace the word diverse routing with another word like "separated" or phrase like "redundancy designed to avoid a single point of failure".
In FERC Order 817 (Para. 47), NERC was directed to address "redundancy and diverse routing of data exchange capabilities" in the IRO/TOP standards. However, the SDT has duplicated this language in R20 and R23 identically. The challenge to TOPs and BAAs is to know what "diverse routing" means and how to implement it. Based upon comments from the SDT subsequent to releasing the proposed TOP-001-4 changes, it is clear that the SDT meant to assure that single point-of-failures do not compromise data exchange. Therefore, it is recommended to replace (in R20) "redundant and diversely routed data exchange infrastructure within the Transmission Operator's Control Center" with the following: "redundant data exchange infrastructure not susceptible to a single point-of-failure within the Transmission Operator's Control Center".
Duke Energy requests clarification, and further information from the drafting team on the specifics of carrying out compliance with these requirements. We suggest that a definition for redundancy and diverse routing would be helpful in aiding the industry in achieving compliance. Currently, it is unclear if the requirements call for an entity to have physically redundant hardware, redundant cabling and path, or does each entity need to establish its own definition for redundancy and diverse routing. Also, we think clarity would be improved by adding more information regarding the data exchange infrastructure aspects of the requirements and how redundancy and diversity would support the data exchange infrastructure. Ultimately, Duke Energy believes that an industry accepted definition of redundancy and diverse routing would improve understanding with the requirements, and aid entities in their implementation of said requirements.
Also, we request more information from the drafting team regarding whether the TOP area is included in the expectations outlined in R20. The requirement states that Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure with entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments.
Texas RE appreciates the efforts of the Standard Drafting Team to address the various FERC directives set forth in Order No. 817. However, Texas RE is concerned that the proposed requirement implementing FERC's directive "that the data exchange capabilities of the transmission operators and balancing authorities require redundancy and diverse routing" is overly narrow. (p. 34, ¶47). In particular, the current draft of IRO-2-5 R2 applicable to Reliability Coordinators (RCs) and TOP-001-4 R20 applicable to Transmission Operators (TOPs) specified that these functions shall have redundant and diversely routed data exchange infrastructure "within the [RC's and TOP's] Control Center."
However, the FERC directive does not contain language explicitly limiting data redundancy and diverse routing capability solely to infrastructure within an applicable entity's Control Center. Rather, FERC Order No. 817 contemplates an approach that is designed to ensure that no one event can eliminate an entity's data exchange capability. For instance, FERC drew a clear analogy between the redundancy requirements for voice communications under the COM standards and the data communication redundancy and diverse routing requirements at issue here. FERC specifically noted that "[r]edundancy for data communications is no less important than the redundancy explicitly required in the COM standards for voice communication." (p. 35, ¶48). This analogy illuminates the Control Center issue. In particular, the touchstone of the diverse routing and redundancy requirements in the COM standards is the existence of two separate and independent means for voice communication. As an example, entities may employ landline and satellite phones to satisfy the COM standards. The diverse routing and redundancy inherent in this approach in essence requires two distinct and independent events to eliminate voice communications capability. That is, the loss of phone service and the loss of satellite communications.
In contrast with this application of diverse routing and redundancy in the voice communication context, it is possible to read the IRO/TOP requirements, as currently drafted, as permitting registered entities to satisfy the redundant and diversely routed data communications requirements within a single Control Center. For example, one could argue that the data communications requirements as permit two servers served by separate cables within the Control Center, but linked to a common network point outside of the Control Center as both redundant and diversely routed within the Control Center. In such circumstances, a single event could eliminate data communications capabilities. This is in stark contrast to the layered protections created through the COM standards for voice communications and appears inconsistent with the intent underpinning the FERC directive.
Texas RE is aware of the concern that Registered Entities have regarding being held responsible for data network architecture that is outside their facilities and beyond their control. However, if the SDT wishes to address this concern by retaining the Control Center concept, Texas RE recommends at least ensuring that registered entities satisfy data communications redundancy and diverse routing requirements by using separate and independent data communications facilities located at distinct Control Centers.
CenterPoint Energy does not agree with the language in R20. Specifically, CenterPoint Energy believes options for redundancy and diverse routing of data exchange capabilities could exist outside of the Transmission Operator's Primary Control Center, and therefore, infrastructure within the TOP's Primary Control Center may not be necessary. While FERC Order 817, paragraph 47 explains that the redundancy described with Interpersonal Communications and Alternative Interpersonal Communications in COM-001-2 are not to rely on EOP-008: CenterPoint Energy does not agree this is a direct correlation to data exchange capabilities. For example, a situation could exist where remote infrastructure for data exchange capabilities can communicate and provide redundancy to a Transmission Operator Control Center where as redundant hardware has to be present at the Transmission Operator Control Center to achieve Alternate Interpersonal Communications. CenterPoint Energy suggests the following language:
R20. Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure accessible from or within the Transmission Operator's Control Center, for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments. [Violation Risk Factor: High] [Time Horizon: Same-Day Operations, Real-time Operations]
R20. Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange accessible from or utilizing infrastructure within the Transmission Operator's Control Center, for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments. [Violation Risk Factor: High] [Time Horizon: Same-Day Operations, Real-time Operations]
It would be helpful to add clarity in the Rationale that 'redundant and diversly routed' does not also require 'dual data connection links' (aka dual fiber) to each entity. Any revisions to the rationale should also be reflected in the 'Note to Auditor' section(s) of the RSAWs.
Does the requirement for redundant infrastructure also apply to data exchange capabilities housed at the backup Control Center?
We also suggest some additional rationale to clarify that loss of redundancy (loss of a single component within the Control Center infrastructure) due to a contingency and thus operating after that contingency for a period of time while the redundancy is recovered does not constitute a violation of TOP-001-4 R20. The example is loss of a network switch that must be replaced. Until it can be ordered and installed, the redundancy may not be present. How would that situation fit into the context of R20? Any revisions to the rationale should also be reflected in the 'Note to Auditor' section(s) of the RSAWs.
The proposed TOP-001-4 requirements (R22, R23, and R24) would better fit in TOP-003-Operational Reliability Data.
TOP-003-3, "R2. Each Balancing Authority shall maintain a documented specification for the data necessary for it to perform its analysis functions and Real-time monitoring" is more closely linked with the requirement proposed in TOP-001-4 R23.
(1) We ask the SDT to clarify the criteria around Transmission Operator data exchange, particularly for the performance of Real-time monitoring and Real-time Assessments. We also suggest clarifying whether the loss of redundancy (i.e. loss of a single component within the Control Center infrastructure) could constitute a violation of TOP-001-4 R20. This is especially of concern when infrastructure replacement parts may take an extended time to procure, leaving a gap in a redundant network. To address this, we suggest rephrasing the requirement to align with the format used in COM-001, such as "Each TOP shall have data exchange capabilities with the following entities, unless the TOP detects a failure of its data exchange capabilities, in which case [another requirement] shall apply."
(2) We believe the Rationale section needs to clarify the meaning of "redundant and diversely routed," and that it does not apply to dual data connection links to each entity. Many entities utilize the infrastructure owned and operated by their RCs to obtain information regarding their neighboring entities. These entities would incur a significant financial burden for installation and maintenance costs associated with these additional data links. Moreover, we have concerns that network performance would be affected with the addition of these redundant links too.
Hydro One Networks Inc. will only be commenting on TOP-001-4 R20 for this question (IRO-002-5 R2 and TOP-001-4 R23 are not applicable to Hydro One Networks Inc.). Hydro One Networks Inc. is satisfied with TOP-001-4 R20.
The requirement needs to be reworded to indicate it is a Transmission Operator and Balancing Authority's primary Control Centers. Suggested wording "Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure within the Transmission Operator's primary Control Center, ...".
On the NERC webex for this project, it was stated the intent was not to have 2 telecom rooms in a control center to achieve redundancy. However, in reading the requirement this is not completely clear with the words "within the Transmission Operator's Control Center". Suggest that the SDT have some guidelines and technical basis included in the standard to provide guidance to the industry on what is required to achieve redundany and diversely routed data exchange.
The requirements for redundancy and diverse routing of data exchange capabilities used by RC, TOP, and BA are vague and not sufficiently developed. This will lead to unnecessary standard violations as both regulator and the industry learn by trial and error what is appropriate and what isn't. At a minimum, information contained in the proposed rationale needs to be incroporated into the actual requirement as FERC has ruled that guidance (such as the rationale) cannot change the scope or intent of a requirement. I suggest at a minimum the SDT define specific important equipment and include rationale wording such as Requirement R2 does not require automatic or instantaneous fail-over of data exchange capabilities and infrastructure that is not within the RC's Control Center is not addressed by this requirement.
NVE has concerns that the language in these requirements are too vague and that the scope of equipment that would be required to have diverse and redundant routing is not clearly defined. NVE recommends guidelines or examples perhaps in the "Guidelines or Technical Basis' section on what equipment would be expected to be diverse and redundant. NVE also requests that clarification is given as to whether the diverse and redundant routing applies equally at the Primary and Backup Control Centers.
3. The SDT has developed IRO-002-5 Requirement R3 and TOP-001-4 Requirements R21 and R24 to address directives for testing redundancy of data exchange capabilities used in RC, TOP, and BA control centers. Do you agree with the proposed requirements? If you do not agree, or if you agree but have comments or suggestions for the proposed requirements provide your recommendation and explanation.
The draft TOP-001-4 R21 and R24 are ambiguous as to the scope of the testing required under the Requirements. As drafted, it is unclear if the testing is inclusive of back-up Controls Centers and if the Requirements apply only within a Control Center or includes exchange of data between Control Centers.
R21: The TOP shall test its Control Center data exchange capabilities with its RC and BA. The Requirement considers only Control Centers which, by definition, "One or more facilities hosting operating personnel that monitor and control the Bulk Electric System (BES) in real-time to perform the reliability tasks..." (See NERC Glossary Terms). The Requirement language suggests only active Control Centers are to test data exchange capabilities and is silent on testing at back-up Control Center facilities.
Also, entities with back-up Control Centers will likely have primary and secondary data exchange servers and connections at their active Control Center and at their back-up Control Center. The Requirement's language does not address this set of facts.
While some TOPs do not have back-up Control Centers, adding language to the Requirement that either includes or excludes testing data exchange capabilities of back-up Control Centers will provide clear expectations and additional clarity for purposes of compliance.
In addition to, or as an alternative to including language in the Requirement that includes/excludes back-up Control Centers, a guidance and technical basis addendum to the Standard would provide clarity for purposes of implementation and compliance purposes.
R24: The language of this requirement creates similar issues outlined, above, regarding R21, and incorporated by reference.
Also, the language of R24 does not address, from a practical view, how a BA will test its data exchange capabilities without potentially impacting and interrupting every related RC and TOP's Control Center operations. Recognizing the capabilities and design of BA data systems are likely unique to each BA, such testing may actually put the reliability of the BES in peril should, as the Requirement contemplates, the test fail. It is not an unreasonable scenario that an unsuccessful fail over to the redundant system can cause a failed return to the primary system with the effect of disabling both systems. While other Standards address the failure of control systems, the potential, as the Requirement is written, may create unintentional consequences, like a disabling of a Control Room's view of real-time data.
Additionally, the Standard would be enhanced by adding technical considerations, as an addendum to the Standard, that address the technical implications of BAs testing data exchange capabilities and assessment of the potential risk of a complete disruption of the availability of real-time data or other unintended consequence.
As stated previously, consideration of related drafting actions in process or other outstanding FERC directives would be prudent as the efforts continue to respond to the existing expectations for redundant and diverse data exchanges.
While PNMR agrees there needs to be testing of the data exchange capabilities to address the directives, we do not agree the propose requirement is the correct way. Paragraph 51 of the FERC Order 817 states, "We believe that the structure of Reliability Standard
COM-001-2, Requirement R9 could be a model for use in the TOP and IRO Standards." FERC states the R9 in COM-001-2 COULD be a model, but not that it "must be" or even "should be". The COM-001-2 model is testing voice capabilities that typically have no ability to be actively monitored or generate alerts upon failure, and thus more frequent manual testing is required. In addition to "tests…data exchange capabilities…for redundant functionality…" requires what exactly? Is it just taking down one component in the primary path from the data exchange device (i.e. ICCP server, mailbox RTU) to the Telco demark? Or is it taking down every possible component on the primary path to ensure automatic failover to the redundant path? If it is taking down every possible component then for one our Control Centers that is approximately 21 components per path. Testing that many monthly seems excessive.
PNMR believes that a better model for testing already exists in the NERC standards in PRC-005-6. The data exchange capabilities are similar in nature to the Communication Systems used in relaying. Many can be monitored on a continuously basis or with periodic automatic testing, and with alarming for loss of function. However the data exchange capability should probably have only time-based maintenance methods to reduce the complexity of the requirement language and because there is little benefit to performance based over time based for the data exchange systems in scope. Also time-based maintenance methods are in line with COM-001-2 model proposed by FERC. Below is proposed requirement language where [X|Y] denotes choose X or Y and comments begin with "NOTE: and are encapsulated in parenthesis.
<Start proposed language; kit bashed from existing PRC and CIP standards as well as the proposed TOP-001-4>
[R21|R24]. Each [Transmission Operator | Balancing Authority] shall implement one or more documented processes comprising a Data Exchange Maintenance Program (DEMP) that collectively addresses each of the following requirements.
[21|24].1. Identify the components that comprise the data exchange capabilities specified in Requirement [R20|R23] and designate the components that comprise the redundant functionality.
[21|24].2. Identify the Entity that is responsible for the operations of each component. (NOTE: Some components of the system that comprises the data exchange capabilities may be owned by a TOP or BA and located at its Control Center, but managed and maintained by the Reliability Coordinator. The SDT needs to give consideration to some language, not included, as to who is responsible for coordination of testing and that other party must be available to support such testing.)
[21|24].3 Identify the Component Attributes in Table [DESIGNATION HERE] applicable to each component identified in [21|24].1. (NOTE: SDT would need to determine if RC controlled components need to be included as part of this identification. If they do then the standards need to have language where the RC must provide the information to the TOP or BA upon request.)
[21|24].4 Identify the maintenance intervals for each component identified in [21|24].1 where the interval shall not exceed the Maximum Maintenance Interval assigned to the corresponding Component Attribute in Table [DESIGNATION HERE]. (NOTE: SDT would need to determine if RC controlled components need this identification. If they do then the standard needs to have language governing who identifies the maintenance interval and how testing is coordinated.)
[21|24].5 Verify every five (5) years that loss of all components not identified as part of the redundant functionality results in data exchange capabilities being operational within five (5) minutes of the loss. (NOTE: The DEMP in x.1 through x.4 address individual component testing and maintenance. This requirement addresses testing the entire system of components to ensure all the components in the redundant infrastructure work in concert to maintain data exchange capabilities.)
[21|24].6 Include plans for restoring data exchange capabilities if failure of a component does not result in the failover to a redundant component or path and maintain data exchange capabilities. (NOTE: R20 and R23 do not require the data exchange capabilities to have failover capabilities to switchover to redundant components or diverse routes. Thus the Data Exchange Maintenance Program should include how failures are addressed as they arise if such failover capabilities do not exist.)
The proposed language includes a time horizon limit of five (5) minutes for any failover scheme to restore data exchange capability. While most network failover schemes operate within seconds, the failover schemes for ICCP servers may be between two (2) and five (5) minutes. The proposed text for Table [DESIGNATION HERE] which is mentioned in the proposed language is shown below in CSV format. This allows the SDT to copy and paste into a text file; open as a CSV in Excel; copy the table from Excel; and paste into Word to maintain table formatting with little effort.
Component Attributes,Maximum Maintenance Interval,Maintenance Activities
Any component not having any of the attributes below,4 Calendar Months,Verify the component is functional.
Any component part of an automatic failover scheme to preserve the data exchange capabilities within five (5) minutes of a failure,3 Calendar Years,Verify the automatic failover scheme preserves the data exchange capabilities within five (5) minutes of a failure.
"Any component with continuous monitoring or periodic automated testing for the functional state of the component, and alarming on loss of function.",3 Calendar Years,"Manually verify the component is in a functional state.
Verify the loss of function results in generation of an alarm."
"Any component with all of the following attributes:
*Part of an automatic failover scheme to preserve the data exchange capabilities within five (5) minutes of a failure
*Continuous monitoring or periodic automated testing for the functional state of the component, and alarming on loss of function",5 Calendar Years,"Verify the automatic failover scheme preserves the data exchange capabilities within five (5) minutes of a failure.
Manually verify the component is in a functional state.
As stated above, if redundancy and diverse routing are achieved by the use of a Primary Contol Center and a Backup Control Center as provided for in EOP-008, testing of this capability needs to reference the testing required for the Backup Control Center. While the EOP-008 testing requirement is for an annual test, R21 is proposing one test per calendar month. The FERC in paragraph 51 has provided no directive on how often to conduct these tests. The FERC directive only requires that the standard revison "addresses a data exchange capability testing framework for the data exchange capabilities used in the primary control centers to test the alternate or less frequently used data exchange capabilities of the reliability coordinator, transmission operator and balancing authority." There is nothing in the directive that would prevent the annual testing of the Backup Control Center's redundancy and diverse routing capabilities from meeting the requirements of the FERC directive. The SDT should change the testing from calendar month to annual and should also add a reference that states "If a Backup Control Center is used to provide the necessary redundancy and diverse routing as required in R20, TOPs will include tests to verify the alternate or less frequently used data exchange capabilities in the annual testing of the Backup Control Center as requred by EOP-008."
AEP believes that region-wide testing of data exchange capabilities for redundant functionality at least once each calendar month would be excessive. There is already an element of risk associated with this volume of testing, and testing on a monthly basis would potentially exacerbate that risk with no benefit to reliabilty. Rather than testing once a month, we believe testing once a calendar quarter is more appropriate. As a result, AEP recommends R21 be re-written as "Each Transmission Operator shall schedule a test of its data exchange capabilities specified in Requirement R20 for redundant functionality at least once each calendar quarter, subject to system conditions, with a test to be completed no less than once per calendar quarter. If the test is unsuccessful, the Transmission Operator shall initiate action within two hours to restore redundant functionality."
It appears that a hardware failure would put us in concompliance, since we would not be able to restore in under 8 hours.
Testing of backup capability is already included in EOP-008 and is only tested on an annual basis. While this requirement adds specificity for data exchange capability, a monthly testing requirement is excessive and could be quite burdensome for some entitites.
Since every test involves a forced interruption of the data, TAL recommends the testing be required QUARTERLY.
The requirement for testing the redundant communications paths on a monthly basis is unnecessarily onerous. Some pieces of the data path could easily be tested though normal failovers when patching while others may require isolating routers and letting data failover to the redundant communication paths. This could potentially degrade real-time operations reliability if the failover is unsuccessful. While testing backup circuits is important, too much testing could decrease system reliability. Quarterly or bi-annual testing of the redundant circuits would be more appropriate.
The FERC order only mentions testing facilities in primary control center. Does the standard intend to require testing the redundancy of data exchange capabilities at primary and back-up control centers?
Testing redundancy of data exchange capabilities used in RC, TOP, and BA control centres is necessary. However, monthly tests seem to be excessive. Quarterly tests would be more appropriate and sufficient to ensure functionality.
Defer to comments from Oliver Burke of Entergy:
While SMUD/BANC agrees with the proposed requirement we repecfully request clarification on the following:
Whether the data exchange capability test is for each link or verification of the link's infrastructure; and,
Whether verification of continuous real-time data exchange of the data links constitute testing.
The FERC directive addresses testing the unused route for data exchange. The requirement as written would require testing both the primary and alternate data exchange infrastructure even if it is used every day. SRP recommends reqriting the requirement to more closely reflect the directive to test only the communication infrastructure that is not used during the month. SRP also recommends providing the opportunity for an entity to use a successful operation of the communication capabilities to alternatively verify the capabilities.
The requirement for testing the redundant communications paths on a monthly basis is unnecessarily onerous. Some pieces of the data path could easily be tested though normal failovers when patching while others may require isolating routers and letting data failover to the redundant communication paths. This could potentially degrade real-time operations reliability if the failover is unsuccessful. While testing backup circuits is important, too much testing could decrease system reliability. Bi-annual testing of the redundant circuits would be more appropriate.
The FERC order only mentions testing facilities in the primary control center. Does the standard intend to require testing the redundancy of data exchange capabilities at primary and back-up control centers?
R21of TOP-001 and R3 of IRO-002 should be revised to be a quarterly test rather than a monthly test. We propose a complete test (EMS failover from the primary to backup) to be conducted quarterly instead of an incomplete test of different components once a month. A thorough test is more effective to ensure reliability
A functional entity is required to "initiate action" after an unsuccessful test of redundant communications functionality. However, there is no requirement for that functionality to eventually be restored - there is no check or requirement regarding repair actions that are initiated but fail to be completed. This does not appear to adequately address the reliability risk.
Southern believes that the SDT should not include a requirement for monthly testing. The standard (as currently written), fails to address data exchange architectures that reside outside of the RC control center. In addition, the standard (as currently written), has a great deal of overlap with EOP-008, which already addresses redundancy, diversity, along with testing at a system level for a broad range of functionalities.
Recommend that testing be done quarterly.
Also in the 'Rationale for Requirement R21' box add a statement like 'for example either planned or unplanned failovers' immediately after: 'When an actual event successfully exercises the redundant functionality, it can be considered a test for the purposes of the proposed requirement.'
R21 should be revised to be a quarterly test rather than a monthly test. We propose a complete test (EMS failover from the primary to backup) to be conducted quarterly instead of an incomplete test of different components once a month. A thorough test conducted quarterly is more effective to ensure reliability.
Duke Energy believes that testing of data exchange capabilities every month would be overly burdensome. For an entity to have to power down, and physically test redundant switches and firewalls to ensure they do in fact switch, would be challenging to accomplish monthly. We request the drafting team to consider extending the timeframe for testing to once a year. Requiring testing once a year reduces burden on entities while maintaining the spirit of the FERC directive. Duke Energy also recommends that in an instance where an event occurs, and failovers work as intended, this should count as evidence that the entity tested the redundant functionality of its data exchange capabilities for that year.
We understand that these changes are in direct response to a FERC Directive and neither agree nor disagree; however, given the impact on real-time monitoring/operations, we are concerned with the periodicity and suggest it be modified from monthtly to quarterly.
Please see Texas RE's answer for #2 regarding the language "within the [RC's and TOP's] Control Center."
CenterPoint Energy does not agree with the monthly periodicity for testing redundancy of data exchange capability. If an operational failover is required for testing, performing that task once per month is not practical. CenterPoint Energy recommends the periodicity should be no less than semi-annually.
CenterPoint Energy requests R21 be more descriptive in its requirement for testing. There could be configurations, which provide redundancy for data exchange capabilities, which are continuously monitored, alarmed, etc. while sharing and communicating information between 'primary' and 'alternate' infrastructure. CenterPoint Energy suggests the following language:
R20. Each Transmission Operator shall verify redundancy of data exchange capability by perfoming one of the following semi-annually:
R20.1. Functional test of redundant functionality
R20.2. Successfully exercising redundant functionality due to an actual event
R20.3. Continuoulsy monitor redundant functionality for status, accuracy, and availability.
If the method used for verification is unsuccessful at any time, the Transmission Operator shall intiate action within two hours to restore redundant functionality.
Does testing of the redundancy imply a requirement to also test the redundancy found within the backup Control Center monthly? The backup Control center functionality is tested at least annually per EOP-008. The requirement in TOP and IRO to test data exchange should apply only to the primary Control Center. We suggest editing the requirement to be clear that the testing should only apply to the primary center's redundancy.
We have a concern that monthly testing could lead to a vast increase in the amount of 'outage requests' submitted by TOP's to RC's. TOP's are required to coordinate (and in some cases gain approvals) with the RC and others when failing over their data exchange tools. We suggest perhaps increasing the time period to require perhaps quarterly testing or even test once every six months.
(1) We request clarification on the application and intent of testing the redundancy of data exchange capabilities. Does testing imply a requirement to also test the redundancy found within the backup Control Center? The backup Control Center functionality is tested at least annually, per EOP-008. The requirement in TOP and IRO to test data exchange should apply only to the primary Control Center. We suggest editing the requirement to be clear that the testing should only apply to the primary center's redundancy.
(2) We have a concern that monthly testing could lead to an increase in the amount of 'outage requests' submitted by TOP's to RC's, and therefore is unduly burdensome. TOP's are required to coordinate with the RC and others when failing over their data exchange tools. We suggest increasing the testing time period to quarterly, or even annually to align with EOP-008.
Hydro One Networks Inc. will only be commenting on TOP-001-4 R21 for this question (IRO-002-5 R3 and TOP-001-4 R24 are not applicable to Hydro One Networks Inc.). Hydro One Networks Inc. has cast a negative ballot on the standard due to the following two concerns with R21. A favourable ballot, however, has been cast on the poll associated with the VRFs/VSLs and Implementation Plan.
I. Hydro One Networks Inc. would like to support the NPCC RSC's comment on suggesting that the drafting team consider R21 be modified to require quarterly (and not monthly) testing of redundant capability. This is because in order to conduct a thorough redundancy test, the primary system would need to be failed intentionally by shutting it down, thereby increasing the risk to reliability during the completion of the failover. Therefore, such a risk to reliability, even for the purpose of conducting a test, should be minimized by performing the test quarterly at most (or ideally, twice annually) and not once per calendar month as is presently specified in R21.
II. Hydro One Networks Inc. would also like to thank the drafting team for providing us with clarity (during the Industry Webinar held on July 22, 2016) that actual events could typically constitute testing of redundancy for those hot standby systems where failover from the primary path to a secondary one is exercised in real-time and where both data exchange paths are continuously monitored for any failure. However, Hydro One Networks Inc. strongly recommends that the drafting team adds this to M21 in order to provide clarity to those entities who own such hot standby systems.
Need to indicate within the requirement the testing is required for only the primary control centers and not back up control centers. The FERC Order indicates testing is only needed for the primary control centers. Also, recommend that testing be conducted quarterly instead of monthly.
Again, guidelines and technical basis on what is required testing would be helpful for the industry. For example, can real-time failovers be constituted as a test?
The requirements for testing are vague and not sufficiently developed. This is wasteful of resources (time and capital) and will lead to unnecessary standard violations. The current zero defect regulation approach requires clear bright line criteria to define when an entity has met compliance. The vague rationale box language suggesting that entities examine "various failure modes" is fine in concept, but doesn't practically work in a zero defect mandatory standard and should be removed.
Excelon Utilities agrees with the comments filed by PJM, specifically, we recommend the time be changed from monthly to quarterly. We believe this will be sufficient for reliability and a more efficient approach.
NVE believes that testing the data exchange capabilities for redundant functionality at least once each calendar month is excessive. Since the test would require a forced interruption of the data, NVE feels that there is a an element of risk associated for the high volume of testing with no benefit to reliability. NVE feels that testing once a calendar quarter (or longer) would be more appropriate.
4. Do you agree with the Implementation Plan for the proposed standards? If you do not agree, or if you agree but have comments or suggestions for the Implementation Plan provide your recommendation and explanation.
The Infrastructure changes and implications for the existing CIP standards may take time to enable. These changes may necessitate investments requiring entities to work with external suppliers, as well as entity approval processes through budget cycles and implementation time. We believe 24 months to be more appropriate for any infrastructure build out what may be necessary for entities to comply.
PNMR disagrees with the IRO and TOP having two separate implementation dates. If the purpose is to require the RC to establish redundancy first then the implementation should be phased on a requirement level. The data exchange capabilities for Reliability Coordinators (IRO-002-5 R2) could be "the first day…that is three months after the effective date…." The data exchange capabilities for TOP and BA (TOP-001-4 R20 and R23) could be "the first day…that is twelve months after the effective date…." However the testing requirements could be muddled based on our response to question #3. Thus the implementation for RC, BA, and TOP for testing requirements should be implemented at the same time especially if an RC to TOP or RC to BA coordination is required for testing as mentioned in our response to question #3.
The three-month implementation period for IRO-002-5 may not be adequate for all entities; suggest it be six-months.
TAL believes three months is insufficient to perform the physical modifications that may be needed to obtain the "diverse routing" as proposed. At least 12-months will be required since the modifications to storm hardened buildings may be required.
The effective date for IRO-002-5 of three months is too short. Testing processes would need to be put into place with each TOp and BA in the RC area. Three months does not seem like sufficient time to develop and coordinate the testing processes in order to ensure that they can be done consistently and on time. Suggest the implementation period for IRO-002-5 be the same 12 month period as with TOP-001-4 based on the similarity of the new requirements
The effective date for IRO-002-5 of three months is too short. Testing processes would need to be put into place with each TOp and BA in the RC area. Three months does not seem like sufficient time to develop and coordinate the testing processes in order to ensure that they can be done consistently and on time. Suggest the implementation period for IRO-002-5 be the same 12 month period as with TOP-001-4 based on the similarity of the new requirements.
Defer to comments by Oliver Burke of Entergy:
No concerns with the timelines proposed with implementation plan for TOP-001-4 assumed 4/1/2018 based on current schedule. . No comments on the IRO-002-5 implementation timeline as it does not apply to ATC directly.
APS is concerned that the acceptability of the implementation plan depends upon the final wording of TOP-001-4, requirements R20 and R23, and what is required to demonstrate redundancy and diverse routing as described in our comments to question #2 above. Until those concerns are resolved, it cannot support the implementation plan as proposed.
The effective date for IRO-002-4 of three months is too short. Testing processes would need to be put into place with each TOP and BA in the RC area. Three months is not sufficient time to develop and coordinate the testing processes in order to ensure that they can be done consistently and on time. The implementation period for IRO-002-5 should be the same 12 month period as with TOP-001-4 based on the similarity of the new requirements.
Three months is not enough time to ensure the appropriate infrastruction and documentation is in place to meet the expectations of the requirements. FMPA suggests a 12 month implementation period for IRO-002-5.
Given the expansion of TOP-001-4 scope to include non-BES equipment in the data specification for Operational Planning Analyses, Real-time monitoring, and Real-time Assessments performed by the TOP in accordance with TOP-003-3, significant preparation, study, and coordination is necessary for all TOPs to comply with the new requirements. Therefore, the implementation plan of 12-months is too short to reasonably complete all preparations and testing. A minimum of a 36-month implementation plan is recommended to best achieve the required changes to reliability functions.
Duke Energy disagrees with the suggested implementation for IRO-002-5. We feel that the Implementation Plans should mirror each other, and should be a 12 month implementation plan for both standards. The standards are basically asking each function to do the same thing, and all should have equal time to implement. Also, in some instances the RC may need to request data from TOP(s), which could take some time to turn around.
CenterPoint Energy does not agree with the Implementation Plan associated with the current language in the revised Requirments. If Entities are going to be expected to provide redundant infrastructure for data exchange capabilities to exist within one Control Center, then CenterPoint Energy recommends an Implementation Plan of no less than 36 months. A configuration as decribed in the current proposal could require purchasing, installing, and training on new infrastructure that cannot be realistically completed in 12 months.
Depending on the response to the level of required redundancy (and testing of said redundancy) within backup Control Center's, some entities may need to purchase equipment or redundant servers for their backup sites. This may require lengthening the implementation period beyond 12 months for TOP-001-4 and 3 months for IRO-002-5.
We have concerns that some entities may need to procure additional equipment for redundant servers at their backup sites. This may not be feasible for smaller entities. We recommend lengthening the implementation period for TOPs to 24 months.
Hydro One Networks Inc. is generally satisfied with the May 2016 draft of the NERC Implementation Plan for TOP-001-4.
Our response is in relation to our registration as a TOP and not for the RC.
Suggest that the implementation period for IRO-005-2 be the same 12 month period as TOP-001-4 based on the similiarity of the new requirements.
"Necessary" and "testing" are not defined enough to support the implementation plan.
Depending on which non-BES facilities are identified in Requirement 10, additional infrastructure may be required to bring back the necessary information. In that scenario, more time may be needed than what is proposed. Until the scope of work is better clarified for the diverse and redundant routing, it is unclear whether or not the implementation plan is appropriate.
5. Do you agree with the Violation Risk Factors (VRFs) and Violation Severity Levels (VSLs) for the requirements in the proposed standards? If you do not agree, or if you agree but have comments or suggestions for the VRFs and VSLs provide your recommendation and explanation.
The VRFs and VSLs mirror the proposed revisions to the Standard is currently offered,
PNMR agrees with the Violation Risk Factors for all the requirements and most of the Violation Severity Levels. However the SDT may find the VSLs for TOP-001-4 R21 and R23 may need further revision after consideration of our response to question #3.
The VFRs for R10 seem to severe for the large number of points that will be required. R21, R24 a hardware failure will likely put us in violation.
Based on NERC's Violation Risk Factors guidance document (dated May 16, 2014), APS recommends that the SDT consider revising the VRFs for requirements R20 and R23 to Medium as the best fit definition because "if violated, they could directly affect the electrical state or the capability of the bulk electric system, or the ability to effectively monitor and control the bulk electric system" in real-time. "However, violation of a medium risk requirement is unlikely, under emergency, abnormal, or restoration conditions anticipated by the preparations, to lead to bulk electric system instability, separation, or cascading failures, nor to hinder restoration to a normal condition."
The VRF and VSL table would need to be adjusted to reflect the requested changes
Duke Energy recommends that the drafting team re-word the Severe VSL for R20 and R23 to more closely align with the language of the requirements. We suggest the following:
"The Transmission Operator did not have data exchange capabilities with its Reliability Coordinator, Balancing Authority, or the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments."
For TOP-001-4 Requirement R10, Texas RE interprets the VSLs to mean the following:
If an entity fails to monitor all Facilities per part 10.1, there is a violation with a lower VSL.
If an entity fails to monitor 1/10 Facilities per part 10.1, there is a violation with a lower VSL.
And so forth for parts 10.2 – 10.6.
Adding the word "all" in subparts of TOP-001-4 R10 would add clarity to the requirements:
10.1 Monitor all Facilities…
10.2 Monitor the status of all Remedial Action Schemes…
10.3 Monitor all non-BES facilities…
10.4 Obtain and utilize status, voltages, and flow date for all Facilities outside…
10.5 Obtain and utilize the status of all Remedial Action Schemes…
10.6 Obtain and utilize status, voltages, and flow data for all non-BES facilities…
In Requirement 10 of TOP-001-4, some of the items will only need to be exchanged if the TOP determines them to be necessary. However the VSL for R10 does not recognize that some of the items may not be necessary and the TOP may not be obtaining them.
We do not agree with the High Violation Risk Factors identified for the proposed requirements. Testing, by itself, should not directly cause or contribute to a Bulk Electric System instability, separation, or a cascading sequence of failures. Hence, a Medium risk should be assigned to align with redundant communications capabilities. We ask the SDT to also provide clarification for requirement R10, stating that some of the items will only need to be exchanged if the TOP determines them to be necessary. This is contradictory to the VSLs for R10 that do not recognize that some of the items may not be necessary for TOPs.
Hydro One Networks Inc. is generally satisfied with the VRFs and VSLs presented in Draft 1 (May 2016) of TOP-001-4. Accordingly, a favourable position has been indicated in the associated poll.
6. Provide any additional comments for the SDT to consider, if desired.
As a general comment, the SDT's work may materially be impacted by the scope and work on the Project 2016-02, Modifications to CIP Standards. There are areas that may conflict, such as the definition of Control Center and communication between Control Centers. It raises questions like, "Is the Standard only applicable to communication links between internal Control Centers or apply only to links with external Control Centers, such as between a TOP and BA's Control Centers?"
Recognizing the SDT can only address what is "true" today in advancing the project, consideration of the work and direction of the Project 2016-02 may provide insight and an opportunity to address and incorporate into the TOP and IRO Standards language that would better align with the potential modifications to the CIP Standards.
The recent webinar for this project demonstrated the current draft is still ambiguous and needs more language to clarify intent within the actual requirement and not just the rationale box. While we have tried to comment and provide language for the SDT to use, please consider making it clear in the standard what is required since that is what is enforceable and leaving any meat in the rationale box. For example R21 requires monthly testing, but the rationale indicate "…testing practices should, over time, examine the various failure modes of its data exchange capabilities." The rationale and requirement do not fully agree. The rationale gives an ambiguous time horizon for testing various failures modes while the requirement seems to indicate all failure modes are tested monthly. Please make sure any requirement language fully and clearly reflects the rationale.
AEP has chosen to vote negative on TOP-001-4, driven by the concerns expressed above.
We appear to be on a slippery slope of expanding the reach of the NERC Standards and mandatory compliance. If 100 kV is the appropriate threshold, then stick with it. If the threshold should be lower then build the case and make it official, not a piece at a time infiltrating our Distribution systems.
It would be good to stabilize these two standards. The TOP standard is approaching 40 requirements and sub-bullets. Future changes should be based on data that shows a need for new requirements.
City Light subject matter experts believe that the periodicity stated in R21 and R24 testing requirement of "at least once each calendar month" is excessive. The FERC directive states "TOP and IRO standards that addresses a data exchange capability testing framework". Based on our SMEs system experience, they believe that quarterly testing would be sufficient. Thank you for your consideration.
Proposed TOP-001-4 R10 requires TOP's to monitor its facilities, Remedial Action Schemes and Non-BES facilities that it identifies as necessary to determine SOL exceedances in R10.1, R10.2 and R10.3. For Sub-Requirements R10.4, R10.5 and R10.6 the wording has changed to "obtain and utilize" instead of the former "monitor" used in previous drafts of TOP-001-3. These Sub-Requirements also use the wording "identified as necessary by the Transmission Operator". The proposed TOP-001-4 RSAW requires the Transmission Operator to provide evidence that it monitored all the data stated in the Sub-Requirements without requiring the TOP to providing reasoning or qualifications for how the TOP determined what or how the data "obtained and utilized" was "identified as necessary". This creates unenforceable requirements that have no reason to be added to a Standard.
Proposed TOP-001-4 R10.5 requires TOPs to obtain and utilize statuses of Remedial Action Schemes in neighboring TOP areas. Currently TOP SPS statuses is communicated through notifications required to the RC and affected TOPs. This notification process requirement works and keeps the wide area system monitoring and control responsibility on ERCOT the Reliability Coordinator and not on individual TOPs.
In closing, the ERCOT region is structured to support a deregulated market in which ERCOT monitors facilities for all TOPS and has a centralized view of the entire region to maintain reliability. TOPs operating within ERCOT currently do not have the technical capability to obtain and utilize data specified in R10.4, R10.5 and R10.6. This requirement imposes a "one size fits all" regional structure which would place an unreasonable financial burden on all TOPs to both install and maintain additional hardware in each station or install and maintain multiple ICCPs between control centers. This requirement would place this financial burden on TOPs for nothing more than to replicate an RC function with no benefit to the BES. At no point in proposed Standard TOP-001- 4 does it require TOs to supply neighboring TOs with this data. Oncor requests R1O.4, R10.5, R10.6 be removed from the standard due to lack of regional flexibility.
No additional comments
Tri-State believes the Standard Drafting Team should clarify that TOP-001-4 R20 and R21 are applicable to only a TOP's primary Control Center. As the draft is currently written, it is unclear if the backup Control Centers are inadvertently included because NERC EOP-008-1 requires an entity to meet its functional obligations in the event of the loss of the primary Control Center. The testing requirement for EOP-008-1 is on an annual basis which is not the same periodicity of the monthly test required in R21. Requiring monthly testing of the backup Control Center in accordance with the proposed TOP-001-4 R21 would add undue burden. Tri-State would like the Standard Drafting Team to explicitly exclude the backup Control Centers from these requirements.
The revised language of R19 in TOP-001 results in what some might consider a new requirement. While it is clear what type of data exchange capabilities are expected for exchanging real-time information, it is less clear what is expected for day-ahead information. Does email satisfy the requirement? Can third party systems, such as SDX, be used? FMPA believes additional clarity is needed.
Whenever a Standard references "Control Center", Texas RE considers the reference to include any Control Center (primary, back-up, tertiary, etc.) as capabilities must be present (and redundant and diversely routed) for an entity to do Real-time monitoring and Real-time Assessments.
In the Evidence Retention section of TOP-001-4, it states: "For instances where the evidence retention period specified below is shorter than the time since the last audit, the Compliance Enforcement Authority may ask an entity to provide other evidence to show that it was compliant for a full-time period since the last audit." Rather than ask the entities to keep evidence outside of the specified evidence retention period, Texas RE recommends aligning all evidence retention to "since the last audit of the requirement".
Texas RE noticed in Section F of TOP-001-4, "Associated Documents", it appears that 'Operating Plan" has an explanation that is inconsistent with the NERC Glossary of Terms. Texas RE recommends making them consistent.
TOP-001-4 R23 states: "Each Balancing Authority shall have data exchange capabilities…with its Reliability Coordinator". There could be times when the Balancing Authority might need to coordinate and exchange data with a Reliability Coordinator that is not its own. Texas RE suggests changing "its" to "the applicable Reliability Coordinator".
As written IRO-005-2 R3 refers to the test being unsuccessful and implies the test itself could not take place. Texas RE recommends revising the requirement is to say: "if the results of the test reveal there is no functionality…"
TOP-001-4 R7 is an extremely vague requirement. Texas RE suggests it might be better suited as a guideline.
(1) We have concerns regarding the financial implications smaller entities will face with the increased level of redundancy and testing proposed for backup Control Centers. In order to meet these proposed requirements, some entities would need to make sizeable investments to procure redundant equipment and staff for their backup sites. We feel the cost factor would constitute an unduly and unreasonable burden placed on smaller entities.
(2) We thank the SDT for this opportunity to comments on these standards.
We ask that you reevaluate the TOP standard with the Enhanced Periodic Review Team if it is not already scheduled.
It would be good to stabilize these two standards. The TOP standard is approaching 40 requirements and sub-bullets.
The use of "Facilities" capitalized in Requirement 10.1 means it is part of the BES. It may be helpful to reword as "Monitor BES Facilities" so it's obvious without having to review the definition of Facilities that this requirement is for BES facilities.
Both section 10.4, 10.5 and 10.6 describe a Transmission Operator Area which is a defined term in the NERC standard. This term is also utilized in many regional joint registration organizations. If this term is going to be utilized in the NERC standard to provide direct responsibility of the RAS. LCRA TSC believes that the responsibility descriptions should be better defined in the standard. One example of this is defining that if the operator/owner has an RAS they are responsible for monitoring it. If the RAS is owned and operated by another entity but is in a Transmission operators area the BA or owner/operator should be responsible for monitoring it. Not the TOP.
See above, quarterly vs monthly schedule . Thank you.