* This transcript was created by voice-to-text technology. The transcript has not been edited for errors or omissions, it is for reference only and is not the official minutes of the meeting. AHA. [00:00:01] WE'RE GONNA CALL A MEETING TO [1. Call General Session to Order] ORDER. WELCOME EVERYONE TO THE TECHNOLOGY AND SECURITY COMMITTEE. UH, I'M JOHN SWENSON, THE COMMITTEE CHAIR, AND THIS IS OUR SEPTEMBER 16TH COMMITTEE MEETING. I'M GONNA CALL THE MEETING TO ORDER. THIS MEETING IS BEING WEBCAST LIVE TO THE PUBLIC ON ERCOT WEBSITE, AND I CAN SEE THAT THERE'S A QUORUM PRESENT IN PERSON. NEXT, I'M GONNA ASK THE PUBLIC UTILITY COMMISSION CHAIR THOMAS GLEASON, IF YOU'D LIKE TO CALL AN OPEN MEETING OF THE PUBLIC UTILITY COMMISSION OF TEXAS. YES, I WOULD. THANK YOU, MR. CHAIRMAN. THIS MEETING, THE PUBLIC UTILITY COMMISSION OF TEXAS WILL COME TO ORDER TO CONSIDER MATTERS THAT HAVE BEEN DULY POSTED WITH THE SECRETARY OF STATE FOR SEPTEMBER 14TH, 2026. THANK YOU, THOMAS. UM, BEFORE MEETING ONTO TODAY'S BUSINESS, I'D LIKE TO HIGHLIGHT TO MEMBERS THE ANTITRUST ADMONITION AND THE EVACUATION MAP THAT ARE IN FRONT OF YOU ON THIS SCREEN ONTO AGENDA ITEM TWO, NOTICE OF PUBLIC COMMENT. TODAY'S MEETING AGENDA WAS POSTED PUBLICLY ON SEPTEMBER 4TH AND PROVIDED INSTRUCTIONS TO THE PUBLIC TO, UH, COMMENT OR ASK QUESTIONS TO DATE. TO THE BEST OF MY KNOWLEDGE, NO ONE HAS EXPRESSED INTEREST, BUT IS THAT CORRECT, BRENT? THAT IS CORRECT. CHAIR. THANK YOU VERY MUCH. OKAY. THEN ON TO THE AGENDA. SORRY, ON TO THE MINUTES [3. June 1, 2026, General Session Meeting Minutes] FROM THE LAST MEETING, WHICH IS AGENDA ITEM THREE, JUNE 1ST, 2026, GENERAL SESSION TECHNOLOGY AND SECURITY COMMITTEE MEETING MINUTES. UH, THERE IS A DRAFT IN YOUR MATERIALS. DOES ANYONE HAVE ANY COMMENTS OR WISHED TO MAKE A MOTION? SO MOVED. THANK YOU VERY MUCH, JULIE. A SECONDER PLEASE. THANK YOU. ALL IN FAVOR? AYE. ANY OPPOSED? ANY ABSTENTIONS? HEARING NONE. THE THE VOTE IS POSITIVE. I'M PLEASED TO WELCOME [4. Voltage and Frequency Ride-through for Large Computational Loads – State of the Art] OUR SPECIAL GUESTS FOR THIS MEETING. UM, WE'VE HAD SEVERAL DISCUSSIONS ABOUT THE IMPACT OF LARGE COMPUTATIONAL LOADS, UH, ON THE, ON GRIDS IN GENERAL, AND WE'RE GONNA CONTINUE THAT AND GET A LITTLE MORE TECHNICAL, UM, IN THIS SESSION WITH, UH, SEVERAL SPEAKERS FROM THE SANDIA NATIONAL LAB. UM, THEIR TOPIC IS, AS YOU CAN SEE HERE, IS VOLTAGE AND FREQUENCY RIDE THROUGH FOR LARGE COMPUTATIONAL LOADS, STATE OF THE ART, AND THEY'RE GONNA TALK ABOUT WHAT CAN BE DONE. UM, SOME OF THIS IS FUTURE LOOKING. UH, I WOULD, I WOULD CAUTION YOU THAT, AND, AND THEY'LL EXPLAIN WHAT'S SORT OF AVAILABLE NOW AND WHAT'S AVAILABLE IN THE NEAR FUTURE AND WHAT'S PR POTENTIALLY AVAILABLE AS PEOPLE REDESIGN DATA CENTERS, WHICH THINK NOW 10, 10 YEARS OUT KIND OF TIMEFRAME. UH, RYAN ELLIOT IS A PRINCIPAL MEMBER OF THE TECHNICAL STAFF AT THE ELECTRICAL POWER SYSTEMS RESEARCH AT SANDIA. HIS WORK CENTERS ON CHALLENGES AND GRID DYNAMICS. UM, AND I WON'T READ HIS WHOLE, UH, HIS WHOLE BIO BECAUSE IT'S VERY INTERESTING, BUT HE RECEIVED HIS PHD IN ELECTRICAL AND COMPUTER ENGINEERING FROM THE UNIVERSITY OF WASHINGTON IN 2020. AND MATTHEW RENO IS A DISTINGUISHED MEMBER OF THE TECHNICAL STAFF, THE ELECTRICAL POWER SYSTEMS RESEARCH DEPARTMENT OF SANDIA NATIONAL LABS. AND DR. RENO LEADS A DIVERSE PORTFOLIO OF RESEARCH INITIATIVES, AIMS AT CUTTING EDGE, UH, NEXT GENERATION PROTECTIVE RELAYING SCHEMES AND APPLYING CUTTING EDGE MACHINE LEARNING ALGORITHMS TO THE GRID. HE RECEIVED HIS PHD FROM GEORGIA, GEORGIA INSTITUTE OF TECHNOLOGY. SO WITH THAT, UH, LEMME TURN IT OVER TO RYAN AND MATTHEW. THANK YOU. OKAY. , THANK YOU TO THE CHAIR, AND, UH, THANK YOU ALL FOR HAVING US. WE'RE, WE'RE EXCITED TO BE HERE. LAST MONTH, ERCOT BECAME THE FIRST OPERATING ENTITY IN THE UNITED STATES TO IMPOSE ENFORCEABLE DISTURBANCE RIDE THROUGH REQUIREMENTS THAT ALL LARGE COMPUTATIONAL LOADS AT 75 MEGAWATTS OR ABOVE. SO THIS INCLUDES DATA CENTERS AND CRYPTO MINING FACILITIES MUST FOLLOW. THESE CAME ABOUT THROUGH UPDATES TO ERCOT NODAL OPERATING GUIDES THROUGH A REVISION, UH, REQUEST PROCESS CALLED NOER 2 82. IN TODAY'S PRESENTATION, WE'RE GOING TO DISCUSS HOW EXISTING DATA CENTERS CAN BECOME COMPLIANT WITH THESE NEW RIDE THROUGH REQUIREMENTS AND THE TYPES OF TECHNOLOGIES THAT ARE AVAILABLE TO HELP THEM DO THAT. GREAT. SO FIRST, SOME, SOME BACKGROUND. LET'S REACQUAINT OURSELVES WITH THIS QUESTION OF VOLTAGE RIDE THROUGH FOR DATA CENTERS IN THE CONTEXT OF DATA CENTERS. THE DRIVING QUESTION BEHIND VOLTAGE RIDE THROUGH IS FOR HOW LONG MUST A DATA CENTER REMAIN CONNECTED TO THE GRID FOR A DISTURBANCE IN VOLTAGE OF A PARTICULAR SIZE OR MAGNITUDE? AND THIS IS ESPECIALLY RELEVANT IN THE CONTEXT OF TODAY'S DATA CENTERS THAT CONTAIN BACKUP POWER SOURCES CALLED UNINTERRUPTIBLE POWER SUPPLIES OR UPSS. AND, UM, THIS IS RELEVANT BECAUSE, UM, THIS, THESE BACKUP POWER SOURCES GIVE THEM THE CAPABILITY TO TRANSFER THEIR LOAD FROM BEING SUPPLIED BY THE GRID TO BEING SUPPLIED BY THE UT UH, BY THE UPS WHEN A DISTURBANCE IS DETECTED. [00:05:02] UM, AND AS WE GO THROUGH TODAY'S PRESENTATION, WE'RE GONNA DISCUSS FIVE OF THE KEY TECHNOLOGIES THAT ARE, UH, AVAILABLE TO HELP TODAY'S DATA CENTERS COMPLY WITH THESE, UH, NEW AND EMERGING RIDE THROUGH REQUIREMENTS. UH, AS THE CHAIR NOTED, SOME OF THESE THINGS ARE, UM, MATURE AND COMMERCIALLY AVAILABLE TODAY, SUCH AS CENTRALIZED BATTERY ENERGY STORAGE, AND OTHERS ARE FARTHER OUT ON THE INNOVATION PIPELINE. UM, SO THINGS WE EXPECT TO SEE DEPLOYED IN THE NEXT, SAY, TWO TO FIVE YEAR TIMEFRAME. UM, AND IT ALSO STANDS TO REASON THAT IF VOLTAGE RIDE THROUGH IS IMPORTANT, THAT FREQUENCY RIDE THROUGH IS ALSO AN IMPORTANT, UH, QUESTION FOR DATA CENTERS. AND IT IS, UH, IT TURNS OUT THAT BECAUSE DATA CENTER LOAD IS PRIMARILY POWER ELECTRONIC IN NATURE, DATA CENTERS ARE IN MANY WAYS, UM, MORE ROBUST OR MORE RESILIENT TO DEVIATIONS IN FREQUENCY. HOWEVER, THERE ARE STILL MANY IMPORTANT QUESTIONS, UH, ASSOCIATED WITH FREQUENCY RIDE THROUGH, ESPECIALLY IN THE AREA OF PROTECTION IMPLEMENTATION. AND WE'LL TALK ABOUT SOME OF THOSE. OKAY. SO BY WAY OF BACKGROUND, UM, WHY IS IT SO IMPORTANT FOR OPERATING ENTITIES LIKE ERCOT TO ESTABLISH CLEAR RIDE THROUGH REQUIREMENTS THAT ALL LARGE COMPUTATIONAL LOADS HAVE TO FOLLOW? WE CAN EVALUATE THIS QUESTION THROUGH THE LENS OF THE RECENT DISTURBANCE THAT OCCURRED ON JULY 22ND IN THE EASTERN INTERCONNECTION. THE PRECIPITATING EVENT FOR THIS, UH, DISTURBANCE WAS A SINGLE LINE TO GROUND FAULT THAT OCCURRED IN THE TRANSMISSION NETWORK. A FAULT IS ESSENTIALLY A SHORT CIRCUIT TYPE CONDITION THAT OCCURS IN THE, UH, IN THE, IN THE POWER SYSTEM. AND A COMMON CAUSE COULD BE SOMETHING LIKE TREE CONTACT, SO A BRANCH OR A TREE, UM, FALLING ON THE LINE. UH, IN THIS CASE, THIS FAULT CAUSED 3,800 MEGAWATTS OF DATA CENTER LOAD TO TRANSFER TO ITS BACKUP POWER SOURCES, TO ITS UPSS. UH, AND WHEN THAT HAPPENED, YOU CAN SEE THAT, UM, A LARGE INCREASE IN FREQUENCY MANIFESTED ITSELF THROUGHOUT THE ENTIRE EASTERN INTERCONNECTION. UM, THIS, THE SCALE OF THIS DISTURBANCE IS REALLY NOTABLE BECAUSE HERE WE'RE TALKING ABOUT A DISTURBANCE THAT WAS ALMOST FOUR GIGAWATTS, AND THAT STARTS TO RIVAL WHAT WE CALL THE LARGEST SINGLE HAZARD IN THE SYSTEM, WHICH IS GENERALLY THE LOSS OF MAYBE YOUR LARGEST ONE OR TWO GENERATING UNITS. BUT HERE THE DISTURBANCE IS ORIGINATING ON THE DEMAND SIDE RATHER THAN ON THE GENERATION SIDE. IT'S ALSO WORTH NOTING THAT IF THE, UM, RIDE THROUGH REQUIREMENTS ADOPTED BY ERCOT HAD BEEN IN PLACE, THAT THIS DISTURBANCE WOULD NOT HAVE HAPPENED. SO THE FAULT WOULD'VE STILL OCCURRED, BUT THE DATA CENTERS WOULD'VE KEPT THEIR LOAD, UH, AS GRID CONNECTED. OKAY. OKAY. SO AS WE SURVEY, UM, THESE VARIOUS GRID TECHNOLOGIES, WE WILL EVALUATE THEM THROUGH SEVERAL DIFFERENT KEY DIMENSIONS. SO ONE OF THESE IS THE ONETIME OR THE UPFRONT COST THAT A DATA CENTER IS GOING TO NEED TO PAY TO INSTALL THEM AT THEIR FACILITY. IT'S OBVIOUSLY A KEY ONE. THE NEXT ONE THAT WE'LL EVALUATE IS THE IMPACT ON FACILITY EFFICIENCY. SO DOES INSTALLING THAT TECHNOLOGY CAUSE THE DATA CENTER TO CHANGE THE WAY IT CONSUMES ELECTRICITY? THEN WE'LL ALSO EVALUATE IS IT POSSIBLE TO RETROFIT AN EXISTING DATA CENTER WITH THAT NEW TECHNOLOGY, OR WOULD IT RE REQUIRE ESSENTIALLY A COMPLETE REDESIGN? UH, AND THEN FINALLY WE'LL TALK ABOUT THE TIMELINE FOR AT SCALE DEPLOYMENT. SO IS THIS SOMETHING THAT'S AVAILABLE TODAY OR IS THIS SOMETHING THAT'S STILL, UH, IN THE, UH, INNOVATION PIPELINE? ALRIGHT, SO WE'LL BEGIN OUR TOUR OF THESE VARIOUS TECHNOLOGIES WITH THE SO-CALLED E STAT COM, AND THAT STANDS FOR ENHANCED STAT COM. SO FIRST OF ALL, WHAT IS THAT? SO LET'S DEFINE IT. UH, A STAT COM IS ESSENTIALLY, UH, AN INVERTER THAT INSTEAD OF HAVING A REAL POWER SOURCE BEHIND IT HAS, UH, A LOW ENERGY CAPACITY STORAGE DEVICE, SUCH AS A SUPER CAPACITOR. E STAT COMS PLAY AN IMPORTANT ROLE IN REGULATING VOLTAGE BY THE WAY THAT THEY ADJUST REACTIVE POWER. HOWEVER, IN THE CONTEXT OF, UH, VOLTAGE RIDE THROUGH, WE'RE GONNA BE TALKING ABOUT THEIR ACTIVE POWER CAPABILITY. AND ONE OF THE INTERESTING THINGS, OR THE MAIN ADVANTAGES OF A SUPER CAPACITOR IS THAT IT HAS THE ABILITY TO CYCLE ITSELF, MEANING TO CHARGE AND TO DISCHARGE AN INCREDIBLY HIGH NUMBER OF TIMES, ESSENTIALLY WITHOUT AN, UH, SUFFERING ANY KIND OF DEGRADATION. SO THIS MEANS THAT THEY CAN HAVE VERY LONG LIFETIMES IN THE FIELD. THE TRADE OFF FOR THAT IS THAT THEY CAN GENERALLY ONLY STORE ENOUGH ENERGY TO SUPPLY POWER FOR SEVERAL SECONDS AS OPPOSED TO SEVERAL MINUTES TO SEVERAL HOURS AS WITH LITHIUM ION BATTERIES. OKAY. SO NOW LET'S TALK ABOUT ETA COMS AND VOLTAGE RIDE THROUGH. HOW DOES AN ETA COM HELP US IN A VOLTAGE RIDE THROUGH SCENARIO? UM, THE PLOT ON THE TOP HERE, IF I, I DON'T DUNNO WHAT HAPPENED TO THE MOUSE. MY APOLOGIES. UM, THE PLOT ON THE TOP HERE IS SHOWING A FAULT THAT OCCURS, UH, ON THE GRID AND THAT FAULT IS INDUCING, UH, DEVIATION IN THE GRID VOLTAGE. IF THAT CHANGE IN VOLTAGE IS LARGE ENOUGH, THE DATA CENTER IS GONNA TRANSFER ITS COMPUTATIONAL LOAD FROM BEING SUPPLIED BY THE GRID TO BEING SUPPLIED BY ITS UPS, WHICH IS EXACTLY WHAT WE SEE IN THE SECOND PLOT HERE. AND WHEN THAT HAPPENS, THE GRID SEES A STEP CHANGE IN ITS GENERATION [00:10:01] DEMAND BALANCE. WE HAVE LOST A HUNDRED PERCENT OF THE LOAD THAT WAS AT THAT DATA CENTER. THAT'S WHAT THE GRID OBSERVES. AND WHAT THE E AT COM CAN DO IS THAT, UH, AS QUICKLY AS POSSIBLE. AFTER THAT, UM, UH, LOAD HAS BEEN TRANSFERRED TO THE UPS, WE CAN BEGIN CHARGING THE SUPER CAPACITORS TO MAKE UP THAT DEFICIT, ESSENTIALLY TO RECOVER THE DEMAND OF THE DATA CENTER SO THAT THE GRID DOESN'T SEE THAT STEP CHANGE IN THE GENERATION AND DEMAND HAND. OKAY. SO, UM, THE NEXT TECHNOLOGY THAT WE'LL TALK ABOUT IS CENTRALIZED BATTERY ENERGY STORAGE. SO AS WE NOTED, E COMS CAN BE GREAT TOOLS, BUT THEY CAN GENERALLY ONLY SUPPLY POWER FOR SEVERAL SECONDS. SO WHAT DOES THIS MEAN FOR FAULTS OR VOLTAGE DISTURBANCES THAT TAKE A LONGER TIME TO RECOVER FROM? THIS IS WHERE CENTRALIZED BATTERY ENERGY STORAGE CAN COME IN, CENTRALIZED BATTERY ENERGY STORAGE CAN DO EVERYTHING THAT AN E-COM CAN DO AND, AND THEN SOME, RIGHT? SO, UM, AND ONE OF THE THINGS THAT'S BEING DEPICTED ON THIS SLIDE IS THAT, UM, UH, CENTRALIZED BATTERY ENERGY STORAGE SYSTEMS ALSO PROVIDE ADDITIONAL BENEFITS BEYOND VOLTAGE RIDE THROUGH SO THEY CAN, UM, PROVIDE THE CAPABILITY TO SHAPE THE DEMAND OF THE DATA CENTER AS A FUNCTION OF TIME. AND THIS CAN BE IMPORTANT IN ADDRESSING WORKLOAD VARIABILITY, FOR EXAMPLE, FROM, UH, AI TRAINING DATA CENTERS. OKAY. UM, UH, NEXT ON OUR TECHNOLOGY LIST IS SERIES BATTERY ENERGY STORAGE. UM, TRADITIONAL CENTRALIZED BATTERY ENERGY STORAGE SYSTEMS ARE CONNECTED SO THAT THEY ARE IN PARALLEL WITH THE DATA CENTER LOAD. IN THIS CASE, WE'RE TAKING A DIFFERENT APPROACH AND WE'RE CONNECTING IT IN SERIES SO THAT THE ENTIRE DATA CENTER FACILITY IS COUPLED TO THE GRID VIA A SINGLE POWER ELECTRONIC INTERFACE. AS YOU SEE HERE, ONE WAY TO THINK ABOUT THIS IS THAT WE'RE, WE'RE PUTTING THE ENTIRE DATA CENTER BEHIND WHAT'S ESSENTIALLY A DOUBLE CONVERSION UPS. SO, UM, THE, THE QUESTION THAT WE MIGHT HAVE IS, WHY IS THIS AN INTERESTING APPROACH? WHY IS THIS SOMETHING THAT'S BEING DISCUSSED? EXCUSE ME. OKAY. UM, ONE OF THE BIGGEST PERCEIVED ADVANTAGES OF THIS SERIES, BATTERY ENERGY STORAGE TOPOLOGY, IS THAT IT HAS THE POTENTIAL TO, UM, TO REDUCE THE AMOUNT OF TIME THAT PROSPECTIVE DATA CENTERS SPEND IN THE INTERCONNECTION QUEUE. AND THE THINKING IS THAT THAT CAN HAPPEN AS A RESULT OF SPEEDING UP THE INTERCONNECTION STUDIES THAT NEED TO BE RUN AS PART OF THAT PROCESS. AND THE PRIMARY MECHANISM FOR THAT IS THAT BY INTRODUCING THIS SINGLE POWER ELECTRONIC INTERFACE WITH THE GRID, WE CAN SUBSTANTIALLY SIMPLIFY THE WAY THAT WE MODEL OR REPRESENT THAT DATA CENTER IN OUR SIMULATION STUDIES. AND THAT'S NOT JUST FOR INTERCONNECTION STUDIES, BUT DISADVANTAGE EXTENDS TO SAY PLANNING AND OPERATIONAL STUDIES AS WELL. AND THAT'S EXACTLY WHAT'S BEING SHOWN HERE. SO THE, THE, UM, THE DIAGRAM ON THE TOP IS SHOWING THIS MODEL WHERE WE HAVE TO REPRESENT ALL THE DISPARATE COMPONENTS WITHIN THE DATA CENTER. AND THEN THE MODEL ON THE BOTTOM IS SHOWING WHAT HAPPENS WHEN WE CAN REPLACE ALL OF THOSE WITH THAT SINGLE POWER ELECTRONIC INTERFACE FROM THIS SERIES BATTERY ENERGY STORAGE SYSTEM. OKAY. UM, THE NEXT TECHNOLOGY THAT WE'LL TALK ABOUT IS THE SO-CALLED GRID INTERACTIVE UPS. AND THIS ONE IS DIFFERENT FROM THE ONES THAT WE'VE DISCUSSED SO FAR. ALL OF THESE THINGS THAT WE'VE DISCUSSED PREVIOUSLY, THE E AT COM AND THESE DIFFERENT STORAGE SYSTEMS, THESE ARE THINGS THAT, UH, REQUIRED NEW HARDWARE TO BE INSTALLED AT THE DATA CENTER FACILITY. HOWEVER, ONE OF THE COMPELLING ASPECTS OF A DIS, UH, OF A GRID INTERACTIVE UPS IS THAT, UH, AS LONG AS THE FACILITY ALREADY CONTAINS UNIN, UH, UNINTERRUPTIBLE POWER SUPPLIES, WE MIGHT BE ABLE TO SIMPLY UPDATE THE FIRMWARE THAT'S RUNNING ON THOSE DEVICES. ESSENTIALLY, THE INSTRUCTIONS THAT DICTATE HOW THEY RESPOND TO DISTURBANCES. SO THIS IS A COMPELLING ASPECT THAT CAN DEFINITELY REDUCE COST AND, UM, SAY FACILITY DOWNTIME FOR DATA CENTER OWNER OPERATORS. ONE OF THE INTERESTING CAPABILITIES OF A GRID INTERACTIVE UPS IS THIS ABILITY TO PERFORM WHAT'S CALLED POWER SHARING. SO ESSENTIALLY HERE WE HAVE THE RESPONSIBILITY OF SERVING THE DATA CENTER'S COMPUTATIONAL LOAD BEING SHARED, BOTH BETWEEN THE GRID AND THE BATTERIES ON THE UPS. AND THIS CAN BE DONE DOWN TO ABOUT 50% OF NOMINAL VOLTAGE. HOWEVER, ONCE WE'VE CROSSED THAT THRESHOLD, AGAIN, WE HAVE TO DISCONNECT FROM THE GRID, MEANING WE HAVE A DISCREET SWITCHING ACTION AND THE GRID IS GONNA SEE THAT STEP CHANGE IN THE GENERATION AND DEMAND. ALRIGHT. UM, THE FINAL TECHNOLOGY THOUGH THAT WE'LL LOOK AT IS REALLY SOMETHING THAT IS MORE FUTURE LOOKING. SO INSTEAD OF TALKING ABOUT THINGS THAT WE CAN INSTALL IN A DATA CENTER, WHAT THIS IS LOOKING AT IS WAYS THAT WE CAN FUNDAMENTALLY RETHINK HOW POWER IS DELIVERED AND DISTRIBUTED WITHIN THE DATA CENTER. UH, AND ONE OF THE KEY WAYS THAT THAT'S HAPPENING IS THROUGH INCREASED USE OF DC DISTRIBUTION WITHIN THE DATA CENTER. SO, UM, AS YOU SEE HERE IN THIS DIAGRAM, SO WHAT'S BEING SHOWN HERE IS THAT WE CAN SUPPLY A, A RELATIVELY HIGH DC VOLTAGE. WHAT'S [00:15:01] BEING DISCUSSED IS 800 TO 1500 VOLTS DC AT THE MOMENT DIRECT DIRECTLY TO THE COMPUTATION RACKS THEMSELVES. AND THIS HAS SEVERAL ADVANTAGES. SO ONE OF THE ADVANTAGES IS THAT IT, IT ALLOWS US TO SKIP A POWER ELECTRONIC CONVERSION STEP THAT IS NEEDED TODAY, WHERE WE CONVERT THAT POWER BACK FROM DC TO AC. BUT REALLY THE MAIN ADVANTAGE IS THAT, UH, IT HELPS MANAGE SUPPLY CHAIN RISKS BECAUSE USING THIS KIND OF APPROACH, YOU NEED MUCH LESS HEAVY GAUGE WIRE TO CONNECT THE COMPUTATION RACKS. AND AS A RESULT, IT SIG SIGNIFICANTLY REDUCES THE, UM, U UH, THE FACILITY'S DEPENDENCE ON COPPER, WHICH CAN BE AN IMPORTANT ASPECT AS WE'RE TALKING ABOUT THE SCALE, UH, AT WHICH THESE COMPUTE FACILITIES ARE, ARE BEING BUILT. THE OTHER REALLY INTERESTING THING HERE IS THAT BY USING THIS, UM, YOU KNOW, UM, MORE IN INNOVATIVE, UH, UH, TOPOLOGY IS THAT WE CAN KEEP THIS ENERGY STORAGE THAT IS PRESENT WITHIN THE, UM, THE, THE UNINTERRUPTIBLE POWER SUPPLY CONNECTED ON THE DC BUS ALL THE TIME. UH, AND WE'LL TALK ABOUT WHY THAT'S SOMETHING INTERESTING. SO BY USING THIS MORE INNOVATIVE APPROACH WHERE WE KEEP THE, THE, UM, THE UPS'S BATTERY ON THAT DC BUS ALL THE TIME, WE CAN ESSENTIALLY RIDE THROUGH THESE VOLTAGE RIDE THROUGH SCENARIOS WITHOUT EVER HAVING A DISCREET SWITCHING EVENT. SO WE'RE NEVER PRESENTING THAT STEP CHANGE IN WHAT THE GRID IS SEEING. SO WHAT THE, UM, THE FIGURE ON THE TOP IS SHOWING. SO IT'S ALSO WORTH NOTING THESE ARE ESSENTIALLY LABORATORY EXPERIMENTS AT THIS POINT BECAUSE THIS TECHNOLOGY HASN'T BEEN DEPLOYED AT SCALE. BUT WHAT WE'RE SHOWING ON THE TOP IS A COMPLEX VOLTAGE DISTURBANCE THAT THAT'S BEING PLAYED IN, IN SIMULATION. UH, AND HERE WE SEE THAT WE HAVE, UM, UH, KIND OF A, AGAIN, A COMPLEX FAULT PROFILE, UH, AND THEN A SHALLOW FAULT FOLLOWED BY A DEEPER FAULT. AND THE PLOT ON THE BOTTOM IS SHOWING WHAT'S HAPPENING TO THE POWER CONSUMPTION OF THE DATA CENTER FACILITY ITSELF, AS LONG AS IT USES THIS MORE INNOVATIVE, UM, UH, POWER DELIVERY TECHNOLOGY. AND WHAT WE SEE IS THAT THE, UM, THE POWER CONSUMPTION OF THE DATA CENTER IS PROPORTIONAL TO THE CHANGE IN VOLTAGE. SO WHAT WE WOULD SAY IN POWER ENGINEERING PARLANCE IS THAT THIS LOAD IS BEHAVING AS A CONSTANT CURRENT LOAD BECAUSE THERE'S A LINEAR RELATIONSHIP BETWEEN THE CHANGE IN VOLTAGE AND THE CHANGE IN POWER. AND THIS IS SOMETHING THAT IS HIGHLY DESIRABLE BECAUSE IT CAN PROVIDE SOME RELIEF DURING STRESS SYSTEM CONDITIONS BY REDUCING THE AMOUNT OF POWER THAT WE HAVE TO PROVIDE AS THE VOLTAGE IS DEGRADED. ALRIGHT, GREAT. SO LET'S RETURN TO THIS TECHNOLOGY COMPARISON SUMMARY. NOW THAT WE'VE TALKED ABOUT SOME OF THESE DIFFERENT TECHNOLOGIES, UM, WHAT WE CAN SEE IS THAT IN TERMS OF THE UPFRONT OR THE ONE-TIME COSTS, THE GRID INTERACTIVE UPS, AS LONG AS, UH, IT CAN BE EXECUTED BY PURELY UPDATING THE FIRMWARE, THAT IS GENERALLY GOING TO BE THE CHEAPEST SOLUTION STRICTLY TO ADDRESS THESE VOLTAGE RIDE THROUGH PROBLEMS. UM, AND THEN WHEN WE TALK ABOUT THE IMPACT ON FACILITY EFFICIENCY, ONE THING TO NOTE HERE IS THAT THIS SERIES BEST CONCEPT, ALTHOUGH IT DOES POTENTIALLY HAVE THE, THE ABILITY TO SIMPLIFY HOW WE REPRESENT THESE, UH, FACILITIES IN SIMULATION, IT COULD ALSO POTENTIALLY BE LESS EFFICIENT BECAUSE IT REQUIRES US TO INTERFACE NOT ONLY THE COMPUTATIONAL LOAD, BUT ALSO THE COOLING LOAD OF THE DATA CENTER THROUGH THAT POWER ELECTRONIC CONVERSION SYSTEM, WHICH CAN INCUR HIGHER LOSSES. SO THAT'S, UH, AN IMPORTANT CONSIDERATION THERE. THE NEXT ONE IS, IS IT POSSIBLE TO RETROFIT THESE TECHNOLOGIES AT AN EXISTING DATA CENTER? AND IN MOST CASES, THE ANSWER IS YES. HOWEVER, WITH THESE, UM, CASES WHERE WE HAVE TO RETHINK AND REDESIGN HOW POWER IS DELIVERED WITHIN THE DATA CENTER ITSELF, PROBABLY NOT. THAT WOULD REALLY BE, UM, A MUCH MORE DRASTIC CHANGE AT A DATA CENTER SITE. SO WE ANTICIPATE THAT, UH, INSTEAD OF BEING INSTALLED IN EXISTING DATA CENTERS, THIS WILL BE, UM, PRESENT IN SYSTEMS THAT WE SEE COME ONLINE, SAY IN THE TWO TO FIVE YEAR KIND OF A TIMEFRAME. AND THEN FINALLY, THE TIMELINE FOR ATSCALE DEPLOYMENT. SO WHICH OF THESE THINGS ARE AVAILABLE TODAY WITH CENTRALIZED BATTERY ENERGY STORAGE? THAT'S PROBABLY FOR SURE THE MOST MATURE ON THIS LIST AND, UH, AND, AND IS ALREADY BEING, UM, BEING USED AT DATA CENTER FACILITIES. UM, WHEREAS SOME OF THESE OTHER THINGS, UM, UH, AGAIN, MIGHT TAKE SEVERAL YEARS. IN THE CASE OF E AT COM, THESE HAVE BEEN DEPLOYED AT DIFFERENT, UH, IN DIFFERENT PARTS OF THE GLOBE, BUT NOT FOR APPLICATIONS ASSOCIATED, UH, WITH DATA CENTERS TO TO DATE. OKAY. UM, AND RETURNING TO THIS QUESTION OF FREQUENCY RIDE THROUGH, UM, THIS IS SOMETHING THAT, UH, IF YOU'D LIKE TO DISCUSS IT MORE IN THE Q AND A SESSION, WE'D BE, WE'D BE HAPPY TO. UM, ONE OF THE THINGS THAT WE'VE LEARNED IS THAT IT'S IMPORTANT THAT DATA CENTER RESOURCES NOT TO BE ALLOWED TO TRIP BASED ON INSTANTANEOUS FREQUENCY MEASUREMENTS, BECAUSE THOSE FREQUENCY MEASUREMENTS CAN BE DISTORTED BY FAULT TYPE CONDITIONS NEAR THE POINT WHERE THE MEASUREMENT IS BEING MADE. AND THIS IS SOMETHING THAT WE REALLY LEARNED IN THE CONTEXT OF INVERTER BASED GENERATION, ESPECIALLY DURING THE BLUE CUT FIRE OF 2016. GREAT. OKAY. SO, UH, IN CONCLUSION, UM, UH, FIRST OF ALL, AGAIN, UH, THANK YOU ALL FOR YOUR TIME. WE REALLY APPRECIATE YOU HAVING US. UM, AND AS WE DISCUSSED, [00:20:01] SO THERE ARE MULTIPLE TECHNOLOGIES THAT ARE AVAILABLE TO HELP DATA CENTERS COMPLY WITH THESE NEW NOUR 2 82 RIDE THROUGH REQUIREMENTS. UM, AS WE'VE DISCUSSED, CENTRALIZED BATTERY ENERGY STORAGE, UH, IS NOT ONLY THE MOST MATURE, BUT IT'S PROBABLY ALSO IN MANY WAYS THE MOST VERSATILE BECAUSE IT GIVES US THE ABILITY NOT JUST TO ADDRESS THE VOLTAGE RIDE THROUGH PROBLEM, BUT ALSO TO ADDRESS THESE ISSUES ASSOCIATED WITH WORKLOAD VARIABILITY, WHICH IS ANOTHER MAJOR, UH, MAJOR POINT OF CONCERN FOR, FOR MANY DATA CENTERS. UM, YES, AND, UH, YEAH, I THINK WE'D, UH, FINALLY LIKE TO JUST CLOSE WITH THE FACT THAT, UH, TODAY'S FACILITIES WILL BE RETROFITTED TO COMPLY WITH THESE RIDE THROUGH REQUIREMENTS, BUT THE FACILITIES THAT WE SEE COME ONLINE IN THE NEXT TWO TO FIVE YEARS WILL BE DESIGNED WITH THESE REQUIREMENTS IN MIND, ESSENTIALLY FROM THE GROUND UP, WHICH IS ANOTHER VERY IMPORTANT REASON THAT THEY EXIST. SO, UM, THANK YOU VERY MUCH AND WE'LL BE HAPPY TO TAKE ANY QUESTIONS. QUESTION, DO YOU HAVE A FEEL FOR WHAT THE, UH, ADDITIONAL COST TO A DATA CENTER IS TO BE, UH, COMPLIANT WITH THAT? IF THEY WERE TO START FROM SCRATCH TODAY? WHAT IS IT A BIG COST, SMALL COST? WHAT IS IT THAT, THAT'S A FANTASTIC QUESTION. SO WE'VE TRIED TO PERFORM A MARKET SURVEY, AND THIS IS DIFFICULT BECAUSE, YOU KNOW, WE'RE JUST, YOU KNOW, GRASPING AT THE INFORMATION THAT'S AVAILABLE IN THE PUBLIC DOMAIN. SO WE DON'T HAVE INFORMATION BASED ON QUOTES FROM DIFFERENT OEMS OR VENDORS OR ANYTHING LIKE THAT. BUT WHAT WE'VE TRIED TO DO HERE IS TO CHARACTERIZE OR TO ESTIMATE HOW MUCH MIGHT IT COST FOR 100 MEGAWATT DATA CENTER TO BECOME COMPLIANT WITH THESE RIDE THROUGH REQUIREMENTS. AND WE THINK THAT AS LONG AS THAT FACILITY ALREADY HAS UPSS AND YOU KNOW, UH, WITHIN IT, THAT IT COULD BE AS LOW AS PERHAPS 1 MILLION, $1 MILLION IN TERMS OF, UM, AND THAT'S REALLY JUST KIND OF THE, THE, THE, UH, EFFORT AND, AND THE EXPENDITURE, UH, TO INSTALL THE NEW FIRMWARE AND TO CHARACTERIZE IT AND TO MAKE SURE THAT THE FACILITY IS OPERATING AS INTENDED. UM, BUT AGAIN, YOU, YOU CAN SEE THAT THAT SPANS QUITE A LARGE RANGE DEPENDING ON THE ADVANTAGES THAT YOU'RE TRYING TO ACHIEVE. AND THAT'S ONE OF THE THINGS THAT'S IMPORTANT TO NOTE ABOUT THIS TABLE IS THAT WE ARE TRYING TO PROVIDE A COST COMPARISON HERE, BUT IT'S NOT ALWAYS APPLES TO APPLES. SO $1 MILLION THAT WE SPENT ON A, ON A GRID INTERACTIVE UPS IS GONNA BE DIFFERENT THAN THAT SAME AMOUNT OF MONEY THAT WE SPEND ON CENTRALIZED BATTERY ENERGY STORAGE BECAUSE THEY HAVE DIFFERENT CAPABILITIES. WHAT DO I JUST ADD? WE'RE WE'RE TRYING TO ARRANGE FOR A SPEAKER TO COME IN WHO IS ACTUALLY BUILDING ONE OF THESE DATA CENTERS. AND, AND WE SHOULD ASK THEM THAT QUESTION AS WELL. IN FACT, WE WILL ASK THEM THAT QUESTION SO THEY CAN ADDRESS IT AS PART OF THEIR PRESENTATION PROBABLY MAYBE DECEMBER OR EARLY NEXT YEAR. YEAH, I GUESS THE QUESTION WAS THE DIFFERENCE IN COST IF I HAVE TO RETROFIT SOMETHING VERSUS IF I WAS DESIGNING FROM SCRATCH, OOH. OR FROM THE VERY BEGINNING I KNEW THAT WAS MY REQUIREMENT. IS IT A BIG, IS THERE A DIFFERENCE IN WHAT THAT WOULD COST A DATA CENTER? THAT'S AN EXCELLENT QUESTION. UM, FRANKLY, MAYBE I DON'T HAVE ENOUGH OF A FEEL FOR THE DYNAMICS, UH, AT, AT, AT PLAY, UM, IN TERMS OF THE, THE BUSINESS AND ECONOMIC SIDE SIDE OF IT. SO I MIGHT WANT TO DEFER YEAH. TO, AS JOHN MENTIONED, YEAH, TO SOME OF THE FOLKS THAT ARE ACTIVELY INVOLVED IN DOING SOME OF THESE BUILD OUTS. I, I THINK WOODY, AS YOU KNOW, THE GENERAL ENGINEERING ANSWER IS IT'S ALWAYS CHEAPER TO DO IT FROM SCRATCH IF YOU, YOU'RE DESIGNING IT FROM THE BEGINNING. BUT GIVEN THAT WE'VE GOT DATA CENTERS THAT ARE ALREADY EXISTING, THAT MAY NOT BE AN OPTION FOR SOME OF THOSE GUYS. BUT I THINK PABLO QUESTION, THANK YOU FOR THE PRESENTATION. THAT WAS REALLY VERY INTERESTING. THE, UM, POWER ARCHITECTURE CHANGE THAT YOU MENTIONED, THE SHIFT TO 800 VOLT EC. YES. HOW QUICKLY DO YOU EXPECT THE INDUSTRY TO GO THROUGH THIS, UH, TRANSITION? WHEN WILL WE START TO SEE THAT, UM, AT SCALE AND THEN, AND THEN A BIT OF A PART TWO, WHAT DO YOU EXPECT THE IMPACT TO BE TO GRID OPERATORS AND UTILITIES SEEING AT SCALE THAT KIND OF A SHIFT? IS THERE GONNA BE MUCH A, FROM A UTILITY SIDE IMPACT ON THAT SHIFT VERSUS WHAT WE'RE SEEING TODAY WITH THE AC ARCHITECTURE? I, I THINK BOTH OF THOSE ARE YEAH. FANTASTIC QUESTIONS. THANK, THANK YOU. UM, ALL THE FOLKS THAT I TALKED TO IN THE POWER ELECTRONICS, UM, REALM, WHETHER, WHETHER IT'S SANDIA OR OR OUTSIDE OF OUR ORGANIZATION, THEY SEEM TO ALL, UM, UH, BE, UH, ALIGNED IN THAT THIS IS WHERE, UH, THE TECHNOLOGY IS HEADING. AND IT'S REALLY JUST A MATTER OF HOW, HOW QUICKLY WE, WE CAN GET THERE. UM, I DON'T HAVE A GREAT FIELD. POWER ELECTRONICS IS NOT MY PRIMARY DOMAIN, SO I DON'T KNOW WHETHER THESE ARE PRIMARY TECHNICAL PROBLEMS OR MANUFACTURING PROBLEMS, THINGS OF THAT NATURE. BUT I, I WOULD EXPECT, YOU KNOW, THE DATA CENTERS THAT WE SEE COME ONLINE PROBABLY IN THREE YEARS, MY EXPECTATION IS THAT THEY WILL BE BUILT YEAH. IN THIS WAY. AND THEN TO YOUR SECOND QUESTION, UM, YES, I DO THINK THAT THERE ARE NUMEROUS ADVANTAGES AS LONG AS THESE THINGS THAT WE SEE IN SIMULATION THAT'S BEING TALKED ABOUT BY THE, BY THE VENDORS OF THESE TECHNOLOGIES, AS LONG AS THIS MANIFESTS ITSELF AS WE SCALE THE DATA CENTER UP TO, YOU KNOW, 100 MEGAWATTS AND ABOVE THESE VERY LARGE, YOU KNOW, KIND OF HYPERSCALE FACILITIES THAT WE'RE SEEING ON THE GRID, UM, THAT THERE WOULD BE BENEFITS INCLUDING THIS, THIS ABILITY TO RIDE THROUGH WITHOUT ANY KIND OF DISCREET SWITCHING ACTION. BECAUSE [00:25:01] THAT'S AN UNDERRATED POINT ACTUALLY, IS THAT, UM, ALL THESE OTHER TECHNOLOGIES THAT WE TALKED ABOUT, THE OTHER WAYS THAT THESE ENERGY STORAGE SYSTEMS WORK, THERE'S SOME KIND OF A DELAY BETWEEN WHEN THE DATA CENTER TRANSFERS TO ITS UPS AND WHEN WE CAN BEGIN CHARGING THOSE BATTERIES. BUT WITH THIS APPROACH, IT SEEMS TO BE, UH, SOMEWHAT MORE SEAMLESS, WHICH I THINK IS ALSO AN ADVANTAGE. SEAMLESS BEING MILLISECONDS OR, UM, UH, YEAH. SO, SO, UH, AGAIN, IT'S, UH, AT LEAST THE, THE EARLY RETURNS FROM THE EXPERIMENTAL RESULTS THAT WE'RE SEEING IS THAT WE HAVE KIND OF A PROPORTIONAL CHANGE SO THAT IT TENDS TO BE MORE DIRECT. AND BECAUSE THERE'S NO SWITCHING, ESSENTIALLY, THERE'S NOTHING TO WAIT FOR, UM, IN TERMS OF, YEAH. HAVING THAT KIND OF RESPONSE DELAY. JULIE, THANKS FOR YOUR PRESENTATION. THANK YOU. UM, VERY TIMELY TOPIC. CAN YOU JUST, UH, REMIND US WHO'S FUNDING THIS RESEARCH? OH, UM, THAT'S A, THAT'S A FANTASTIC QUESTION. SO, UM, I DO LEAD A PROJECT AT SANDIA THAT'S FUNDED BY THE LEGACY GDO OFFICE AT, AT DOE. UH, HOWEVER, UM, UH, THE, THE PRESENTATION THAT WE PREPARED FOR, FOR YOU TODAY, UM, ISN'T A DIRECT WORK OF ANY KIND OF, YOU KNOW, OF SANDIA LABORATORY RESEARCH. UM, REALLY, I THINK THE, THE, UM, THE CONTEXT IN WHICH WE'VE BEEN INVITED TODAY IS ESSENTIALLY TO SERVE AS A KIND OF A NEUTRAL ARBITER OKAY. FOR YEAH. TO CONVEY HOW THESE TECHNOLOGIES WORK. UM, BUT THERE ARE MANY LARGE EFFORTS. SO I THINK ONE OF THE THINGS IS JUST THAT THE SCALE OF THE INVESTMENT IN THIS TECHNOLOGY THAT WE'RE SEEING, RIGHT, WHERE THE HYPERSCALERS HAVE SPENT OVER $3 TRILLION ALREADY IN THEIR NEW COMPUTE CAPACITY, THAT'S ALSO DRIVING A SIGNIFICANT AMOUNT OF INNOVATION IN THE R AND D SPACE THAT'S OUTSIDE OF ANYTHING THAT MIGHT BE GOING ON, UH, FOR EXAMPLE, AT THE NATIONAL LABS. THANK YOU, BILL. THANKS, JOHN. APPRECIATE YOUR, YOUR, UH, PRESENTATION TODAY. LOOKING AT THE DC ARCHITECTURE OPTION, HOW, HOW DEVELOPED IS THE SCIENCE AND TECHNOLOGY ON THAT? SO YOU'VE GOT, YOU'VE GOT TO DEVELOP THE SCIENCE AND TECHNOLOGY, THEN YOU'VE GOTTA BUILD THE EQUIPMENT TO DO IT, THEN YOU HAVE TO IMPLEMENT IT. SO WHERE ARE WE ON THAT FIRST STEP? HOW FAR, HOW MATURE IS THAT? I MEAN, IT SAYS YOU'RE IN SIMULATION LAB TESTING, BUT ARE YOU CLOSE TO THE FINISH LINE SO THAT, UH, SOMEBODY CAN GO BUILD THE EQUIPMENT THAT MAKES THIS WORK? TH TH THIS IS, THIS IS A FANTASTIC QUESTION. SO, SO MY DOMAIN IS ACTUALLY POWER SYSTEM DYNAMICS AND, AND STABILITY. SO I'M TRYING TO UNDERSTAND HOW THE GRID IS GONNA BE AFFECTED BY THESE SORTS OF TECHNOLOGIES. UM, BUT YOU'RE RIGHT. I, I THINK ONE OF THE CHALLENGES IS JUST THE POWER DENSITY, RIGHT? SO WHEN WE TALK ABOUT 800 VOLTS DC OR 1500 VOLTS DC UH, IN, INTO THE RACK, THAT'S SUBSTANTIALLY, UM, UH, IT'S A SUBSTANTIAL INCREASE, POTENTIALLY TWO TIMES, FOUR TIMES, UH, OF THE POWER WITHIN THE SAME FOOTPRINT. MM-HMM . SO WE'RE STILL TALKING ABOUT, YOU KNOW, A STANDARD, YOU KNOW, ESSENTIALLY NINE 19 INCH COMPUTE RACK. UM, SO, UH, YES, THERE ARE, UM, STILL TECHNICAL DIFFICULTIES THAT ARE STILL BEING WORKED THROUGH. UM, AND I THINK THAT, YOU KNOW, PERHAPS SOME OF THE FOLKS THAT COME TO SPEAK TO YOU GUYS MORE FROM THE POWER ELECTRONICS DOMAIN, WHERE THAT'S CORE BUS, THEIR, THEIR CORE BUSINESS MIGHT BE ABLE TO, UH, SAY MORE ABOUT, UM, YOU KNOW, TO WHAT EXTENT ARE THESE PROBLEMS KIND OF FUNDAMENTAL OR ARE THEY MORE ISSUES ASSOCIATED WITH COMMERCE IN THE MANUFACTURING AND PACKAGING OF THESE KINDS OF DEVICES. OKAY. THANK YOU, JOHN, I COULD JUMP IN HERE. PLEASE, PLEASE. UH, MY UNDERSTANDING IS THAT THE SCIENCE TECHNOLOGY IS THERE, UM, FOR THE INVERTER SIDE, FOR THE DC LINES, UM, THAT REALLY, IT'S NOW ON THE, UH, THE RACK ASPECTS. SO AT THE RACK, YOU NOW NEED TO CONVERT DOWN TO THE GPU VOLTAGES. UM, YOU NOW HAVE HIGHER VOLTAGES, SO YOU HAVE TO WORRY ABOUT ARC FLASH IN THE RACKS. UM, SO REALLY IT'S THE, THE SAFETY AND BUILDING OF THE RACKS MORE THAN THE SCIENCE AND TECHNOLOGY ASPECTS. YEAH, I BELIEVE THE MILITARY ALREADY USES SOME, SOME TYPE OF THIS TECHNOLOGY. THEY, THEY'RE, A LOT OF, A LOT OF THINGS IN NUCLEAR SUBMARINES AND STUFF LIKE THAT ARE DIRECT DCS. ANY OTHER QUESTIONS? IF NOT, THANK YOU VERY MUCH AND WE'LL SEE YOU BACK IN EXECUTIVE SESSION. THANK YOU SO MUCH. GREAT. UH, LET US MOVE [5. Notice of Annual Committee Self Evaluation Questionnaire] ON TO AGENDA ITEM FIVE. PURPOSE OF AGENDA ITEM FIVE IS JUST TO DRAW YOUR ATTENTION TO THE FACT THAT WE'RE GONNA HAVE AN ANNUAL COMMITTEE SELF-EVALUATION QUESTIONNAIRE THAT'S GONNA COME OUT FROM THE STAFF SOMETIME REALLY SOON. UM, THIS IS, UM, THE ONE THAT WE TALKED ABOUT AT THE LAST MEETING. UM, IT'LL BE ADMINISTERED BY BRA AND HIS TEAM, AND YOU SHOULD EXPECT TO SEE IT SOMETIME IN THE NEXT COUPLE OF WEEKS. AND WE'RE LOOKING FOR FEEDBACK BY OCTOBER 30TH, AND WE'LL TALK ABOUT IT IN THE DECEMBER MEETING. LET'S NOW MOVE ON TO ANY, ANY QUESTIONS ABOUT, ABOUT THAT? YOU CAN SEE THE MATERIAL IN THE, IN THE BOOK? NO, BUT I GOT A SUGGESTION YES. THAT WE USE AI TO UPDATE THE QUESTIONS. , SERIOUSLY, WHAT, WHAT, WHAT UPDATE ARE YOU LOOKING FOR? WHAT, WHAT, WHAT ARE YOU [00:30:01] HOPING THE WILL DO? SEE THE BEST QUESTIONS BEING ASKED OF TECHNOLOGY AND SECURITY COMMITTEES OF THE BOARD. I, I WILL LEAVE THAT TO THE TEAM TO RUN IT THROUGH THEIR FAVORITE AI TOOL AND SEE WHAT THEY COME UP WITH. LET'S, LET'S MAKE SURE WE DON'T LOSE THE INTENT OF THIS, BUT, UH, SURE. UM, IF WE CAN NOW MOVE ON [6.1 Projects and Technology Update] TO COMMITTEE BRIEF SIX, UM, WE'RE GONNA GO INTO OUR, OUR VARIOUS, UH, REGULAR TECHNOLOGY UPDATES. AND THE FIRST ONE IS MONDY AND NICK, OR SORRY, I'M GONNA, UM, KILL THAT. UM, MONDAY AND NICK ARE GONNA GIVE US A QUICK UPDATE ON A COUPLE. THERE'S ACTUALLY A COUPLE OF TOPICS HERE. THEY'RE GONNA, WE'RE GONNA START WITH, UM, THE PROJECT AND TECHNOLOGY UPDATE, AND THEN WE'RE GONNA GO INTO SOME VERY SPECIFIC TOPICS ON THE USE OF, UM, TECHNOLOGY FOR OUR CRR AND, UM, AND SOME TIME ON DEVIATIONS THAT WE'VE SEEN IN THE DAY AHEAD MARKET. SO, BONNIE. ALRIGHT, GOOD MORNING. UM, SO AS STATED, I'LL DO OUR STANDARD, UH, COVERAGE OF SOME BASIC PROJECT UPDATES, AND NICK WILL PROVIDE SOME ADDITIONAL INFORMATION ABOUT OUR IDENTITY AND ACCESS MANAGEMENT PROGRAM. OKAY. UM, AS WE'VE BEEN TALKING ABOUT THE LAST FEW MEETINGS, AND I THINK YOU'RE USED TO SEEING THIS CHART, UH, THAT EARLY YEAR SURGE OF PROJECT STARTS AND ACTIVITY HAD TO DO WITH THE LABOR THAT WAS BECOMING AVAILABLE AFTER RTC. UM, WE HAD REALLY THAT SURGE OF, OF PROJECT INITIATIONS AT A LEVEL WE DON'T TYPICALLY SEE. BUT SINCE THEN WE'VE SEEN THAT STABILIZE AND WE'RE BACK TO OUR, OUR KIND OF MORE TYPICAL LEVELS OF WHAT WE WOULD EXPECT TO START IN A MONTH IN TERMS OF PROJECTS. SO WE REALLY ARE STABILIZED AND SUSTAINING AT THE, WHAT I GUESS I'LL LOOK AT IS OUR NEW NORM OF AROUND 80 TO 85 ACTIVE PROJECTS RUNNING, UH, CONCURRENTLY. UM, AND WORKING AROUND THAT AND JUST STANDARD WHAT WE'RE SEEING IN TERMS OF CAPACITY AVAILABLE FROM RESOURCES. UH, IF YOU THINK ABOUT THE SHIFT OF THAT LABOR THAT BECAME AVAILABLE, I KNOW LAST TIME WE WERE HERE, UM, TROY DID SOME COVERAGE ON HOW WE WERE ADDRESSING OUR, UM, AGING REVISION REQUESTS. SO WE ARE SEEING SOME OF THAT WORK PICKED UP HERE, BUT IF WE LOOK AT SOME OF THE HEAVY HITTER PROJECTS THAT ARE CONSUMING THAT LABOR THAT BECAME AVAILABLE, IT'S THOSE THAT WE'VE ALREADY HIGHLIGHTED AS OUR STRATEGIC INITIATIVES THIS YEAR. SO THE IAM PROGRAM FOR ONE IS A NUMBER OF PROJECTS THAT WE'VE GOT HIGH LABOR ON. WE'VE ALSO GOT, UM, SOME MODERNIZATION EFFORTS AROUND OUR, UH, INTEGRATION APPLICATIONS THAT WE USE, OUR IT SERVICE MANAGEMENT PRODUCTS, UM, AND OUR WEBSITE REDESIGN THAT WE'VE SPOKEN WITH YOU GUYS ABOUT AS WELL IN THE PAST. SO, UH, THAT'S REALLY WHERE WE'RE SEEING THAT, THAT CONCENTRATED EFFORT SHIFT, BUT OF COURSE A HIGH DEGREE OF PROJECTS. SO WE'RE SEEING IT SPREAD ACROSS OTHER EFFORTS AS WELL. OKAY. AND A MOMENT WE'VE BEEN WAITING FOR, FOR SOME TIME NOW, IS, UH, JUST LOOKING AT KIND OF THE, THE COMPLETION AND THE WRAP UP OF THE RTC PROGRAM. SO, UM, TO REWIND A BIT, UH, THE PROGRAM LAUNCHED LATE JUNE OF 2023, AND AT THAT POINT IN TIME, WE HAD AN EARLY ESTIMATE THAT BALLPARK THIS AT $50 MILLION, UM, A GOAL LIVE THAT WAS LIKELY TO HIT MID 2026 AND AN OVERALL DURATION OF ABOUT THREE AND A HALF YEARS. FAST FORWARD, WE COMPLETED THE PLANNING THROUGH THE PROGRAM. UM, WE WERE IN LATE 2024, WE BASELINED IN ON OUR GO LIVE DATE, UM, AND OUR BUDGET. AND AT THAT POINT IN TIME, WE SET THE GO LIVE DATE TO DECEMBER 5TH, 2025 AND OUR BUDGET TO THE $34.9 MILLION VALUE. UM, THEN, AS YOU GUYS KNOW, WE HAD THE ON TIME GO LIVE DELIVERY IN DECEMBER, AND HERE IN THE END WE'RE COMING IN AND WE WILL COME IN ABOUT 4% UNDER BUDGET. SO ON TIME UNDER BUDGET, WE'LL BE, UH, WRAPPING THIS UP WITHIN THAT ORIGINAL ESTIMATED DURATION OF THE THREE AND A HALF YEARS. THERE, THERE WERE SOME, UM, THERE'S WORK THAT NATURALLY LAGS AS WE THINK ABOUT THE TIMING AND CADENCES OF OUR REPORT CYCLES THAT POST 180, 80 DAYS OUT, FOR INSTANCE. SO, UM, SOME OF THAT WORK DID CARRY OUT, BUT, UM, OVERALL I THINK, YOU KNOW, IT'S BEEN SUCH A SUCCESS IN TERMS OF WHAT THAT ESTIMATE WAS WHEN WE LAUNCHED THE PROGRAM VERSUS WHAT WE'RE ABLE TO DO WITH THAT. AND, YOU KNOW, I JUST WANTED TO HIGHLIGHT A FEW KEY POINTS AROUND THAT, THAT WE COULDN'T HAVE DONE THAT WITHOUT IT BEING THE NUMBER ONE REALLY FOR THE ORGANIZATION SUPPORTED BOTH EXTERNALLY AND INTERNALLY. UM, THE ORGANIZATION, INCLUDING YOU ALL, WAS WILLING TO MAKE CHANGES, UM, TO PRIOR OTHER PRIORITIES THAT WERE JUST NOT AS HIGH OF A PRIORITY OR, OR OTHER TRADE-OFFS THAT HELPED US KEEP THIS CENTERED AS THE NUMBER ONE. AND THAT MADE A BIG DIFFERENCE. IT WE, YOU KNOW, WITH ABOUT 200,000 HOURS OF, OF LABOR ON THIS, NOT [00:35:01] COUNTING THE VENDOR WORK THAT, UM, WE WERE REALLY ABLE TO BE CONCENTRATED AND FOCUSED ON IT. SO THAT WAS ONE THING. UM, I THINK THE OTHER THING IS REALLY THE TONE FROM THE START WAS, OKAY, IT LOOKS LIKE THIS WILL GO INTO 2026, LET'S SET THE MINDSET, WE'RE GONNA GET THIS INTO 2025. SO THAT BECAME A, A THEME AROUND WHICH PLANNING DECISIONS WERE MADE. UM, WE LOOKED AT WAYS TO JUST ATTACK IT AS BEST WE COULD AND BRING THAT DATE IN, RIGHT. AND KEEP IT, UM, AS ACCELERATED AS WE COULD INTO 2025. UM, IT HELPED THAT OUR BUSINESS FOLKS IN THE MARKET HAD BEEN THINKING ABOUT THIS FOR QUITE SOME TIME LEADING UP TO STARTING THE PROGRAM. SO THEY WERE ALREADY HEAVILY INVESTED IN WHAT THOSE REQUIREMENTS MIGHT BE. SO THAT WAS ALSO A VERY GOOD JUMPSTART. AND AT THE END OF THE DAY, OBVIOUSLY WAS A, A MAJOR PUSH BY A LOT OF PEOPLE. SO, UM, WE WERE ABLE TO HIT THAT DATE AND IT, IT WAS NOT WITHOUT, UH, HEAVY HUMAN SUPPORT, RIGHT? SO, UH, WE CERTAINLY APPRECIATE THAT FOLKS WENT ABOVE AND BEYOND AND, AND PUT IN THE EXTRA EFFORT AND THAT IT'S NOT JUST IN MARKET. THAT'S, OR INTER COT THAT'S OUTSIDE WITH, UH, MARKET PLAYERS AS WELL GETTING THROUGH THEIR TESTING SO THAT WE COULD HIT THAT DATE. UM, SO THOSE, THOSE WERE THE TOPICS I WANTED TO COVER FOR RTC. JUST TO COMMENT, IF I MAY. YEAH, I MEAN, THIS IS A VERY IMPRESSIVE ACHIEVEMENT AND, AND THE TEAM REALLY DESERVES A LOT OF CREDIT FOR GETTING IT DONE ON TIME AND UNDER BUDGET. HOWEVER, FROM A BUSINESS POINT OF VIEW, MR. KEITH, WHAT WE NEED TO BE ABLE TO DO IS SHOW THAT THERE WAS BUSINESS VALUE THAT HAPPENED WITH ALL THIS. WE HAD A, WE HAD A HYPOTHESIS OF WHAT THIS WOULD DO IN TERMS OF SAVINGS AND OPTIMIZING THE GRID. AND, AND I THINK WE NEED TO JUST KEEP, MAKE SURE WE MEASURE THAT AND MAKE SURE WE LOOK AT BACK AT OUR ORIGINAL ASSUMPTIONS. SO NOT FOR TODAY'S MEETING, BUT AS YOU THINK ABOUT, YOU KNOW, GOING FORWARD, IF YOU COULD JUST INCLUDE THAT IN YOUR, IN YOUR PRESENTATION. ABSOLUTELY. I THINK WE ARE GONNA COVER SOME OF IT TODAY ACTUALLY. WE HAVE SEEN BITS AND PIECES AND SOME OF THE PRESENTATIONS I'LL BE GIVING, I WILL BE NOTING THE BENEFITS OF RTC, BUT I, I HEAR YOUR POINT. SOMETHING MORE COMPREHENSIVE. YEAH, A LITTLE MORE COMPREHENSIVE. I, I KNOW YOU MENTIONED IT IN A FEW PLACES AND THAT'S GREAT, BUT YEP. OKAY. THANKS MANDY. YEP, CHAIRMAN. AND I THINK THAT'S A GREAT, UH, WAY TO POINT OUT THE TRANSITION NOW FROM WHAT WAS A PROJECT OR A PROGRAM INTO THE ONGOING OPERATIONS AND HOW THEY'LL CONTINUE TO LOOK AT THAT GOING FORWARD. OKAY. AND THEN COMING UP, WE'VE GOT OUR ONGOING RELEASES. UH, A FEW OF THESE ARE ACTUALLY FROM THE AGING REVISION REQUEST BACKLOG. SO, YOU KNOW, AS TROY TALKED ABOUT LAST TIME, WE'RE CONTINUING TO BURN DOWN AGAINST THAT LIST. UM, AND THEN THE FINAL ONE HERE, THE I AM STAKEHOLDER LOGIN PORTAL IS A EXTERNALLY FACING CHANGE THAT'S, UM, PART OF WHAT NICK'S GONNA SPEAK WITH YOU ABOUT. SO I'LL PASS IT OFF TO NICK ALL WITH ME. ALL RIGHT, THANK YOU VERY MUCH. ALRIGHT. UM, SO A VERY, VERY BUSY SLIDE. SO OBVIOUSLY YOU'VE HEARD ABOUT THE I AM PROJECTS AND PROGRAM THAT WE HAVE RUNNING AND IT, IT'S TOUCHING ALL FACETS, BUT I THINK THE THING THAT IS PROBABLY THE MOST INTERESTING IS THE ONE THAT GOES TO OUR MARKET PARTICIPANTS, OUR OUR STAKEHOLDERS. AND BASICALLY WHAT WE LIKE TO REFERENCE IS B2B OR BUSINESS TO BUSINESS. AND SO THIS IS REALLY KIND OF OUR TIMELINE OF HOW WE'RE GOING AND HOW WE'RE PROGRESSING. AND REALLY THE GOAL IS TO GET AWAY FROM DIGITAL CERTIFICATES THAT EVERYBODY HAS ON THEIR MACHINES AND GO TO SOMETHING THAT IS A MORE MODERN, UM, LOGIN EFFORT, UH, SIMILAR TO EVERYTHING THAT YOU USE ON YOUR PHONE WHERE YOU GET LITTLE CODES, ET CETERA, AND THAT KIND OF STUFF. AND REALISTICALLY, GIVE OUR STAKEHOLDERS THE ABILITY TO LOG INTO OUR SYSTEM ONCE AND TRANSACT FOR ALL OF THEIR BUSINESS WITHOUT HAVING TO LOG IN AND OUT, IN AND OUT FOR ALL THE DIFFERENT THINGS THAT THEY DO WORK FOR. SO IT, IT IS THAT, AND THEN ON THE SAME TOKEN, PUT IN SOMETHING THAT IS MODERN IS USING THE MOST MODERN TECHNOLOGY. AND HONESTLY, WE'VE ALL HEARD ABOUT AI, THESE SOLUTIONS, THESE LARGE COMPANIES THAT WE INVITE TO BE OUR LOGIN WILL, WILL BASICALLY HELP US WITH AI AND, AND THE POTENTIAL THREATS THAT WE CAN GET FROM THAT BECAUSE THEIR COMPANIES ARE ALREADY WORKING TO DO THAT. THEY'RE WATCHING THESE THINGS, THE SYSTEMS WILL BE ABLE TO GO OUT THERE AND CHECK AND VERIFY, WELL, SOMEBODY JUST LOGGED IN FROM TEXAS AND THEN 30 MINUTES LATER THEY'RE LOGGING IN FROM CHINA PROBLEM, RIGHT? SO THAT'S WHAT WE'RE AFTER, RIGHT? WHERE DIGITAL CERTIFICATES DON'T DO THAT FOR US, BUT THESE MODERN SYSTEMS ALLOW THOSE KINDS OF THINGS, PLUS WHATEVER ELSE COMES DOWN THE PATH OVER THE NEXT COUPLE YEARS. SO THAT'S WHAT WE'RE GOING INTO. SO WHAT WE'VE DONE IS WE HAVE, OBVIOUSLY WE'VE GOT DELOITTE IN, UH, AND THEY'RE HELPING US DO THE IMPLEMENTATION. WE HAVE, AND I, AND I WON'T USE THE NAME TOO MUCH, BUT EVERY ONCE IN A WHILE I WILL, WE ARE USING [00:40:01] PING, WHICH IS ONE OF THE TOP VENDORS FOR AUTHENTICATION AS OUR METHOD TO AUTHENTICATE ALL OF OUR STAKEHOLDERS. AND SO THIS IS KIND OF WHERE WE ARE IN A TIMELINE. SO, UH, LET'S SEE HERE. SO SEPTEMBER, SO WE HAVE WHAT ARE CALLED EARLY TESTERS, AND THESE ARE PEOPLE WITHIN THE STAKEHOLDERS THAT HAVE VOLUNTEERED. AND EVERY TIME I'VE GONE TO ANYTHING, WE'VE ASKED FOR STAKEHOLDERS TO VOLUNTEER AND COME IN AND HELP US. AND WHAT WE'RE REALLY TRYING TO DO IS DO THIS AS A COLLABORATIVE PROJECT. THERE ARE A LOT OF USE CASES. WE HAVE ABOUT 35,000 UNIQUE USERS THAT LOG IN. AND SO LOTS OF DIFFERENT USE CASES AMONGST THESE MARKET PARTICIPANTS AND STAKEHOLDERS. AND SO IT'S REALLY JUST GOING OUT THERE AND GETTING THESE PEOPLE WHO HAVE A VESTED INTEREST, GET THEM TO LOG INTO THE SYSTEM, SET UP THE MFA, HOW DID THAT WORK, FIND OUT FROM THEM, HOW DOES THIS WORK? DID THIS WORK WELL FOR YOU? IS THERE SOMETHING ELSE WE NEED TO CHANGE BEFORE WE CONTINUE TO GO DOWN A PATH? AND SO IT'S REALLY MORE OF THIS AGILE DELIVERY SYSTEM VERSUS JUST A GREAT BIG OLD WATERFALL CLIFF AT THE END, RIGHT? SO WE'RE TRYING TO DO THIS. SO RIGHT NOW WE'VE UH, SET UP THE FIRST SET OF MEETINGS FOR OUR STAKEHOLDERS. UH, WE'VE ALREADY TALKED TO THEM, WE'VE KIND OF GIVEN THE STUFF. UM, NEXT, PROBABLY LATER ON THIS MONTH, WE'RE ACTUALLY GONNA GIVE THEM ACCESS INTO OUR SYSTEM, OUR MOAT SYSTEM, SO THEY CAN THEN LOG IN AND START DOING THINGS AND START JUST CHANGING THEIR PASSWORDS AND, AND CHANGING THEIR MFA AND THAT KIND OF STUFF JUST SO THEY CAN GET KIND OF A NICE LOOK AND FEEL BEFORE WE PUT ANYTHING IN IT. AND THEN WE ARE ACTUALLY GOING TO, IN OCTOBER, WE'LL GO LIVE WITH THAT SYSTEM TO LOG IN. NOTHING WILL BE BEHIND IT, BUT THE SYSTEM WILL BE THERE. UM, AND THEN WE'RE ACTUALLY GOING TO GO LIVE WITH, THIS IS KIND OF NEW NEWS, UH, POWER GADS. SO THAT IS A NET NEW APPLICATION FOR US, AND WE'RE GONNA PUT IT IN THIS SYSTEM. UH, THE POWER GADS TEAM IS VERY EXCITED ABOUT IT. AND SO WE'LL GET MAYBE 40 TO 80, 60 A HUNDRED, I'M NOT SURE USERS. WHAT THAT'LL ALLOW US TO DO FOR A COUPLE MONTHS IS ACTUALLY SEE THAT PROCESS, SEE PEOPLE LOGGING IN AND REALLY KIND OF DETERMINING HOW IT WORKS FOR A VERY SMALL USER SET, LIKE I SAID, 35,000 USERS. THAT'S A MASSIVE USER SET, BUT IF WE CAN START KIND OF LIKE GOING THROUGH IT WITH 80 TO A HUNDRED PEOPLE, THEN WE CAN MAKE SURE THINGS ARE WORKING PROPERLY AND THAT KIND OF STUFF. THEN IN FEBRUARY, WE'RE GONNA GO LIVE WITH A, UH, MARKET PARTICIPANT SERVICE PORTAL. AND SO THAT'S GONNA BE A BIGGER APPLICATION, RIGHT? SO THAT ONE WILL HAVE A BIGGER IMPACT. BUT THE COUPLE OF MONTHS BEFOREHAND, WE'VE BEEN TESTING IT, WE'VE HAD SOMETHING LIVE, RIGHT? PEOPLE HAVE BEEN LOGGING IN AND DOING THINGS, SO THAT'S GONNA BE GOOD FOR US. THEN THROUGHOUT 2027 AND PERHAPS EVEN INTO 2028, WE'LL ACTUALLY START INGESTING ALL THE OTHER APPLICATIONS BEHIND IT THAT WE HAVE. AND SO WHAT WE'RE GONNA DO, SO THOSE STAKEHOLDERS, RIGHT, THOSE, THOSE EARLY TESTERS WILL BE TESTING IT EVERY SINGLE STEP, RIGHT? AND SO IT'S GONNA BE THE PEOPLE THAT USE APIS, IT'S GONNA BE THE PEOPLE THAT USE ONE OF THESE APPLICATIONS. THEY'LL ACTUALLY GET TO LOG INTO IT IN THE MOAT ENVIRONMENT, MAKE SURE EVERYTHING'S WORKING FOR THEM, MAKE SURE THAT THINGS ARE GOING PROPERLY. AND THEN PART WITH THE STAKEHOLDER, THESE EARLY TESTERS IS IT'S A BACK AND FORTH RELATIONSHIP, RIGHT? THIS IS A COLLABORATIVE EFFORT FROM START TO FINISH. AND SO THIS IS KIND OF HOW WE'RE APPROACHING THIS ENTIRE THING. ANY QUESTIONS ON THIS? ALL RIGHT, LET'S SEE. AH, IT WORKED THE FIRST TIME I HIT IT. UM, PART OF THIS THOUGH, WHAT WE HAVE TO DO IS, UM, DO A LOT OF ENGAGEMENT AND TRAINING AND THAT KIND OF STUFF. AND SO ONE OF THE THINGS I REALLY WANTED WHEN WE CAME HERE, WHEN I CAME HERE TO DO THIS PROJECT WAS I WANTED ORGANIZATIONAL CHANGE MANAGEMENT. I'VE SEEN IT WORK AND IT'S ALL ABOUT CONVINCING PEOPLE TO DO SOMETHING NEW. 'CAUSE NOBODY WANTS TO DO SOMETHING NEW. UM, CONVINCE THEM TO DO SOMETHING NEW IN A MANNER THAT IT'S NOT FORCED UPON THEM. IT'S A VERY MUCH, THESE ARE THE BENEFITS, THIS IS WHAT WE'RE GOING TO GET, RIGHT? A LOT OF TIMES WHEN I TALK TO PEOPLE, ESPECIALLY THE USAS AND I TELL THEM, YOU'RE NOT GONNA HAVE TO LOG IN A HUNDRED TIMES IF YOU'RE ONE OF THE BIG COMPANIES, RIGHT? YOU'RE NOT GONNA HAVE TO MANAGE A THOUSAND DIGITAL CERTIFICATES. YOU CAN JUST TELL YOUR EMPLOYEES JUST TO GO, GO TO A PORTAL, LOG IN, AND THEN THE USA LOGS IN BEHIND THEM AND APPROVES THAT ACCESS. AND THEY'RE LIKE, I DON'T HAVE TO DO ANYTHING WITH DIGITAL CERTIFICATES ANYMORE. AWESOME. RIGHT? AND THAT'S WHAT OCM IS. IT IS REALLY THAT WHAT'S THE BENEFIT TO YOU? WHAT'S THE BENEFIT TO US AND HOW DO WE MAKE SURE THAT EVERYBODY'S EXCITED ABOUT THOSE BENEFITS? SO, UM, WE'VE GOT THE EARLY ADOPTERS IN THAT BLUE GOT MARKET PARTICIPANTS, UM, AND THEN WE'VE GOT EXTERNAL CONSUMERS AS WELL. THERE ARE SOME THINGS OUT THERE THAT THE EXTERNAL CONSUMERS, PEOPLE IN THE STATE OF TEXAS GET ACCESS TO. UM, A LOT OF IT IS SOMETIMES IT'S THE TEXANS THINGS. [00:45:01] SOMETIMES THERE'S SOME VIDEO TRAINING AND THINGS LIKE THIS. IT'S HOW DO WE SECURE THAT? AND SO THAT'S THAT B2C STUFF, IT'S A, IT'S NOT AS MUCH RIGHT NOW, BUT WHAT WE WANTED TO DO IS BECAUSE THINGS EVOLVE, RIGHT? IF THE LEGISLATURE OR THE PUC OR SOMEBODY ELSE OVER THE NEXT FIVE, 10 YEARS WANTS US TO DO SOMETHING ELSE, RIGHT? WE HAVE A SYSTEM ALREADY IN PLACE TO DO THAT SO WE CAN REACT QUICKLY AND DO WHAT WE NEED TO DO FOR ERCOT. SO THIS IS REALLY JUST, IT, IT'S UM, KIND OF, YOU CAN SEE THE PLANS THAT WE'RE GONNA DO A LOT OF COMMS, UM, A LOT OF THOSE KINDS OF THINGS. AND WE'RE REALLY JUST GONNA INTERACT WITH EVERYBODY. AND YOU'LL SEE IN THERE ON THE MARKET PARTICIPANTS, WE'RE GONNA HAVE JOB AIDS AS WE INGEST EVERY NEW APPLICATION. NOT ONLY DO WE HAVE THE EARLY TESTERS GET INVOLVED AND TEST FOR US, UH, WE HAVE A LOT OF COMMUNICATIONS, BUT THEN WE ALSO HAVE JOB AIDS THAT ROLL OUT. AND THE JOB AIDS ARE, ARE VERY ROBUST. UM, I'VE SEEN A COUPLE OF THEM. AND , WHEN THEY SAVE THEM IN THE PDF, IT'S A VERY LARGE FILE. SO YOU CAN TELL THERE'S A LOT OF GOOD PICTURES TO REALLY KIND OF HELP. SO THAT'S, THAT'S REALLY OUR APPROACH. AND SO THAT'S KIND OF HOW WE'RE GOING FORWARD AND MARCHING TOWARDS SUCCESS FOR THE MARKET PARTICIPANTS, OUR STAKEHOLDERS, AND ERCOT AS A WHOLE. SO, AND I ONLY HAD TWO SLIDES. SO NICK, I I HAVE A QUESTION FOR YOU. I'M A LITTLE, LITTLE CONFUSED BY SORT OF THE END DATE HERE. IT LOOKS LIKE FROM YOUR GANTT CHART THAT YOU'RE SORT OF EXPECTING TO BE MORE OR LESS DONE BY FIRST QUARTER OF NEXT YEAR, AND YET YOU DON'T PLAN TO DISCONTINUE THE OLD TECHNOLOGY UNTIL 28. WHAT'S, WHAT'S THE DIFFERENCE BETWEEN THAT? WHAT'S THAT GAP? YEAH, THE, THERE'S A PLUS THERE. I DON'T THINK THAT'S PROBABLY THE MOST APPROPRIATE WAY OF SAYING THAT. UH, THAT'S WHY THERE'S A PLUS RIGHT AFTER THE 27. OKAY, I SEE THE, YEAH, IT JUST KEEPS ON GOING. BUT YEAH, I, I, YEAH. SO, SO WE SHOULD THINK ABOUT JAN THROUGH 27 BASICALLY IS THE, IS THE FINAL ROLLOUT AND IT, IS IT GO APPLICATION BY APPLICATION? IS THAT, YEAH, WE'RE GONNA BATCH APPLICATIONS BASED UPON WHAT MAKES THE MOST SENSE AND HOW WE CAN INGEST THEM AS A WHOLE WITH THE HIGHEST PRIORITY APPLICATIONS FIRST IN THE LOWEST. SO, SO THERE WILL BE A LONG TAIL OF, YOU KNOW, YES, SLIGHTLY OR LIGHTLY USED APPLICATIONS THAT, SO BASICALLY WE ARE CLUSTERING THEM INTO WAVES AND BASED ON WHAT CAN BE, YOU KNOW, THE LEAST RISK, UM, MOVE THAT SO THAT WE IMPLEMENT INTEGRATION INTO WAVES. SO THAT'S WHAT YOU SEE IN CYCLES. OKAY, I SEE. BUT WITH THE MAIN PLATFORM WE STOOD UP, UM, THE EASY ONES, WE INTEGRATE FIRST GET THE MARKET THE MOST VALUE, AND THEN THE HARDER ONES LIKE RUNNING THE GRID, THE MARKET, THOSE WILL BE AS, AS, AS THINGS STABILIZE AND WE UNDERSTAND IT BETTER. OKAY. I GOT IT. THANK YOU. ALRIGHT, ANY OTHER QUESTIONS FOR NICK? YEAH, BILL, UH, TWO QUESTIONS. UH, ONE IN THE LOWER RIGHT HAND CORNER, YOU'VE GOT A BOX CALLED HYPERCARE. WHAT IS THAT? UH, SO HYPERCARE IS REALLY, UM, IT'S, SO IT'S POST GO LIVE SUPPORT AND, AND IT'S CALLED HYPERCARE. BECAUSE TYPICALLY WHAT YOU DO IS YOU HAVE PROJECT TEAMS ALL JUST LIKE WATCHING IT VERY, VERY CLOSELY. AND YOU THINK ABOUT YOU HAVE SLAS, RIGHT? AND SO THINGS COULD BE LIKE A TWO DAY, THREE DAY, FOUR DAY SLA AND HYPERCARE. IT'S NOT, IT'S A MATTER OF HOURS IS OUR SLA. SO IT'S BASICALLY EVERYBODY JUMP ON BOARD, EVERYBODY FOCUSED, THIS IS ALL I'M DOING TODAY IS SOLVING THIS PROBLEM. AND SO THAT'S WHY IT'S CALLED HYPERCARE. AND MY SECOND QUESTION IS, SO LET'S ASSUME THAT, UH, YOU GET THIS COMPLETED IN FEBRUARY OF NEXT YEAR, AND THEN QUANTUM COMPUTING IS SOLVED, BECOMES, UH, USABLE THE NEXT DAY. HOW, HOW RESISTANT IS THIS TO THE, UH, ABILITY OF QUANTUM COMPUTING TO, TO, UH, PENETRATE A SYSTEM LIKE THIS? WELL, YEAH. IS IS YOUR, IS THAT AN ENCRYPTION QUESTION BILL? YEAH. UM, SO BECAUSE WE'RE GOING WITH PING, RIGHT? AND THAT'S THE LARGEST, BIGGEST BADDEST BOY OUT THERE RIGHT NOW. IT'S PING AND OKTA ARE LIKE THE BIGGER ONES, THE BIGGEST ONES, AND WE'VE GOT OKTA GOING IN INTERNALLY, UM, PING BECAUSE WHAT THEY'RE DOING AND BECAUSE THAT'S THEIR FOCUS AND BECAUSE THEY HAVE ENTIRE TEAMS SET ON THAT. UM, IF QUANTUM COMPUTING GOES LIVE IN APRIL, LIKE YOU SAID, RIGHT, LIKE RIGHT AFTER THAT, UM, THEY WILL HAVE TEAMS THAT ARE ALREADY ON IT AND ALREADY PREPARED TO DO WHAT THEY NEED TO DO TO CHANGE OUR SYSTEMS, IMPROVE OUR SYSTEMS, AND THAT KIND OF STUFF. AND THEN WE WOULD BE IN A BETTER POSITION TO ACCEPT IT, DO IT, AND INTEGRATE IT WITH US BECAUSE WE'RE DOING THE BIGGER THE BAD SYSTEMS. OKAY. SO PING IS SORT OF THE LINK, THE THE CRITICAL COMPONENT YEAH. OF THE VENDOR TO PROTECT THE ENCRYPTION. YEAH. RIGHT. AS AS TECHNOLOGY EVOLVES, CORRECT. 100%. YEP. AND IT'LL BE A, I CAN SHOW YOU THAT QUANTUM IS NOT GOING LIVE NEXT YEAR OR THE YEAR AFTER, MAYBE 30. UH, AND SECONDLY, [00:50:01] I'M NOT SURE, I, I THINK THAT MOST OF THESE VENDORS ARE, ARE READYING THEMSELVES WITH WHAT THEY CALL QUANTUM RESISTANT, UM, ENCRYPTION. SO YEAH, 100%. YEP. AND THAT'S WHY WE WANTED THE BIG BOYS INSTEAD OF LIKE THE, THE GUY THAT JUST STARTED UP LAST MONTH. OKAY. SO WE'VE GOT THAT IN THE BACK OF OUR MINE OR IN, OR IN THE FOREFRONT OF OUR, OUR PLANNING. OKAY, THANK YOU. YEAH. AND, AND AG AI AND EVERYTHING, IT'S ALL SITTING THERE AND THAT'S WHY WE'RE EMBRACING THESE GUYS. YEP. ANY OTHER QUESTIONS FOR NICK? OKAY, WELL THANK YOU VERY MUCH. WELL, THANK YOU. I APPRECIATE THE, UH, OPPORTUNITY. OKAY, WE'RE [6.2 Cloud-Based GPU Solution to Improve Congestion Revenue Rights (CRR) Application Performance - Implementation Update] GONNA MOVE ON TO TWO RELATED BUT SLIGHTLY DIFFERENT, UH, TOPICS. UM, ONE, ONE AS IT RELATES TO, UM, HOW WE'RE GONNA IMPROVE PERFORMANCE OF OUR CRR RIGHTS CONGESTION, REVENUE RIGHTS, UM, APPLICATION, WHICH AS YOU'LL RECALL FROM PRIOR MEETINGS, HAS BEEN A LITTLE BIT OF A CON A PROBLEM FOR THE LAST COUPLE OF YEARS. UM, AND, AND WE'VE IMPLEMENTED SOME NEW, SO WE'VE BEEN WORKING WITH THE VENDOR TO IMPLEMENT SOME NEW SOFTWARE. WE'VE IMPLEMENTED SOME NEW TAR HARDWARE TECHNOLOGY, UM, AND IT LOOKS LIKE WE'RE GONNA HAVE TO DO SOME MORE THINGS TO KEEP UP WITH THE DEMAND. THIS, THIS, THIS, AS YOU'RE, YOU'LL SEE FROM THE MATERIAL, THIS IS DRIVEN LARGELY BY DEMAND FOR PEOPLE WANTING TO, YOU KNOW, TRADE CRR RIGHTS. GOOD MORNING EVERYONE. ALFREDO AND CVAS ARE GONNA BE THE PRESENTERS. SORRY, I SHOULD HAVE INTRODUCED YOU . THANK YOU CHAIR. UH, YES, GOOD MORNING. UH, MY NAME'S ALFREDO MORENO. I'M THE MANAGER FOR FORWARD MARKETS, UH, WHICH IS, INCLUDES THE CONGESTION, REVENUE RIGHTS COR MARKET. UM, SO I, I WILL GET INTO A LITTLE BIT OF THE BACKGROUND JUST TO KIND OF SET EVERYTHING UP ON, UH, THE CLOUD-BASED GPU SOLUTION, UH, WHICH SBA WILL, UH, GET INTO A LITTLE BIT MORE OF DETAILS. BUT I, I WILL SET IT UP BY TALKING A LITTLE BIT ABOUT THE BACKGROUND AND, AND KIND OF WHERE WE'RE AT NOW BASED ON, UH, THE OBSERVATIONS THAT WE'VE BEEN HAVING SINCE EARLY 2024 IN OUR PERFORMANCE ISSUES. SO I, I WANT TO START OFF, UH, BY REALLY TALKING ABOUT THE CR PARTICIPATION. UM, THE CR MARKET PARTICIPATION HAS INCREASED. I CAN, YOU CAN SEE HERE ON THE VISUAL ON THE RIGHT SIDE FOR THE LAST DECADE. UH, THE NOTE HERE IS THAT WE ARE NEARLY TRIPLED SINCE 2016, BUT AS OF TODAY, WE ACTUALLY HAVE, UH, TRIPLED. SO, UM, BASED ON THE, THE UPDATED NUMBERS THAT, UH, THAT WE ARE SEEING. SO, UH, I DID WANT TO TALK A LITTLE BIT ABOUT SOME OF THE MILESTONES OR SOME OF THE THINGS THAT WE HAVE OBSERVED. WE WANNA LOOK BACK AT, UH, OCTOBER, 2019. AND, UH, BEFORE I KIND OF JUMP INTO THAT, I DO WANT TO SET IT UP, UH, TO, TO MAKE SURE THAT, UH, WE'RE, WE'RE ALL AWARE. SO THE CRR MARKET, UH, WE DO AN ANNUAL ALLOCATION FOR UP TO THREE YEARS IN ADVANCE, AND THEN WE DO A MONTHLY AUCTION AND A LONG TERM AUCTION EVERY SINGLE MONTH. SO WHAT WE'RE GONNA DO IS WE'RE GONNA FOCUS ON THE LONG-TERM AUCTIONS, WHICH ARE SEQUENCE AUCTIONS. AND THEY ARE, THEY ARE AUCTIONED OFF IN SIX MONTH CHUNKS, SO FOR UP TO THREE YEARS IN ADVANCE. SO WE'LL FOCUS ON THE LONG-TERM AUCTIONS, WHICH, UH, THAT IS WHERE WE HAVE BEEN OBSERVING ALL OF THE, THE PERFORMANCE ISSUES, UH, AS OF LATE. SO BACK IN OCTOBER, 2019, THAT WAS OUR FIRST SEQUENCE, ONE LONG-TERM AUCTION THAT EXCEEDED 300,000 TRANSACTIONS AND TRIGGERED OUR FIRST TRANSACTION ADJUSTMENT PERIOD. SO THE TRANSACTION ADJUSTMENT PERIOD IS EXECUTED TO MAKE SURE THAT WE, UM, HAVE THE ABILITY TO RUN THE AUCTION AND CONVERGE THE AUCTION AND TO ACTUALLY, UH, SETTLE AND PUBLISH OUT THE AUCTION. SO IT IS A PROTECTION SYSTEM SO THAT WE ARE AWARE THAT WE CAN ACTUALLY RUN THE AUCTION. SO THIS IS THE, THE TRANSACTION ADJUSTMENT PERIOD. AT THE TIME WE DID TALK TO OUR VENDOR AND WE, WE DID FIND OUT THAT THERE IS A POTENTIAL FOR US TO SOLVE OUR AUCTIONS ABOVE 300,000, WHICH DID TRIGGER THIS NPR 9 72. THAT GAVE US THE ABILITY, UH, TO REALLY TO HAVE DISCRETION TO DECLARE THE TRANSACTION ADJUSTMENT PERIODS. SO WHAT THAT MEANS IS IF WE BELIEVE THAT OUR SYSTEM IS ABLE TO HANDLE THE, UH, AUCTION WITH THE NUMBER OF TRANSACTIONS, THEN WE CAN MOVE FORWARD BY BACK BYPASSING THE, THE TAP. APRIL, 2023. THAT WAS A SEQUENCE ONE. THAT WAS THE FIRST TIME WE EXCEEDED 400,000 TRANSACTIONS. AND THEN, UH, YEAR AND A HALF LATER, OCTOBER, 2024, ANOTHER SEQUENCE ONE WHERE WE EXCEEDED 500,000 [00:55:01] TRANSACTIONS. I'LL ALSO TALK A LITTLE BIT ABOUT THE PERFORMANCE, SOME OF THE PERFORMANCE MILESTONES THAT, UH, ON THE NEXT SLIDE. BUT IT DID LEAD US TO NPR 1261, WHICH WOULD ALLOW ERCOT SOME FLEXIBILITY WITH SETTING TRANSACTION LIMITS, UH, THESE TRANSACTION LIMITS WOULD, UH, MAKE SURE THAT WE WOULD ABLE TO, WE WERE ABLE TO RUN THE AUCTIONS AND POST ON TIME, SO THERE WAS NO DELAYS, UM, REALLY TO TRY NOT TO HAVE ANY DELAYS OR TO HAVE ANY, ANY OF THE AUCTIONS OVERLAP AND CAUSE ANY PROBLEMS. ALFREDO, CAN YOU COMMENT ON WHY WE'RE SEEING THIS, I'LL CALL IT EXPLOSIVE GROWTH, BUT, YOU KNOW, FAIR, FAIRLY SUBSTANTIAL GROWTH IN THE NUMBER OF AUCTIONS THAT WE'RE HOLDING. WHAT, WHAT'S DRIVING IT, WHAT'S BEHIND ALL THIS? WELL, I WILL SAY THAT THE INCREASED PARTICIPATION IS BECAUSE THE CR MARKET IS A, UH, THERE ARE AUCTIONS THAT WILL PROVIDE, UM, REALLY, IT'S JUST, IT'S, IT'S A SIGN OF REALLY GOOD MARKET IT, SO EVERYONE WANTS TO COME IN AND, UH, PARTICIPATE IN OUR MARKETS. SO REALLY THAT'S, THAT'S THE MAIN THING. WE DON'T HAVE MANY FEES AT ALL TO PARTICIPATE. SO IF YOU LOOK AT OTHER ISOS, OTHER RTOS, THERE ARE FEES WHEN IT COMES TO PARTICIPATING IN THE MARKET, UM, EITHER BY HOLDING, UH, CRS OR IN OTHER CASE, FTS, TCCS, THAT KIND OF THING. THERE ARE FEES OUT THERE. SO FOR US, UH, THERE IS NO FEES TO PARTICIPATE EXCEPT FOR, UM, BECOMING A CP. AND THEN, YOU KNOW, THE SMALL AMOUNT TO BECOME A CR ACCOUNT HOLDER. AND OF COURSE, THERE'S AN OPTION BID PRICE, UH, MINIMUM, UH, THAT IS ALSO LIKE A SMALL FEE. BUT AGAIN, IT IS, UH, VERY, VERY SMALL FEES WHEN IT COMES TO PARTICIPATE IN OUR MARKET. THE REASON I'M ASKING THAT IS AS THIS GROWTH CONTINUES, I MEAN, AND, AND I'M KIND OF GETTING TO THE PUNCHLINE HERE A LITTLE BIT, IT'S GONNA COST US MORE AND MORE MONEY TO SATISFY THIS, YOU KNOW, IN A TIMELY WAY. AND THE QUESTION IS, AT WHAT POINT SHOULD WE START TO THINK ABOUT INTRODUCING FEES THAT AT LEAST RE ALLOW US TO RECOVER, YOU KNOW, THE INCREMENTAL COST. ANYWAY, THAT'S, I'M GOING AHEAD. KEEP GOING. . THANK YOU, CHAIR. UM, SO NOW I'M GONNA TALK A LITTLE BIT ABOUT THE AUCTION RUNTIMES AND WHAT WE'VE BEEN OBSERVING. UH, AGAIN, BASED ON, UH, THIS FIGURE HERE, WE'VE, UH, VISUALLY, UH, SET UP SINCE JANUARY, 2024, ALL THE WAY INTO JULY OF THIS YEAR HERE. SO YOU CAN SEE THAT EARLY ON, UM, IN JANUARY OF 24 IS WHEN WE STARTED TO OBSERVE, UH, MORE RUNTIME OR, OR REALLY PERFORMANCE IMPACT, UH, ON OUR CRR LONG-TERM AUCTION. SO BACK IN JANUARY OF 2024, THAT WAS THE FIRST LONG-TERM AUCTION THAT EXCEEDED A HUNDRED AUCTION HOURS. SO THAT IS, THAT IS, THAT IS HUGE. UH, BECAUSE AGAIN, EVEN BEFORE THAT, WE WERE SOLVING POTENTIALLY AROUND THREE DAYS OR SO. UH, AND THEN FEBRUARY, 2024, WHICH IS THE NEXT MONTH, WE EXCEEDED 200 AUCTION HOURS. MARCH, WE EXCEEDED 300 AUCTION HOURS, AND THAT WAS THE FIRST TIME THAT SINCE GO LIVE, THAT WE DID PUBLISH LATE. UM, AGAIN, UM, BASED ON PROTOCOL, WE JUST HAD TO INFORM THE MARKET THAT THE DATES WERE NOT GONNA BE MET. WE SENT MARKET NOTICE AND MADE SURE THAT THAT WAS TAKEN CARE OF THROUGH THE 18 MONTHS. AFTER THAT, WE DID FOCUS, UM, WITH OUR IT DEPARTMENT AND WITH OUR VENDOR TO MAKE SURE TO DO EVERYTHING THAT WAS POSSIBLE TO, UH, TO, TO REALLY, UH, LOOK AT THIS PROBLEM AND TRY TO, UH, REALLY IMPROVE IT AND IMPROVE PERFORMANCE. SO THERE WERE SIX VENDOR SOFTWARE RELEASES. WE DID HAVE HARDWARE TUNING THAT HAPPENED, AND THEN CPUS DID INCREASE. SO THERE WAS A LOT OF WORK THAT CAME IN, UH, TO TRY TO MAKE SURE THAT WE DIDN'T GET INTO SITUATIONS THAT WE DID. UH, IN, IN THE EARLIER THREE MONTHS, SOME OF THE SOLUTIONS THAT WERE DISCUSSED WERE REMOVING THE MULTI-MONTH, UH, FROM THE LONG-TERM AUCTIONS, OPTION PRICING DISCLOSURE, UH, REALLY TRANSPARENCY FOR THE OPTION PRICING, RUNNING THE LONG-TERM AUCTION IN PARALLEL MONTHS. SO CURRENTLY WE RUN 'EM IN THREE TOU OPTIMIZATION. SO THAT MEANS THAT WE WOULD SPLIT 'EM UP INTO SIX MONTHLY, UM, RUNS, THEN DATABASE CLEANUP, AND THEN, UH, FINALLY THE CLOUD-BASED SOLUTION, WHICH, UH, SERVOS BADRI WILL BE TALKING ABOUT HERE, UH, IN A SECOND. UM, SO AGAIN, I JUST, I THINK ONE THING I JUST WANNA MAKE SURE AND, AND, AND COMMUNICATE HERE IS THAT AS OUR MARKET DOES EVOLVE, INCREASED TRANSACTION AND MARKET BEHAVIOR, UH, WILL CONTINUE TO IMPACT RUNTIME. UM, SO, SO THESE ARE SOLUTIONS THAT WE ARE, WE ARE LOOKING AT DIFFERENT SOLUTIONS, UH, BUT I JUST WANTED TO MAKE SURE THAT THAT WAS, UH, OUT THERE. ALRIGHT, NOW I WILL PASS IT OVER TO SERENA RI TO GET INTO ANY DETAILS. BUT BEFORE I DO, ARE THERE ANY [01:00:01] QUESTIONS? YES. HAVE WE EVER LOOKED AT OUTSOURCING THE MARKET, OUTSOURCING THE MARKET? YEAH. I MEAN, WHY, WHY DO WE DO IT IN HOUSE? IT'S NOT, UM, IT'S NOT GRID CRITICAL, SO TO SPEAK. WELL, IT'S HIGHLY PRO PROPRIETARY DATA, HIGHLY PROPRIETARY, BUT IS THAT THE REASON? BECAUSE I MEAN, THERE'S A LOT OF PLACES THAT COULD PROVIDE THIS SOFTWARE AS A SERVICE. SO I WILL SAY, UM, EVERY ISO HAS THIS MECHANISM AS PART OF WHAT THEY DO. IT'S LIKE RUNNING THE DAY AHEAD MARKET. IT'S, IT'S A, IT'S A PART OF THAT. UH, THERE IS AN ENTITY CALLED, UH, UH, THERE'S SOMETHING CALLED NODAL EXCHANGE THAT ATTEMPTS TO MIMIC SOME OF THIS. SO THERE IS A, THERE IS A MECHANISM THAT DOES THAT TO SOME EXTENT. UM, AND, AND THERE IS ACTIVITY ON, ON THAT AS FAR AS I'M AWARE. BUT, UH, I THINK, UH, IN THE FORMATION OF ISOS RTO REGIONS, THIS WAS CONSIDERED A PART OF THAT PACKAGE. AND SO WE INCLUDE THAT AS PART OF THE PACKAGE. AND I'D ALSO ADD THAT WE HAVE A LOT OF CUSTOM RULES HERE. SO THAT CHANGES IT FROM, YOU KNOW, IF YOU ARE DOING SOFTWARE AS A SERVICE, THEY WOULD LIKE MORE OF A STANDARDIZED PRODUCT, RIGHT? THE SAME PRODUCT RUNNING AT CAISO EXECUTES IT 15 MINUTES. SO IT SHOWS, GIVES AN INDICATION THAT THE RULES WE HAVE ARE WAY MORE COMPLEX THAN THAT MARKET. SO BECAUSE THE MARKET RULES ARE SO CUSTOM, OKAY, IT'S NOT GONNA BE EASY FOR A SAAS VENDOR TO IMPLEMENT THAT, BECAUSE IF IT'S GOING TO BE CUSTOM ANYWAY, SO YOU MIGHT AS WELL DO IT. OKAY. THANK YOU. THANKS. GOOD MORNING EVERYONE. I'M REWA BADRI, UM, SENIOR WRITER OF GRAND MARKET SOLUTIONS TEAM. UH, THANK YOU ALL FOR, FOR PROVIDING, UH, UH, GOOD DETAILS ON THE CR PERFORMANCE, UH, ISSUE IN THE BACKGROUND. UH, I WILL DOUBLE CLICK INTO SOME OF THE ASPECTS THAT ALFREDO MENTIONED ON THE PERFORMANCE ITSELF AND THE SOLUTIONING. I'LL START WITH THE ROOT CAUSE ANALYSIS. UM, AS ALFRED MENTIONED, UH, WE STARTED OBSERVING THE SIR LONG-TERM ACTION PERFORMANCE ISSUES SINCE JANUARY 20, 24. AND, AND, UH, SEVERAL AUCTIONS AFTER THAT TOOK SIGNIFICANTLY LONGER TIME TO COMPLETE, UH, THE ONE THAT, UH, RAN IN MARCH, 2024. IT TOOK, UH, 360 3 HOURS. UM, WE VIOLATED THE AUCTION MARKET POSTING DEADLINE, UH, UNDER THEIR ACTIVITY CALENDAR. AND, UH, AT THAT TIME, JP PROVIDED, PROVIDED THE UPDATE IN THIS MEETING ON THIS ISSUE. THERE ARE SEVERAL, UM, UH, AUCTIONS, UM, IN 20 24, 20 25, UH, CAME CLOSER, UH, TO VIOLATING THE CR POSTING, UH, DEADLINE. AS FAR AS THE ROOT CAUSE ANALYSIS CONCERNED, UM, IT TOOK ALMOST, UM, 18 MONTHS, UM, TO INVESTIGATE THE ROOT CAUSE, UM, PROVIDE, UH, I MEAN, DOING THE POCS AND PROVIDING THE SOLUTIONS, UH, TO THE, UH, TO THE PROBLEM. UM, CRR OPTIMIZATION ITSELF IS A, UH, COMPLICATED, UM, MATHEMATICAL PROBLEM AS A VENDOR HAD TO LOOK AT, UH, THE ISSUE FROM DIFFERENT ANGLES, UM, FROM THEIR CODE BOTTLENECKS POINT OF VIEW AND, AND, UH, NETWORK MODEL GROWTH POINT OF VIEW. AND, UM, UH, CRR COMMERCIAL MODEL POINT OF VIEW. UM, THEY IDENTIFIED THAT, UM, EVEN THOUGH NETWORK MODEL ORGANIC GROWTH IS ONE OF THE CONTRIBUTING FACTOR, UH, WHEN I SAY THAT, UH, SO IN EVERY MODEL, UM, THAT WE USE FOR CR AUCTIONS, SO NUMBER OF, UH, NODES, RESOURCE NODES KEPT INCREASING, THAT MEANS IT IS GIVING, UM, MARKET PARTICIPANTS CR RCON HOLDERS TO BID INTO THE SALE MARKET WITH THE VARIOUS, UH, PARTS. SO OBVIOUSLY THAT INCREASE THE SIZE OF THE PROBLEM ALONG WITH THE COMMERCIAL, UH, MODEL, MEANING THAT NUMBER OF, UH, TRANSACTION LIKE ALPHA MENTIONED, NUMBER OF TRANSACTIONS KEPT INCREASING, UH, BIDS AND OFFERS. UM, AND ALSO, UH, YOU KNOW, RISE OF THE, UH, MULTI-MONTH, UH, CR BIDS KEPT INCREASING. UM, THAT SEEMS LIKE A, A ONE OF THE KEY CONTRIBUTING FACTOR THAT VENDOR IDENTIFIED. AGAIN, I JUST WANT TO CLARIFY HERE, MUL, UM, RISING IN MULTI-MONTH SALE BIDS IS NOT ONLY THE PRIMARY ROOT CAUSE THE NUMBER OF SALE TRANSACTIONS INCREASING BECAUSE OF NUMBER OF ACCOUNT HOLDERS KEPT INCREASING, [01:05:01] UH, OR OTHER FACTORS AS WELL. SO FROM THE BEGINNING OF THIS, UM, UH, UH, ISSUE, WE LOOKED AT, UM, VARIOUS SOLUTIONS, SOFTWARE AND HARDWARE SOLUTIONS. IN PARALLEL, UM, UH, VENDOR LOOKED AT, UM, LIKE I SAID, UH, REFACTORING THE CODE, IDENTIFYING THE BOTTLENECKS IN THE CODE, AND, UM, UH, THEY UTILIZE THE PARALLEL PROCESSING WHERE NEEDED TO IMPROVE THE PERFORMANCE. THEY ALSO MADE, UM, ALL GUARD CHANGES AS WELL. EXAMPLE OF THAT IS BEFORE THIS IMPROVEMENT WAS BROUGHT IN, UM, THE SIERRA OPTIMIZATION ENGINE, UH, IS TO CONSIDER ALL TRANSMISSION VIOLATIONS FROM THE FIRST ITERATION. UM, BUT AFTER THIS CHANGE, UH, THE OPTIMIZATION ENGINE IS CONSIDERING HIGH MEGAWATT, UM, TRANSMISSION VIOLATIONS, UM, IN THE FIRST, SECOND, THIRD ITERATIONS HOPE WITH THAT IS THAT CONSIDERING THOSE HIGH MEGAWATT, UM, TRANSMISSION VIOLATIONS FIRST, UM, THE REST OF THE VIOLATION COULD POTENTIALLY COULD RESOLVE BY ITSELF. SO THAT MEANS IT WILL GIVE THE GOOD STARTING POINT FOR THE CRR SOLUTION AND IT COULD CONVERGE FASTER. AND THESE KIND OF CHANGES, UM, GAVE, UH, GOOD, UH, PERFORMANCE GAINS. BUT AGAIN, WE COULDN'T BRING THE, UH, PERFORMANCE, UM, TIMES TO THE EXPECTED, UH, TWO TO TWO TO THREE DAYS, UH, TIMEFRAME. SO VENDOR HAS EXHAUSTED ALL THE IMPROVEMENTS THAT THEY COULD MAKE IN THE SOFTWARE AND ALSO IN THE OPTIMIZATION ALGORITHM. SO WHILE VENDOR IS WORKING ON THESE, UM, UH, SOFTWARE IMPLEMENTS, UH, FROM THE GET GO, WE FOCUSED ON HARDWARE SOLUTIONS AS WELL. UH, WE, UH, WHEN THIS ISSUE HAPPENED, UM, IN, IN, UM, JANUARY APRIL TIMEFRAME, WE INCREASED NUMBER OF, UM, UH, CPUS IN, UH, ON-PREM CR ENGINE SERVERS FROM 16 TO 18. AND ALSO WE TUNED SOME OF THE HARDWARE, UH, UNDERLYING PARAMETERS AS WELL. IT GAVE SOME PERFORMANCE IMPROVEMENT, 15, 20%, BUT OVERALL COMBINING SOFTWARE IMPROVEMENTS AS WELL AS, UM, HARDWARE IMPROVEMENTS DID NOT BRING THE CRR PERFORMANCE BACK TO EXPECTED TWO, THREE DAYS. THEN THAT LED US TO LOOK AT ADVANCED HARDWARE CONFIGURATIONS. UM, AT THE TIME WE FELT THE MOST EFFICIENT AND FASTER WAY TO, UH, DO THAT ANALYSIS IS TO LOOK AT CLOUD, UH, AND, UH, LOOK AT THE GP TECHNOLOGIES. SO THAT LED US TO, UH, INITIATE CLOUD POC. UH, WE WORKED WITH THE VENDOR CLOSELY, UM, ON THIS EFFORT. THEY STARTED BACK IN, UH, UH, LAST YEAR MAY, UH, VENDOR SET UP A POC ENVIRONMENT IN THEIR PUBLIC CLOUD. UM, AND, AND, AND THEY SET UP THE GP TECHNOLOGY AS WELL. THEY RAN, UM, VARIOUS, UH, CRR AUCTIONS CASES THAT WE PROVIDED FROM PRODUCTION AGONIST, UM, VARIOUS HARDWARE CONFIGURATION THAT I MENTIONED HERE, UH, A MDP ARM, INTEL, NVIDIA, UM, THE BENCHMARKED AGONIST, THE SOFTWARE IMPROVEMENT THAT WE ALREADY BROUGHT IN FROM VENDOR, AS WELL AS THE ON-PREM HARDWARE TUNING THAT WE DID. NVIDIA GPS CONSISTENTLY DELIVERED BEST RESULTS, UM, FOR ALL THE AUCTION, LONG-TERM AUCTION CASES THAT WE RAN COMPARED TO OTHER, UH, TECHNOLOGIES THAT WE, UH, TESTED OUT. SO IF YOU LOOK AT THIS, UM, UH, PICTURE HERE, SO EVEN THOUGH WE RAN, UH, SEVERAL PRODUCTION SERIAL CASES TO TEST OUT, TO EVALUATE THE PERFORMANCE, UM, I HAVE SHOWED ONLY, UH, THE THREE PRODUCTION, UH, LONG-TERM AUCTION CASES HERE. SO IF YOU LOOK AT THE, UH, THE FIRST CASE, 9.25 DAY CASE IN PRODUCTION, OR, UH, IT TOOK 2 22 HOURS TO COMPLETE WITH THE SOFTWARE IMPLEMENTS AND ON-PREM, UH, HARDWARE IMPROVEMENTS, WE WERE ABLE TO BROUGHT DOWN THAT TO 28 HOURS SIGNIFICANT PERFORMANCE IMPLEMENT. AND WITH, UM, N-V-D-I-G-P-U TECHNOLOGY THAT, UH, THAT WE RAN IN CLOUD, UH, IT TOOK 12 HOURS. AND, UM, AND, AND WE TIED WITH ANOTHER NINE DAY CASE, UH, WHERE IT, I KNOW IN PRODUCTION IT TOOK 2 0 4 HOURS AND, UH, WITH THE HARDWARE AND, UH, AND IN SOFTWARE IMPROVEMENTS THAT WE BROUGHT IN, IT TOOK 98 HOURS. THEN, UM, WITH THE NVIDIA GP TECHNOLOGY, IT TOOK 36 HOURS, AND THEN ALSO YOU CAN LOOK AT 6.5 DAY CASE AS WELL. UM, AND IF YOU LOOK AT THIS, THE [01:10:01] NVIDIA TECHNOLOGY, G TECHNOLOGY IS CONSISTENTLY PROVIDING, UH, PERFORMANCE THAT WE ARE EXPECTING, WHICH IS TWO, THREE DAYS. UM, SO, SO AFTER THIS, UH, SO WE DETERMINED THAT. SO THE BEST WAY, UH, TO MOVE FORWARD TO SOLVE THIS PROBLEM IS SET UP, UM, OUR CLOUD INFRASTRUCTURE, UH, AND THEN, UM, AND, AND SET UP THE NVIDIA GPUS TO IMPROVE THIS, UH, CRR PERFORMANCE. UM, WE, UM, SET UP THE CLOUD INFRASTRUCTURE PARTNERING WITH HYPERSCALER, UH, AND WE PROCURED THE NVIDIA GP POWERED HARDWARE ON AIR CARD, PUBLIC CLOUD. UM, WE ESTABLISHED VARIOUS SUBSCRIPTIONS, DEV TO PRODUCTION, AND, UH, WE TOOK SEVERAL CASES, UH, FROM PRODUCTION, UM, THAT TOOK MUCH LONGER TIME THAN EXPECTED, AND WE TESTED, UH, THOSE CASES. UM, AND THE RESULTS FROM THAT, UH, IS, UH, WE, WE FOUND THAT RESULTS WERE SIMILAR TO WHAT WE, UM, OBSERVED WITH THE CLOUD, POC, UH, THAT WHOLE EFFORT TESTING EFFORT WENT FOR THREE MONTHS AND FINALLY WENT LIVE LAST MONTH. UM, SO THE, THE, THE AUGUST AUCTION THAT WE RAN, IT TOOK 57 HOURS. UM, IT HAS BEEN ALMOST, I MEAN, THIS FIRST AUCTION, I WOULD SAY THAT COMPLETED UNDER THREE DAYS SINCE, UH, LAST THREE YEARS, THERE ARE FEW AUCTIONS, FOUR OR FIVE AUCTIONS THAT TOOK LESS TIME IN LAST THREE YEARS. BUT, UM, UM, WHAT WE HAVE SEEN LAST MONTH RUNNING THE AUCTIONING CLOUD, UH, TOOK 57 HOURS. THAT CONCLUDES MY PRESENTATION. IF YOU HAVE ANY QUESTION, I'LL TAKE GUESS THE OBVIOUS QUESTION IS HOW MUCH INCREMENTAL DOES IT COST TO RUN IT ON THIS INFRASTRUCTURE? YEAH, I DON'T HAVE THE COST NUMBERS HERE, BUT DEFINITELY A SOLUTION LIKE THIS IS WORTH THE PRICE INSTEAD OF RUNNING INTO, UM, VALUATION ISSUES, VIOLATION ISSUES. RIGHT? IT'S NOT THAT SIGNIFICANT. I DON'T HAVE, I'LL, I'LL SEND YOU, SEND YOU THE ACTUAL NUMBERS, BUT I, IT'S AN IDEAL CASE FOR RUNNING IN THE CLOUD BECAUSE THIS RUNS SO INFREQUENTLY. SO IN AN IDEAL WORLD, WE COULD JUST USE IT FOR THE, UH, AUCTION AND THEN SHUT DOWN. YES. SO IN THAT SENSE, IT'S ACTUALLY MORE EFFECTIVE, BUT THE REALITY IS THAT WE CAN'T RESERVE THESE GPUS BECAUSE THEY'RE SO IN DEMAND. WE HAVE TO KEEP IT RESERVED. AND THAT'S CAUSING A PRICE REFLECTION. NO, I'M NOT, I'M NOT SUGGESTING THAT WE WOULD DO THIS OURSELVES IN HOUSE. I'M, BUT I AM SUGGESTING THAT WE MAY WANNA LOOK AT SOME SORT OF TRANSACTION FEE OR OTHER WAY OF RECOVERING THE COST ASSOCIATED WITH THAT. THAT'S, THAT'S WHERE MY QUESTIONS. OH, OKAY. THAT'S A MORE BUSINESS THING. I'LL REFER TO KEITH ON THAT, UM, BECAUSE MORE THAN THE, IT COSTS, IT'S MORE IMPACT TO FINANCE AND STUFF LIKE THAT AS WELL. SO I'LL, I'LL MAKE A STATEMENT AND I THINK RICHARD PROBABLY IS GONNA FOLLOW THAT UP. I THINK WHAT'S IMPORTANT, AND WE'RE GONNA TALK ABOUT THE DAY HEAD MARKET IN A SECOND WHERE WE ARE PROPOSING A A, A FEE. UM, WHAT'S INTERESTING IS THAT WHEN WE LOOK AT THE DATA IN TERMS OF THE, THE PARTICIPANTS THAT ARE ASSOCIATED WITH LOAD IN THE CRR AUCTIONS, UH, IT'S, AND, AND YOU COMPARE THAT WITH THE NUMBER OF TRANSACTIONS THAT ARE OCCURRING, IT'S ABOUT 25% ARE LOAD RELATED AND LOAD RELATED ENTITIES PAY A SYSTEM, ADMINISTRATIVE SYSTEM ADMIN FEE, WHEREAS THE, UM, THE OTHER TRANSACTIONS DO NOT. SO 75% IS NOT ASSOCIATED WITH A SYSTEM ADMIN FEE DIRECTLY, AND 25% ARE, BUT I THINK IT'S IMPORTANT TO ALSO RECOGNIZE THAT THERE IS AN INDIRECT, UM, THERE IS AN INDIRECT BENEFIT BECAUSE WHEN WE TALK ABOUT CR AWARDS, THE CR AWARDS ARE HELD IN AN ACCOUNT THAT ACCOUNT EARNS INTEREST AND THAT INTEREST OFFSETS THE SYSTEM ADMINISTRATIVE FEE. SO INDIRECTLY, THE CRR ACTIVITY IS IN FACT OFFSETTING THE SYSTEM ADMIN FEE INDIRECTLY. UM, BUT I THINK IT'S ALSO IMPORTANT TO NOTE THAT EVEN OF THAT, NOT EVERYONE THAT IS AWARDED PAYS A FEE. UH, AND THAT HAS TO DO WITH, UM, THE FACT THAT SOME AWARDS ARE, UH, ARE SALES OR SO THEY RECEIVE MONEY INSTEAD OF, UH, PAYING IN MONEY. SOME OF THEM ACTUALLY RECEIVE CRS AT NO COST. AND THEN, UH, THERE ARE SOME THAT RECEIVE AT A VERY MINIMAL COST, AS, AS ALFREDO WAS NOTING THAT SOME OF THEM, IT COULD BE AS LOW AS A, AS A PENNY, SO THEY MAY BE PAYING SOME, SO THERE'S A SENSITIVITY THERE IN TERMS OF, UH, WHAT A, A FEE WOULD DO WITH RESPECT TO THAT. BUT THERE'S, THERE'S DEFINITELY AN OPPORTUNITY TO EXPLORE THAT. SO I'LL HAND IT OVER TO RICHARD AS WELL. THANKS. YEAH. THIS, THIS IS A NUANCED QUESTION. WHEN WE COME DOWN TO THE COST, I MEAN THE ACTUAL COST OF WHAT WE'RE PAYING, WE CAN GET THAT TO YOU. WE HAVE TWO DIFFERENT THINGS THAT WE'RE WORKING. CAN YOU PULL YOUR MICROPHONE CLOSER, PLEASE? SURE, ABSOLUTELY. SO WHAT, WHAT WE'RE [01:15:01] WORKING ON, WE HAVE, WE HAVE THE CLOUD-BASED SUBSCRIPTION, UM, WHICH WE HAVE INITIAL PRICING ON, BUT WE'RE WORKING ON LONGER TERM PRICING ON THAT. AND THE SECOND OPTION IS BRINGING THIS BACK ON SITE WITH GPUS INTERNALLY, WHICH WOULD BE A LOWER COST OPTION IN THE LONG TERM THAN THIS, BASED ON OUR CURRENT, OUR CURRENT. LOOK AT THAT. I, I THINK KEITH HAS ALREADY CALLED OUT SOME OF THIS. AND THIS IS SOMETHING, WE'LL, WE WILL HAVE AS PART OF THE CONVERSATION F AND A AS WELL, WHICH IS AN IMPORTANT PART OF OUR SYSTEM FUNDING, IS BASED ON THE OFFSET AGAINST OUR SYSTEM ADMINISTRATION FEE RATE FROM THE INTEREST INCOME THAT WE EARN ON THE CRR AUCTION BALANCES. SO WHETHER OR NOT WE HAVE A DIRECT FEE, WE ACTUALLY HAVE DIRECT INCOME FROM THE, UH, A DIRECT REVENUE SOURCE, A CASH SOURCE, THAT'S NOT REVENUE PER SE, BUT WE HAVE THIS CASH SOURCE OF INTEREST INCOME, WHICH OFFSETS THE TOTAL SYSTEM ADMINISTRATION FEE RATE. SO IT'S A NUANCED QUESTION. I THINK IT DESERVES SOME MORE INTENSE ANALYSIS, UH, AND WE CAN BRING THAT BACK WHEN WE'RE THINKING ABOUT THE DECEMBER BOARD MEETING AND BRINGING OUT SOME MORE INFORMATION TO RICHARD. I THINK MY, MY CONCERN HERE IS, I, I UNDERSTAND WE ARE IN FLOAT INTEREST, AND, AND THAT'S, THAT'S A GOOD THING AND IT'S BENEFICIAL OVERALL. BUT MY CONCERN IS THAT IF THIS KEEP, IF THE NUMBER OF TRANSACTIONS KEEPS GROWING, THE COST IS GONNA KEEP GROWING. AND THE QUESTION THEN IS, UM, IS ALL OF THAT COST WHAT WE WOULD, IS ALL OF THAT USEFUL ACTIVITY OR IS THERE SOME SPECULATIVE ACTIVITY SURE. IN THERE THAT A TRANSACTION FEE MIGHT HELP SORT OUT. AND I, I THINK THAT'S PART, IT'S, IT'S REALLY A DAY AHEAD. I MEAN, THIS APPLIES MORE IN THE NEXT, IN THE NEXT SECTION THAN THIS ONE I UNDERSTAND, BUT YEAH, THIS IS SOMETHING KEITH THAT BEEN TALKING ABOUT, SO THEN, YEP. YEAH, AND I, I THINK WHEN WE GO AND TALK ABOUT OUR DAY AHEAD ISSUE IN JUST A FEW MINUTES, THAT THE SOLUTION THERE IS MORE TARGETED TO TRY TO SEPARATE THE SPECULATIVE FROM THE HELPFUL. YEAH. I'M JUST WONDERING IF WE HAVE SOME OF THAT GOING ON HERE TOO. THAT'S A FAIR QUESTION. YEAH. OKAY, JOHN, THANK YOU. JEFF. BILL, A QUESTION FOR YOU. SO YOU, BY MOVING TO THE CLOUD WITH YOUR PROOF OF CONCEPT, IT LOOKS LIKE YOU'VE HAD GREAT IMPROVEMENT IN PERFORMANCE. CAN WE ASSUME THAT THIS IS SCALABLE AS VOLUME INCREASES? UH, WILL WE ACHIEVE THE SAME RESULTS MOVING FORWARD? LET'S SAY THAT VOLUME'S TRIPLED AGAIN IN SIX MONTHS. CAN WE CONTINUE TO THROW MORE CLOUD RESOURCES AT THIS AND, AND MAINTAIN THE SAME PERFORMANCE CHARACTERISTICS? YEAH, SO, SO, SO WHAT WE'RE LOOKING AT IS CONTINUE TO LOOK AT SOFTWARE IMPROVEMENTS. UM, CAN YOU HOLD THE MICROPHONE CLOSER? YEAH. SO WE ARE GOING TO CONTINUE TO LOOK AT SOFTWARE IMPROVEMENTS. UM, AND ALSO IF, UH, IF WE START SEEING THE PERFORMANCE, UH, DECLINE, WE HAVE TO LOOK AT, UM, THE NEWER VERSIONS OF, UM, HARDWARE MODELS AT THE TIME TO, UH, TO AGAIN, BRING THE PERFORMANCE BACK. SO YEAH, SO THERE ARE, THERE ARE WAYS WE CAN CONTINUE TO MAINTAIN THIS, UH, PERFORMANCE, UM, UH, EXECUTION TIMES, BUT WE HAVE TO CONSTANTLY EVALUATE, UH, OUR OPTIONS AT THAT TIME. OKAY. SO GENERALLY THE ANSWER IS YES, BECAUSE WE ARE ON THE CLOUD, SO WE COULD MORE EASILY GO TO NEWER ARCHITECTURES WHEN THEY'RE AVAILABLE. IT'S NOT A QUESTION OF MONEY, RIGHT? BUT, BUT YOU CAN GET NEWER TECHNOLOGIES FAR EASIER THAN IF YOU HAD BROUGHT IT IN HOUSE AND YOU PURCHASED HARDWARE. SO ALL THAT IS EASIER TO SCALE AND ALL THAT. SO THAT'S, THAT'S AN ADVANTAGE OF BEING THE CLOUD. THE ONLY RISK IS ARE THOSE AVAILABLE WHEN WE NEED IT? OKAY. AND ARE YOU PREPARED TO PAY FOR IT? I MEAN, YEAH, ALL COSTS MONEY, RIGHT? YEAH. YEAH. ANY OTHER QUESTIONS? THANK YOU. THANK YOU, SABRINA. VA. THANK YOU. THANK YOU. EL , ALFREDO, SORRY. OKAY, NOW [6.3 Day Ahead Market – Recent Timeline Deviations] RELATED, RELATED BUT DIFFERENT. UM, WE'RE GONNA TALK ABOUT THE DAY AHEAD MARKET AND DEVIATIONS THAT WE'VE SEEN IN THE TIMELINES. KEITH, KEITH AND JP ARE GONNA COVER THIS. ALRIGHT, THANK YOU. YES, THIS IS, UH, SIMILAR BUT, BUT DIFFERENT I THINK IS THE WAY TO THINK ABOUT IT AS WELL. UM, I THINK THE SIMILARITY IS WE ARE ALSO EXPERIENCING RUNTIME ISSUES WITH OUR DAY AHEAD, MARKET SOLVE TIME. UH, THERE IS A POSTING REQUIREMENT FOR US AT, UH, ONE 30, UH, IN THE AFTERNOON. UM, AND, UH, THERE HAVE BEEN SOME INSTANCES WHERE WE'VE HAD TO POST BEYOND THAT PERIOD. UH, BUT I THINK WHAT'S ALSO INTERESTING IS THAT WHEN WE, WHEN WE DO IT ON A DAY-TO-DAY BASIS, WE GET, UM, WE, WE MAY BE GETTING CLOSE, UH, BUT NOT EXCEEDING THAT, AND THAT HAPPENS ON A, ON A PRETTY REGULAR BASIS. AND, AND THAT HAS BEEN INCREASING OVER THE LAST JUST A FEW MONTHS REALLY. SO LET'S GO AHEAD AND JUMP IN. THE, THE SLIDE WE'RE SHOWING HERE IS, [01:20:01] UH, THE LAST, ESSENTIALLY SINCE THE END OF LAST YEAR AND INTO THIS YEAR. AND WHAT WE HAVE IS SORT OF A GREEN AT THE BOTTOM, WHICH IS A, IT'S ALL RUNTIME, IT'S IN SECONDS, BUT REALLY THE, THE RANGE IS ABOUT, UH, ABOUT AN HOUR AND A HALF IS THE, THE TRANSITION FROM THE GREEN TO THE YELLOW. AND THEN, UH, ABOUT TWO AND A HALF IS WHERE YOU TRANSITION FROM THE YELLOW TO THE RED, UH, AND THE RED, ULTIMATELY WE'RE SEEING TIMES, UH, IN, IN A, IN AROUND THREE PLUS HOURS. AND THAT'S, THAT'S WHERE WE START EXCEEDING, UH, THE REQUIRED POSTING TIME. AND SO LOOKING BACK TO LATE LAST YEAR, UH, WHERE, WHERE OUR DATASET STARTS IS THAT WE WERE IN THAT GREEN PERIOD, UM, UH, GREEN TO YELLOW PERIOD, WHICH, WHICH WAS GENERALLY ACCEPTABLE, BUT WHAT WE SEE IS THAT, UH, WE HAD ONE DEVIATION IN EARLY, UH, EARLY APRIL, BUT, BUT REALLY THE, THE BIG CHANGE STARTED TO HAPPEN IN THE MAY TO JUNE TIMEFRAME. AND, UH, WE'LL WE, WE'LL JUST JUMP IN IN JUST A SECOND HERE. WHAT WERE THOSE, THOSE PRIMARY CAUSES OF, OF WHAT WAS CAUSING THAT TO HAPPEN? ALRIGHT, SO WHAT, WHAT WE STARTED TO SEE WERE, WERE REALLY A, THERE ARE A FEW ISSUES AND WE'RE GONNA COVER THEM, UM, BOTH MYSELF AND JP. I THINK, UH, THERE ARE TWO ISSUES. THE FIRST TWO I I WILL COVER, UH, JPO WILL COVER THE THIRD. THERE WAS A UNIQUE SORT OF INSTANCE, UH, RELATED TO API USE THAT, THAT, THAT JPO COVER. THE TWO THINGS THAT I WANT TO COVER ARE INCREASE IN GROWTH AND WHAT ARE KNOWN AS POINT TO POINT OBLIGATION BID VOLUMES. UM, REALLY THIS CAN BE SEEN AS A REAL TIME CRR EFFECTIVELY. SO A CRR PRODUCT HEDGES DAY AHEAD CONGESTION, THE POINT TO POINT EFFECTIVELY HEDGES REALTIME CONGESTION. AND SO WE'VE SEEN SUBSTANTIAL GROWTH OF THIS OVER THE LAST SEVERAL YEARS. UH, IT STARTED BACK IN MAY OF 2022 WHERE WE HAD TO, TO BEGIN SORT OF LIMITING THE NUMBER OF POINT TO POINT, UH, UH, OFFERS, BIDS AND OFFERS THAT A PARTICIPANT COULD PUT IN. AND SO WE LOWERED THAT TO 10,000 IN 2022, DOWN TO 1000 IN 2023, AND, AND FURTHER LOWERED IT, UH, BY COUNTERPARTY TO SEVEN 50, UH, BACK IN 2025. SO WE'VE BEEN SLOWLY, UM, REDUCING THE NUMBER OF TRANSACTIONS THAT A PARTICIPANT COULD TAKE THAT ARE POINT TO POINT. UH, AND OUR IMM HAD RECOMMENDED A FEW YEARS AGO TO INSTITUTE A FEE, UH, TO HELP, UH, CURTAIL SOME OF THIS ACTIVITY. UM, AND ULTIMATELY, NPR 1343, WHICH WAS FILED EARLIER THIS YEAR, WORKS TO ADDRESS THAT. NOW, AS, AS WAS MENTIONED JUST A FEW MINUTES AGO, THE WAY THIS IS CONSTRUCTED IN 1343 IS THAT IT IS NOT A, UH, A BLANKET A FEE ON ALL TRANSACTIONS. IT DOES HAVE A MECHANISM IN IT TO IDENTIFY, UM, TO REALLY MAKE A DISTINCTION BETWEEN A PRICE FORMATION AND EFFICIENT, UH, UH, UH, TRANSACTION VERSUS ONES THAT ARE INEFFICIENT. AND REALLY IT GETS TO A SENSE OF HOW, HOW CLOSE TO THE, THE PREVAILING PRICE WERE YOU IN YOUR TRANSACTION? WAS IT, WAS IT TIGHT IN AND ASSISTING IN PRICE FORMATION OR WAS IT SOMETHING THAT WAS FURTHER OUT AND, AND WAS NOT HELPING WITH THAT? AND SO THAT CONSTRUCT IS SOMETHING, UH, THAT IS, IS CURRENTLY IN, UH, NPR 1343 IS MAKING ITS WAY THROUGH THE STAKEHOLDER PROCESS. AND WE HOPE TO HAVE THAT TO THE BOARD BY THE DECEMBER MEETING. AND THE, THE REASON FOR THAT IS THAT WHILE WE DEFINITELY SEEM TO BE, UH, IN A PERIOD WHERE WE'RE ABLE TO MANAGE THE RUNTIME BENEATH THE REQUIRED TIMING, UH, UH, AS, AS WE EXPECT THESE VOLUMES HAVE GROWN AND COULD CONTINUE TO GROW, WE HAVE TALKED AND REACHED OUT TO SOME PARTICIPANTS THAT HAVE INDICATED THAT, UH, THEY ARE, UH, THEY RECOGNIZE THEIR IMPACT ON THE, THE SOLUTION, BUT THEY DO LOOK FORWARD TO THE OPPORTUNITY TO ADD MORE AND MORE TRANSACTIONS TO THE SYSTEM. SO WE SEE THIS AS AN IMPORTANT CHANGE, UH, FOR GOING FORWARD. SO POINT TO POINT, UH, OBLIGATION BID VOLUMES IS THE FIRST ITEM THAT HAS GROWN. THE SECOND ITEM THAT HAS GROWN IS WHAT ARE KNOWN AS ENERGY ONLY SUBMISSIONS. THESE ARE ARBITRAGES BETWEEN DAY AHEAD AND REAL TIME PRICES. UH, WE HAVE SEEN SIGNIFICANT INCREASE BY A CONCENTRATED NUMBER OF PARTICIPANTS, PARTICULARLY IN THAT APRIL TO MAY TIMEFRAME INTO JUNE. AND SO, UH, ULTIMATELY THIS DID CREATE SOME CHALLENGES, PARTICULARLY WITH OUR SOFTWARE BEING ABLE TO, UH, HANDLE THE, THE VOLUME. AND THIS [01:25:01] RESULTED IN SOME INCIDENTS IN EARLY JUNE, UH, IN THE JUNE 6TH TO JUNE 8TH TIMEFRAME. OUR DAY AHEAD MARKET TEAM HAS FLEXIBILITY WHERE THEY CAN IMPOSE SOLUTIONS TO HELP, UH, UH, IMPROVE THE, ALLOW THE MARKET TO SOLVE. AND SO WE HAD TO TAKE ADVANTAGE OF THAT DURING THAT PERIOD IN EARLY JUNE, WHERE WE BORROWED ESSENTIALLY THE MECHANISM THAT WE USE FOR OUR POINT-TO-POINT BID FEES TO LIMIT, UH, THE NUMBER OF TRANSACTIONS THAT A PARTICULAR PARTICIPANT CAN DO. AND AS A RESULT OF THAT, WE WERE ABLE TO, UM, WE WERE ABLE TO SOLVE THE DAY AHEAD MARKET, UH, ON TIME. AND SO NOW LET ME TRANSITION TO JP. THERE WERE SOME TECHNICAL SOLUTIONS AS WELL TO HELP WITH THAT, TO ADDRESS THIS ISSUE OF, OF THE GROWING ENERGY ONLY SUBMISSIONS, JP GUESS YOGI BERRA SAID IS DEJAVU ALL OVER AGAIN SINCE I WAS HERE FOR CRR IN A SIMILAR KIND OF, UH, SITUATION. SO, UH, WITH THIS, UH, THING, WHEN WE HAD THIS ARRAY SIZE INCREASE, UM, WE, WE DID HAVE A PROBLEM WHERE THE DAM MARKET CRASHED AND WE HAD TO RERUN IT, UH, WITH THE CORRECTIONS THAT, UH, KEITH WAS TALKING ABOUT. SO THE FIRST FIX WE DID IS TO INCREASE THE RA SIZE, AND THAT'S NOT A FIX WE DO FREQUENTLY. SO THE LAST TIME WE MIGHT HAVE DONE THIS WAS A FEW YEARS AGO. SO, UM, BUT WE WERE ANTICIPATING THAT, UH, THIS INCREASE IS COMING. SO WE HAD ALREADY BEEN WORKING WITH THE VENDOR TO GET THAT FIX, WHICH IS WHY WE WERE ABLE TO ROLL THIS FIX IN RELATIVELY QUICKLY. BUT, UM, THE, THE TRUE FIX, WHAT YOU'RE LOOKING FOR IS WHAT WE ARE EXPECTING TO PUT IN THE NEXT SIX MONTHS OR SO, WHERE WE GO AWAY FROM THE STATIC ARRAY SIZING AND WHAT WE CALL IN PROGRAMMING DYNAMICALLY ALLOCATED, UH, ARRAY S SIZES. SO AT THAT POINT, THIS ISSUE OF RESIZING ARRAYS GO AWAY COMPLETELY. SO THAT'S WHAT WE ARE LOOKING FOR, UH, TOWARDS IN THE NEXT, UH, UH, SIX MONTHS. UM, SO NO, WHILE THIS WAS HAPPENING, WE HAD, UM, AS KEITH SAID, A UNIQUE INCIDENT WHERE, UM, WE FOUND THAT ON A PARTICULAR DAY, UM, UH, JUNE 18TH, UH, OUR MARKET PARTICIPANTS WERE NOT ABLE TO SUBMIT, UH, BIDS IN. AND, AND THAT CAUSED US TO DELAY THE MARKET CLOSE SO THAT PEOPLE COULD GET TIME TO, UM, SUBMIT THOSE BIDS. NOW, THAT WAS MOST STRICTLY AN IT PROBLEM, NOT SO MUCH OF A MARKET CONSTRUCT OR A, A MARKET GROWTH KIND OF ISSUE, IT'S JUST THAT IT'S HAPPENED IN THIS KIND OF A WINDOW AND WE DON'T USUALLY HAVE THIS KIND OF IT PROBLEMS, BUT UNFORTUNATELY IT DID HAPPEN DURING THIS WINDOW. SO WE THOUGHT WE'LL TALK ABOUT THAT AS WELL, SO THAT IT'S NOT RELATED TO THAT, UH, OTHER ISSUE THAT KEITH TALKING ABOUT. NOW, IN THIS CASE, UM, WE HAVE MARKETS, UH, INTERACT WITH US THROUGH APIS, WHICH ARE PROGRAM PROGRAMMATIC WAYS OF CONNECTING TO OUR MARKET. AND, UM, UM, WE HAD, UM, IT ISSUES WHERE WE COULDN'T HANDLE THAT. AND, AND AT THE SAME TIME WE HAD, WE, WE OBSERVED THAT SOME MARKET PARTICIPANTS WERE DOWNLOADING REPORTS AT A VERY HIGH FREQUENCY. AND WHILE THAT TURNED OUT, NOT TO BE THE ROOT CAUSE OF IT, BUT THE FACT THAT THIS KIND OF THING EXISTED, UH, CAUSED OUR TEAM TO CHASE DOWN, GO DOWN THE WRONG RABBIT HOLE, SO TO SAY, UM, WE REALLY TAKE TRACK DOWN THE ISSUE TO A SERVER THAT NEEDED RESTARTING. AND AS SOON AS THAT HAPPENED, THE PROBLEM WENT AWAY. UM, BUT UM, I THOUGHT WE'LL BRING IT HERE SINCE WE HAD TO ISSUE A COUPLE OF MARKET NOTICES, UH, TO ADDRESS THIS ISSUE. SO ONE OF THE FOLLOW UP FROM THAT IS, UH, WE WANT TO MAKE SURE WE ARE MONITORING FOR THIS KIND OF, UH, ACTIVITY WHERE, YOU KNOW, THE DAY HER MARKET REPORT IS PUBLISHED ONCE A DAY, THERE'S NO REASON FOR SOMEONE TO, TO BE DOWNLOADING IT EVERY FEW SECONDS. SO, SO WE WANT TO LOOK FOR THAT KIND OF THING ON OUR SIDE, WE ARE NOT NOTICING IT BECAUSE OUR SYSTEM IS CAPABLE OF SUPPORTING THOSE KIND OF A DEMAND, BUT IT'S UNNECESSARY. AND WHEN SOMETHING LIKE THIS HAPPENS, WE, WE GO DOWN THE WRONG RABBIT HOLES. UM, SO WE ARE TRYING TO IMPOSE, UM, HOW WE EITHER MONITOR IT OR, UM, PUT SOME RATE LIMITING. AND AT THE SAME TIME, WE ARE ALSO WORKING WITH THE MARKET PARTICIPANTS SO THAT WE DON'T GET TO THIS, UM, EDUCATE THE MARKET PARTICIPANTS WHERE POSSIBLE. SO LASTLY, WE WANNA TALK ABOUT OTHER, UH, POTENTIAL SOLUTIONS UNDERWAY TO IMPROVE THE DAM UH, PERFORMANCE. UM, AND, AND I'LL, I'LL KEITH, YOU WANNA TALK ABOUT THE FIRST ONE, WHICH IS THE, UH, CONSTRAINT MANAGEMENT. UM, ANYWAY, WHAT, WHAT WE ARE DOING IS, UM, TO, TO THIS ALGORITHMIC CHANGE WHERE WE ARE LOOKING AT CONSTRAINTS FOR A FEW HOURS INSTEAD OF THE FULL DAY. SO, SO IT'S SIMILAR TO WHAT WE ALSO TALKED ABOUT IN THE CRR, WHERE JUST USING SMARTER ALGORITHMS WOULD HELP. WHERE WE ARE [01:30:01] LOOKING AT LARGER MEGAWATT CONSTRAINTS, LIKE THREE WAS EXPLAINED. SO, SO CHANGING HOW WE LOOK AT THE ALGORITHM INSTEAD OF GOING AT IT DUMB, YOU KNOW, DO DO A SMARTER WAY OF DOING THINGS. SO THAT'S WHAT THE FIRST ONE IS, AND I THINK WE ARE DOING THAT RIGHT AWAY. THE NEXT ONE IS TYPICALLY, YOU KNOW, IF I WEAR MY IT HAT AS JOHN, YOU KNOW, PROBABLY JOKES, THROW HARDWARE AT IT, LOOK AT SOFTWARE, YOU KNOW, IMPROVEMENT. YES, WE, WE, WE ARE GOING TO DO THAT AND THAT'S GOING TO GET US SOME IMPROVEMENT. UM, UNFORTUNATELY THE PROBLEM FORMULATION FOR DAY HEAD MARKET IS NOT THE SAME LIKE THE CRR WHERE WE COULD LEVERAGE GPU, UM, IS JUST A DIFFERENT KIND OF FORMULATION WHERE IT IS NOT MORE AS CONDUCIVE TO THE GPU WHERE WE CAN IZE IT, BUT NEVERTHELESS, WE ARE LOOKING AT THE ALGORITHMS WHEREVER POSSIBLE PARALLEL, UH, LOOK AT PARALLEL OPTIONS. AND, AND THE MAIN CONSTRAINT IS THE OPTIMIZATION ITSELF WHERE IT USES A SOLVER. UM, WE, WE HAVE A COUPLE OF VENDORS THAT JUST PROVIDE THE SOLVER AND WE ARE GOING TO LOOK AT HOW, UM, HOW WE CAN OPTIMIZE THAT SOLVER AND, AND, AND EARLY RESULTS ARE THAT USING A CLOUD-BASED SOLUTION, NOT GPU, BUT UNDER CLOUD-BASED SOLUTION WE ARE LOOKING AT PROMISING RESERVES. SO I'M LOOKING FORWARD TO COME BACK AND, UM, HOPEFULLY VERY SOON MAYBE YOU HAVE THREE OF US TALK THROUGH SOME OF THE SOLUTIONS THAT WE HAVE OBSERVED, UH, ESPECIALLY WHEN WE ARE CONFIDENT WE ARE PAST POC AND WE'RE READY TO GO LIVE, READY TO TAKE ANY QUESTIONS. JP? YES. SO WE'VE GOT TWO SITUATIONS WHERE WE'RE KIND OF GETTING OVERWHELMED BY VOLUME RESULTING IN DELAYS. SO THE QUESTION WOULD BE WHAT HAVE, WHAT HAVE YOU LEARNED FROM THIS? FOR EXAMPLE, AS WE DESIGN THE SYSTEMS, DO WE NEED TO THINK MORE BROADLY IN TERMS OF THE CAPACITY AND WHAT THE POTENTIAL, YOU KNOW, WHAT WE COULD BE FACED WITH UPFRONT TO AVOID THIS? OR HOW WOULD YOU DESCRIBE THAT? WHAT WOULD BE THE LESSONS LEARNED? I I WOULD SAY THAT, YOU KNOW, WE TEND TO PROJECT FORWARD A BIT, BUT I THINK THE PACE OF GROWTH OF OUR TEXAS MARKET IS EXPONENTIAL. AND, AND THEN NOW WE ARE AWAY FROM A LINEAR THINKING AND SAY, OKAY, MAYBE WE NEED TO SAY, OKAY, THE NEXT TWO OR THREE YEARS, WHAT IF A MARKET IS THREE TIMES? I MEAN, LOOK AT WHAT'S HAPPENING WITH LOAD AND ALL THAT, RIGHT? ABSOLUTELY. SO THEN THE QUESTION IS, IF THAT IS WHAT OUR PROJECTION IS, WE MAY NEED TO START TODAY TO SAY CAN WE CREATE THE SYSTEM OF THAT FUTURE? YES. RIGHT? SO, SO WE ARE INTO THAT KIND OF THINKING AS WELL, WHERE, WHAT DOES IT MEAN WITH THIS KIND OF LOAD AND, AND NOT JUST, YOU KNOW, WE JUST TALKING LOAD THAT IS CORRESPONDING GENERATION WITH WAY MORE BIDS AND ALL THAT KINDA STUFF. SO, SO THAT IS A TOPIC AS WELL. YEAH, I'M NOT, I'M NOT TRYING TO BE CRITICAL LOOKING BACK. I JUST THINK WE HAVE TO MAKE AN ADJUSTMENT IN OUR THINKING AND PLANNING GOING FORWARD, FORWARD GOING FORWARD. YES, ABSOLUTELY. SO BILL, UM, FIVE YEARS AGO DURING 2021 WE HAD 750 UNITS. TODAY WE HAVE OVER 1500. AND THE TREND IS DEFINITELY TO PUT MORE SMALLER UNITS IN PLACE. ABSOLUTELY. AND WHEN YOU THINK ABOUT POINT TO POINT TRANSACTIONS, ALL THESE LARGE LOADS AS WELL, WE DIDN'T HAVE, WE HAD SOME LARGE LOADS, BUT NOTHING LIKE WHAT WE'RE LOOKING AT NOW, SO RIGHT. HINT TO THINK THAT, UH, THE GROWTH IN LARGE LOADS AND THE REDUCTION IN THE SIZE AND THE MORE DISTRIBUTED NATURE OF THE, THE RESOURCES IS PROBABLY GONNA KEEP THIS GROWING. I WOULD AG I WOULD AGREE. I WOULD AGREE. JUST CHANGES YOUR PERSPECTIVE ON KIND OF WHEN UPFRONT IN TERMS OF HOW YOU DESIGN SYSTEMS. SURE. THANK YOU FOR THAT. OKAY. ANY MORE QUESTIONS FOR JP KEITH? IF NOT, [7. Future Agenda Items] JP I THINK YOU'VE GOT THE LAST ITEM. YEP, I'LL STAY HERE. AND THE LAST ITEM IS OUR FUTURE AGENDA ITEM WHERE USUALLY THIS IS A TIME WHEN I ASK, UH, IF THERE ARE ANY OTHER, UM, GUEST SPEAKERS LIKE THAT WE HAD TODAY. IF THERE ARE ANY TOPIC THAT THE COMMITTEE WOULD LIKE THAT I CAN, UH, BRING INTERESTING SPEAKERS HERE. UM, NO, AND I DON'T THINK WE HAVE TO HAVE ONE AT EVERY MEETING, SO, AND I DON'T FEEL LIKE YOU'RE FORCED TO GET AN OUTSIDE SPEAKER AT EVERY COMMITTEE MEETING. WE DON'T HAVE THAT IN OUR OTHER, DON'T HAVE TO, BUT I THINK ADD A LOT OF VALUE IF WE, IF IF IT'S AVAILABLE. YEAH. WE, WE DO HAVE ONE SORT OF PENDING AND THAT'S, JOHN MENTIONED THAT YEAH. THAT I MENTIONED EARLIER THAT WE WILL, I'M NOT SURE WHAT THE, THEIR AVAILABILITY IS FOR THE NEXT MEETING, BUT WE'LL, IN THE NEXT COUPLE OF MEETINGS, I THINK IT'D BE INTERESTING TO SLIDE THIS. THIS IS SOMEONE WHO'S BUILDING A DATA CENTER IN TEXAS. IT'S NOT AN ERCOT DATA CENTER, IT'S IN SPP, BUT THEY'RE BUILDING A DATA CENTER IN TEXAS. BUT IT'S, IT'S ALL BRING YOUR OWN GENERATION. SO IT'LL BE INTERESTING TO HEAR HOW THEY'RE THINKING ABOUT IT. AND WE'VE HAD, WE'VE HAD SEVERAL PRELIMINARY CONVERSATIONS [01:35:01] WITH THEM. I THINK, I THINK IT WILL BE QUITE USEFUL FOR THE COMMITTEE TO UNDERSTAND WHAT THEY THINK ABOUT IN INCLUDING ABOUT, YOU KNOW, SOME OF THESE FREQUENCY RIDE THROUGH ISSUES. 'CAUSE THEY'RE ESSENTIALLY, THEY'RE GONNA HAVE TO MANAGE THAT IN THEMSELVES IN, IN A MUCH SMALLER GRID DOMAIN THAN THE ERCOT GRID. YEAH, AND FEEL FREE TO EMAIL JOHN AND ME IF YOU HAVE SOME THINGS THAT YOU'D LIKE US TO BRING SO FAR WE HAVE BEEN ABLE TO BEEN SUCCESSFUL. WELL, THE OTHER THING, THE OTHER THING THAT'S COMING UP AND, AND YOU MIGHT WANT TO JUST TALK ABOUT THIS, UH, JP IS, IS WE ARE GONNA HAVE A DEEPER DIVE ON AI. UM, AND I THINK THAT'S COMING UP IN OUR DECEMBER MEETING IN THE, IN THE GENERAL SESSION. SO, UM, YEAH, IN THE EXECUTIVE SESSION WE'LL BE DEFINITELY RECORDING THE RISK PART OF IT. OKAY. OKAY. THERE'S NO OTHER QUESTIONS. THANK YOU. GREAT. SO I [8. Other Business] THINK THAT CONCLUDES THE, UM, PUBLIC PORTION OF THE MEETING. WE'RE GOING TO BREAK AND TAKE A FIVE MINUTE BREAK AND THEN GO INTO EXECUTIVE SESSION AND THEN WE'LL COME BACK WITH OUR FRIENDS FROM SANDIA. * This transcript was created by voice-to-text technology. The transcript has not been edited for errors or omissions, it is for reference only and is not the official minutes of the meeting.