Thread Number: 82755
/ Tag: Other Home Products or Autos
Astoria Arc Fault and Manhattan West Side Outage |
[Down to Last] |
Post# 1069126   4/24/2020 at 22:47 (1,456 days old) by chetlaham (United States)   |   | |
Checkrate/Likes
 
     
The official cause finally revealed in newly released NERC PDFs;
Astoria: www.nerc.com/pa/rrm/ea/Le... West 65th st: www.nerc.com/pa/rrm/ea/Le... Great reads, absolutely loved them. Good lessons on avoiding common mode failure and double checking wiring thoroughly. |
|
Post# 1069127 , Reply# 1   4/24/2020 at 22:54 (1,456 days old) by chetlaham (United States)   |   | |
Checkrate/Likes
 
     
|
Post# 1069146 , Reply# 2   4/25/2020 at 07:39 (1,455 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
1    
Never having been privilege to the exact workings of power transmission and distribution, nonetheless can deduce the dynamics to which the unfamiliar terminology refers. At former address (Euless TX) we had consistent/persistent outages in the 3-hour range. I asked the utility for this kind of data. All they sent was an incomplete/inaccurate spreadsheet down to which feeders were open. Like it was either an embarrassment to them or some kind of national security secret.
FAASScinating. As HAL explained in 2001, "This kind of thing has cropped up before, and it has always been due to human error". Unlike in early-season Saturday Night Live spoof of 3 Mile Island, electricity is not just sent to your house to make toast. The amount of power involved is well capable of destroying the equipment handling it. Most conceivable fault/error conditions are anticipated and sensing/safety devices installed to interrupt power before the house burns down. Like the breakers in your house only HUGE, networked, triple redundant.
The added complication also unfort'ly adds opportunity for failure. Many/most of those potential failures are also anticipated and safeguarded, but it is not possible to map EVERY possible sequence of events. When this kind of thing happens, like in aircraft crashes, they are analyzed in minute detail for the purpose of future prevention. Thus "the grid" is as reliable as it is, despite being operated somewhat equivalent to driving your car almost as fast as it will go for hours at a time, day in and day out.
Give these PDFs a patient, redundant read. Knowing what many here know of sequential control, you'll be able to make more sense of them than at first glance.
|
Post# 1069164 , Reply# 3   4/25/2020 at 09:44 (1,455 days old) by chetlaham (United States)   |   | |
Checkrate/Likes
 
     
Thanks
I disagree on the car analogy, T&D systems encounter peaks and troughs varying in magnitude based on the season, with the heaviest load being only on several of the hottest days of the year. In those cases the system is often near the maximum limits, only allowing for N-1 and N-1-1 contingencies often times. Typically the system is running below thermal limits most of the time. Although there are periods during the spring and fall when load is light where equipment outages are scheduled making the system more vulnerable than would otherwise be. Protection on the bulk power system (BPS) is usually doubled with two relays per HV device (transmission line, transformer, cap bank, ect) being protected each fed from its own 125 volt DC battery bank. Each relay has its own trip circuit connected to its own dedicated trip solenoid in the breaker(s). Typically each relay is from a different manufacturer or in the very least a different make and model. The thing about Astoria is that they used the same multiplexers on both the primary and secondary protection in all 3 interconnected substations. So they all had the same failure mode. Step distance protection would have been the last resort, but that requires a proper 3 phase voltage input which was lost from the CCVT (voltage sensing transformer) failing in the first place. |
Post# 1069191 , Reply# 4   4/25/2020 at 10:57 (1,455 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
     
Car being only a coarse analogy. That the system sometimes is going 'as fast as it can go' is derived from the times the utility comes on TV and begs us to reduce AC demand, or cycles us on and off. Analogy breaks down in that, yes, that's nowhere near 'all the time'. Still, it does happen and dependent on weather extremes, with some regularity. The system has to cut back to save itself. The analogy is strong inasmuch as, if any underlying weakness exists in your car, operating it to its limit is more likely to reveal that weakness.
These 2 particular failures were NOT caused by operating the system at or near its limit. They were more like, you're driving along at 35mph and your cam belt breaks. What happens next depends on engine design. You either need a new cam belt ($1000 installed) or a whole new engine ($6000 installed).
Not to confuse anyone by drawing these analogies closer to familiar items, but to provide some context for understanding what to most would be about as alien as how flying saucers work.
Somehow escaped an edit, was 'thanks Chet' for providing this. And if/when I misstate, by all means straighten me out. You know this stuff. I can only about 3/5ths follow it. |
Post# 1069204 , Reply# 5   4/25/2020 at 11:23 (1,455 days old) by chetlaham (United States)   |   | |
Checkrate/Likes
 
     
Agreed. Failures tend to show up when things are pushed to their design limits. Con Edison networks are notorious for proving that- cables burn up on the hottest days necessitating voltage reduction or shedding load...
Yup- this was failure of the protection system to remove a short circuited device. Equipment fails semi often. On any given day a country like the US will see at least one power system component fail. Small pole mounted transformers can be about dozen a day during heat waves. Often times we don't notice it- or at most just see a brief blink of the lights- unaware that a transmission line or transformer the size of a house just tripped out. Breakers open and isolate the device in about 1/20th of a second and the redundancy/replication of the grid takes over for the now missing segments. If curious here is what a 138,000 volt CCVT looks like along with its nameplate: https://www.ebay.com/itm/Capacitor-Volta... These takes transmission level voltages and step them down to about 120 volts for protective relays and grid monitoring- the remote voltage readings grid operators see in their control room is taken from these devices. |
Post# 1069221 , Reply# 6   4/25/2020 at 12:16 (1,455 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
     
|
Post# 1069330 , Reply# 7   4/25/2020 at 23:35 (1,455 days old) by chetlaham (United States)   |   | |
Checkrate/Likes
 
     
I think so too- its really cool that it was giving odd voltage readings before it failed- possibly (guessing) due to shorts in the strings but I can only make bets.
The top part is literally a stack of capacitors. If you tap that stack in the right location you get your desired voltage. That voltage is then fed into a small iron core reactor in the unit to offset the phase shaft of the capacitors (leading power factor off set with a lagging power factor), then into a classical two winding transformer where 5kv goes down to 120 volts leaving over to the relay house in the station yard. |
Post# 1069366 , Reply# 8   4/26/2020 at 07:06 (1,454 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
1    
My field was videotape. Few voltages over 80VDC and none over 240VAC. Power T&D is a whole 'nother world. The crossovers are instrumentation/metrology, feedback, control. Or I'd be pretty much lost.
Dad was apparatus sales engr/mgr for Westinghouse. But he wasn't so much a talker at home. More a yeller. |
Post# 1069386 , Reply# 9   4/26/2020 at 09:06 (1,454 days old) by chetlaham (United States)   |   | |
Checkrate/Likes
 
     
|
Post# 1069448 , Reply# 10   4/26/2020 at 20:14 (1,454 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
1    
Meteor- and metr-. Metr- being measurement. Like the capacitive transformer. In NTSC video we measured primarily in quarter cycles of 3.579545 MHz, which for mechanical parts can be counted down to horizontal or vertical rates.
Early machines had to wait 16ms to measure vertical rate error and generate a correction, then another 16ms to see the result of the correction. The result was overshoot and hunting. From startup it took the machine ~5sec to sync to reference.
With the practical advent of digital logic (ca. 1974) it became possible to perform that vertical correction example in terms of horizontal rate or 64µs. That is, the correction became a predicted ramp incremented at H rate and the next vertical sample error would be at or near zero. Sync time now became 200ms worst case, 50ms nominal, a performance increase of 100x in a time-critical application.
In practice, the correction ramp was updated at capstan optical tach rate, ~3KHz or every 5 horizontal increments. The capstan being the mechanical timing element for vertical rate. Full lockup included horizontal timing and a similar method was applied to the rotary transducer (headwheel). |
Post# 1069481 , Reply# 11   4/26/2020 at 23:50 (1,454 days old) by chetlaham (United States)   |   | |
Checkrate/Likes
 
     
|
Post# 1069505 , Reply# 12   4/27/2020 at 06:12 (1,453 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
1    
Just the metrology, principally measuring/correcting timing/positional errors in electromechanics. Broadcast video is a pretty-well controlled environment.
Having this many closed loop mechanical systems in one machine made for some very entertaining times. Whether watching it work or watching it fail and figuring out why. This machine was so precisely constructed that most subassemblies could be removed and replaced exactly back where they belonged with 2 or 4 allen bolts.
Lotta useless knowledge now, broadcast today employs no user-serviceable moving parts. BOOOOR-ing.
Link is to a video of the machine sequencing short segments from various angles. It rewound/replaced the played tape exactly where it found it, loaded and cued the next one in 7 seconds. That carrousel that holds the cassettes could literally take your arm off if you got in the way.
Videography not the best. And it was pretty inadvisable to shine bright lights inside it while it was running, as many of the sense elements were optical. The operational sounds it made are downright musical. The machine and I were each other's best friends. CLICK HERE TO GO TO arbilab's LINK
View Full Size
|
Post# 1069523 , Reply# 15   4/27/2020 at 08:48 (1,453 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
     
Many markets were like that. The Ampex machine cost about twice what the RCA cost. It was much faster, smoother and more versatile but 'most' stations just wanted 'a cart machine' and beyond that they didn't care.
I went to work at NBC-OKC precisely because they were an Ampex house. We had 4 generations of Ampex 2" recorders, from the first transistor model to these digital ones to the next digital one (AVR-3) and from there 1-inch was taking over, largely because the tape cost less than half as much. I knew all of them inside and out, as well as 3 generations of audio machines, switchers and effects boxes, all the associated Tektronix instrumentation. The transmitter guy did only that, couldn't fix a monitor. Fine by me. |
Post# 1069651 , Reply# 17   4/28/2020 at 06:14 (1,452 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
     
Power is 10:1 step DOWN? That's not how we used to do it, is it? Station got SS Larcan (VHF 4) just before I left. I had some idea what went on in the old RCAs. No idea what goes on in SS. Or digital. Pay me $60K and I might care.
Ampex was not ALL genius. Here's a dirty story about VR 1100/ VR 2000, the ones there were the most of. You could spend all day or all week aligning the servos and the machine still wouldn't lock full V&H on any tape but its own. Everyplace I worked had at least one machine with this problem.
Turns out they had specified the capstan to a very tight tolerance. BUT the motor and flywheel pulleys had just as much to do with tape speed and their tolerances were 'generous'. If the pulley tolerances stacked, the velocity loop would go all the way to end of range, leaving nothing for the phase loop to work with and it would stall just outside vertical lock. Which also inhibited horizontal lock and playback would be unusable.
That's the only problem I never solved by myself. Never imagined 'wrong size pulley' was possible. Read about it in an industry mag and sure enough, replaced the entire capstan assembly and it worked right for the first time in 20yrs.
Not to diminish Ampex achievement. Complex as they were, they made very little trouble over very long lifetimes. The story on RCAs was, jigger this, jigger that, it was always something.
|
Post# 1069677 , Reply# 19   4/28/2020 at 09:55 (1,452 days old) by chetlaham (United States)   |   | |
Checkrate/Likes
 
     
|
Post# 1069754 , Reply# 20   4/28/2020 at 18:34 (1,452 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
     
|
Post# 1070008 , Reply# 21   4/30/2020 at 07:02 (1,450 days old) by chetlaham (United States)   |   | |
Checkrate/Likes
 
     
|
Post# 1070020 , Reply# 22   4/30/2020 at 07:49 (1,450 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
1    
Those too. Wish I had my 1973 Panasonic twintub back.
Besides the nostalgia, there were practical elements. That machine doubled my salary in return for reducing its failure rate from 1 commercial lost per day, to 1 per week. The chief kept records of that. It also let me have on-call weekends off.
BTW there were two of them and both were that well behaved once I got ahold of them. The guy before me was 'good', but I'm obsessive when it comes to machinery. |
Post# 1070367 , Reply# 24   5/1/2020 at 22:30 (1,449 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
     
4160 eh? Count me out. I'm not too uncomfortable with 25~30kV monitors, the danger parts are well known, easily avoided. Even if they do bite you, there's no real oomph behind it.
Nothing like power mains can do. Anything over 240V is pretty unforgiving. One false move is all it takes. When you get up in the kVs, that shxt doesn't necessarily wait for you to blunder into it. It can leap.
Thanks, I'll maintain social distance as the phrase goes these days. To me, social distance from 4160 is in the next room. With the door closed.
|
Post# 1070420 , Reply# 26   5/2/2020 at 06:35 (1,448 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
2    
Pardon the expression, that'snuff for me too. Not the scaredy type, but certain levels of danger I'd sooner avoid.
I was fine with heights. Climbed straight up the front of Diamond Head hand over hand, cliff jumping into water, going up 1400ft tower elevators. Until the third time. Suddenly I was terrified and have been ever since. Now get a little shaky just watching 'Emergency'. What changed? Turning 40 maybe. Lost my immortality. |
Post# 1070596 , Reply# 28   5/3/2020 at 08:07 (1,447 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
     
More people are killed by chevrolets than ferraris. Not because chevrolets are that much more dangerous, there's just a helluva lot more of them.
I've brushed across 120V a number of times. Gets my attention, is about all. Bee sting hurts worse. Graciously, my skin resistance is over 10X normal. That doesn't help much at 240V and not at all at kV. |
Post# 1070674 , Reply# 30   5/3/2020 at 19:09 (1,447 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
     
|
Post# 1070754 , Reply# 32   5/4/2020 at 07:03 (1,446 days old) by arbilab (Ft Worth TX (Ridglea))   |   | |
Checkrate/Likes
 
     
|