Ferus Nitrogen Plant Devicenet
Devicenet is a bus communication system invented by Allen Bradley, and is used to communicate commands to equipment attached to breakers in the motor control centers MCC). There are other applications where they don't have to go through the MCC, using different Devicenet modules and cables. I mention this because I had to work on both systems at the Ferus Nitrogen plant East of Strathmore, Alberta. My involvement with Devicenet is due to the fact that DeltaV can interface to the bus through DeltaV Devicenet card. Essentially the DeltaV GUI, what the operator sees on his screens, can manipulate pumps and motors through the Devicenet card and to the MCC equipment.
I was called out one very stormy day, a day where tornadoes were predicted to touch down near Strathmore. When I was in the building I thought the big bay doors were going to flip right off in the wind, it was so strong. Regardless, I had to troubleshoot why their Devicenet system, the one that didn't connect to the MCC, did not work. Two electricians said they had been working on it for 2 days and just could not figure out why only half of the network was functional, the other half was not. Well, to me, that's a clue right there.
Amidst all the claims by the electricians that they checked here many times and didn't find anything wrong, I did some basic checks like tightening the screws on the terminal blocks. All were tight. Then I performed a basic test: I pulled on the wires in the terminal blocks. One out of the 5 wires came out. Here was the problem. With the type of terminal block used, the carrier inside did not retreat back to an open position when the screw was backed off. The guy who hooked this system up likely pushed the wire into the terminal block and screwed the screw down. When the screw bottomed out, he figured the wire was tight. A simple pull test would have shown that he did not get the wire in correctly. (When I was teaching my Power and Ground course, I told this story as part of a method of troubleshooting). I stuck my screwdriver into the terminal block to reset the block inside, put the wire in, screwed it down, then did a PULL TEST. Voila. Problem solved.
My second story is a call I got Friday night about 6 PM. The operator told me that there was something wrong with Devicenet and lot of equipment did not work. He said he had been testing something and blew a Devicenet card, now lots of stuff was inoperative.
Apparently with his refrigeration papers, he had an electrical rider attached that
allowed him to do limited work on electrical. He figured this made him an expert so he
was trying to troubleshoot a problem with a motor starter (basically a big relay with
overload protection and a disconnect switch), using his trusty little Radio Shack digital
volt meter (DVM). An industrial type DVM states the maximum voltage on the device
that can be safely measured without arcing occurring. My Fluke meter for example
says it's good for 1000 volts. The operator's cheap instrument was good for
something like 400 volts, and he was measuring in a 500 volt system! This and the
fact that he might have touched the wrong wire caused a big arc in the breaker box,
blowing his meter and the Devicenet card. When I got there and inspected the
damage, the entire 1 cubic foot enclosure was totally covered with carbon from the
arcing.
Did they have spare cards? Maybe. He found a couple in storage and I replaced this one card. I don't recall if it made this device work in the system or not. No matter, because many others did not work either. I did some checks and found 3-4 that did not work, so called the guys boss to tell him this, because I did not have enough parts to fix them all. He wouldn't believe what I was telling him, that so many devices were shot, so called his electrician who was in Medicine Hat, to come out and verify my claims.
While waiting for the electrician, I started to draw out the bus system. It was a real mess. Because it was a combination of old equipment and new, there was a mix of technologies that made it hard to determine how the equipment was supposed to work. The bus itself was all convoluted with drops and trunk connections all is disarray. Termination resistors were in the wrong place because when they put the two systems together (old and new), they didn't check how the old system was actually hooked up. Cables even went back on themselves in a loop rather than just go from one cabinet to the next. Once I had my drawing and the electrician arrived, we marked off which devices did not work from DeltaV. The two DeltaV cards were okay, but about 20 cards for the MCC were destroyed. The electrician was able to bypass a few critical motors so that the process could still run, but without parts, it was tough to get all the other things going. The plant was down, so we left the work for the next day, Saturday
Two electricians and the plant boss were there. I didn't have to be there but they wanted me to help since I had done the drawing and knew a little bit about the system. Plus I could check out stuff with my Process Meter and with DeltaV. Some Devicenet modules were static addressing, others were automatic addressing as soon as they connected. This caused us a lot of grief, since the auto addressing would take an address reserved by one of the static modules. We worked our way through it, replaced a couple modules, and took modules from non-important equipment to put into the critical pumps and fans, since the parts we needed would not be available for another week.
I put in 3 days of overtime, and so did the electricians. The cost of some of the modules that were fried was in the thousands of dollars. All together, not counting lost production, the operator's little test cost Ferus about $60,000. The operator wasn't fired and to his credit, he did admit everything he did, not lying about anything as I would have done. (I would have blamed a tornado or something).