BACnet Communications Troubleshooting Procedures

[kontrolphreak]

Wireshark kinda reminds me of fiddler, same idea but I take it wireshark is tracking a particular wire?

Yep that's it. Used to track wires, like DiddleDiddle.

kontrol out

[orion242]

I like to know from the top down all the Device IDs, network connections (ethernet, UDP, MS/TP) with network numbers, MAC address of each device and locations of terminators.

Wouldn't we all. Reality coming into an unknown network, there is a snowball's chance in hell your going to have all that information. What's your plan B?

You could spend days getting all the details or hours finding real issues.

Most issues are stupid simple. That said if trouble shooting goes past one day, its time to whip out more advance techs or tools. With the right tools its a quick diagnosis.

[AutoPilot]

I firmly believe a Building Owner or Building Management should keep all the system documentation for his/her building handy for the system users.

Now I'm sure most control guys on this board have come across the PC in the corner closet covered with dust and fur balls, neglected and seriously in need of upgrade.

Or when the building management decides to promote the head janitor to chief engineer (this saves them considerable coin) the skill sets required tend to be lacking a bit. They go from users to Lusers.

Plan B you ask. I have and use tools similar to those described in this thread for BACnet systems. LON is a different story. If the customer doesn't have the documentation or information needed to solve HIS/HER problem, we call that billable time and quote our hourly rate.

We're very busy so we tend to charge a little more than our competition. When the competitors fail they call back.
Around here it happens more than you think.

In most cases by the time we're done, we know all the political, mechanical, electrical, DDC, programing and system problems. Give them a report on all the issues they need to fix. Then allow them to decide tear down, renovate, upgrade or abandon what ever the problem is they have.

Our biggest problem is finding good technicians. Kids these days are lazy in my opinion. They don't have anywhere near the skills needed to understand the topics discussed in this thread. And they think they should get paid top dollar. All the while we're concerned they will get their ass electrocuted on a VFD some place and collect TDI for the rest of their life.

I think KontrolPhreak is doing a great job. I wish I had more time to spend doing something like this. But, I need to go back to work now.

[kontrolphreak]

Uploaded Optigo's pcapng used in the video files and more importantly Greg Holloway's colorization rules to the third post in this thread.

kontrol out

[stanbyyourword]

awesome , train time sorted for a few weeks

[MaxBurn]

I see the BASRTB-P with 3.0 firmware now also has diagnostics built in. The one I am offering currently doesn't have 3.0 firmware but I have contacted CC to see if I can get it updated to have these new features.

kontrol out

Hmm, looks like the BASRT-B does on the info page too. I have some older ones, anyone know if their firmware update gets you similar functionality?

[kontrolphreak]

Unfortunately the older models require to be shipped back to CC to upgrade not a simple flash. I sent my contact in regards to sending in a LX model or just the portable ones I have.

kontrol out

[MaxBurn]

Unfortunately the older models require to be shipped back to CC to upgrade not a simple flash. I sent my contact in regards to sending in a LX model or just the portable ones I have.

kontrol out

I have the normal BASRT-B and I just reached out to them. They offered to upgrade it for about a third of the cost of the device. My firmware on this one is 2.7.2 and they are offering me 3.0.3, which should start shipping in July.

I asked and have also received the firmware updating instructions with a stern warning it might brick the device. My question is what's this new diagnostic ability look like and it worth taking the risk or expense of a flash update?

This really is nice, we got the same thing in the optiflex ALC routers. Ours captures straight to a file on the device for download later though, no live data. There is an option for a continuous capture too, keeps the last 5MB of data. You can basically leave it running all the time and if you are quick you can go in and pull logs after an event.

[CylonAmerican]

You can filter a live Wireshark feed but you cannot sort it by packet type.

Live is interesting but I learn more from the data once I've stopped the capture and can sort the data back and forth. Either way, click save before you restart the capture so you can more closely analyze it later.

[kontrolphreak]

Maxburn,

The new cut sheet has some info and pictures in regards to the new features with rev. 3 firmware.

kontrol out

[y7turbo]

Thank you for spending time to put together so much useful information for everyone to see.

[kontrolphreak]

You can filter a live Wireshark feed but you cannot sort it by packet type.

Live is interesting but I learn more from the data once I've stopped the capture and can sort the data back and forth. Either way, click save before you restart the capture so you can more closely analyze it later.

With Greg's colorization rules, live captures are actually useful. When all the packets are colored the same (generally white) you can't really see much. Issues really pop out when color coded.

kontrol out

[orion242]

Colorization rules are really the key to deciphering things quickly. Also the Statistics - I/O Graph is another WS feature I find helpful. You can split out different traffic with the same style of rules leaving you with a nice timeline & number of occurrences of whatever you define.

One thing to note, if your working with large capture files applying a wad of colorization rules will bring WS to its knees even on a beefy machine. If I'm doing long term captures, I setup WS to roll the capture file at reasonable intervals to keep the files under ~100MB.

Another thing I have been playing with and finding handy is a raspi & network tap if I need longer term captures. Using a tap eliminates the need to find a hub, switch with span/mirror port and it doesn't pollute the capture with traffic from the device doing the capture. Slip it in the line and your off capturing. The PI is nice because is small enough to leave in a panel and can store many days worth of captures. You can remote in and use WS just as you would on a PC to check on how things are going and pull all the captures remotely. It cannot keep up with 1GB Ethernet and solid file transfers, but more than capable for even large BMS networks. I have it setup so once its powered up, it starts capturing rolling the log every hour. Very easy to just drop it in and pick it back up later or remotely without alot of dicking around.

[crab master]

Nicely done. A lot of great information here.

[orion242]

Visual Bacnet looks nice, bit on the fence that its worth the money over just using wireshark. Seems I normally go in spurts with capturing traffic. Might be months, years never have a need. Then a problem child comes up and it might be 24/7 hourly captures over a week. Seems they have dropped the per capture pricing for a yearly model. Can't say I haven't be able to sort things out with just WS, though having something analyze things right out of the box would save some time. Last time I had a need at a site, it turned out to be a protocol issue and a device replying with an incorrect error code + claiming it supported services that it really didn't. Have my doubts VB would have flagged much in this case since it was subtle bacnet error messages that said one thing but meant another.

One thing that the their cloud model falls apart is getting all the device info. From the looks of it, the only devices it will call out by names are the ones that had this property polled in the capture. Least that's what I'm guessing explains why some devices have names others are just listed by device instance. That's something that wouldn't be going on under normal conditions.

It would be extremely helpful if everything has its device name. Many of their own webnars going through submitted captures end up with them asking the do you know what naughty device instance XYZ is? Answer 9/10 times....Hell no, there are 1K+ devices on the network...are you kidding me?? Now if they asked device HP-21, VAV-2A3-1 or Boiler Plant...

If it was a standalone app, they could do a full discovery of the network to get all the device info then passively capture. But that's not a constant revenue stream now is it?

If you do a full discovery while WS is capturing, does the VB report have everything with its device name?

[numbawunfela]

Super thread.

I went and added pricing accurate as of the posting date.

Hardware:
KMC BAC-5051E BACnetIP/BACnetMSTP Router (with latest firmware / daignostics) - $$$
IF Tools msb-rs485 Serial Interface Analyzer - €€€ (only from Europe apparently)
B&B Electronics 485USBBT-2W - $$$
PicoScope 2205A USB Oscillascope - $$$ with probes


Software:
Wireshark - FREE
YABE MSTP Capture - FREE
KMC BAC-5051E Onboard Webpages - INCLUDED WITH THE ROUTER
IF Tools MSB - FREE
PicoScope 6 - FREE
Optigo Network Visual BACnet Website - pricing not listed

I am looking at some alternates,
LOYTEC LIP-ME201C Diagnostic BacNet router - $$$
In short, the Loytec has all lthe KMC features plus a lot more. I used to use the KMC BAC-5051E, but the Loytec is as much better than the KMC, as the KMC is better than the featureless Contemporaey controls pocket router is on the old firmware. And it has an LCD screen which I like a lot more than I thought I would.
https://www.loytec.com/products/routers-nic/lip-bacnet
It also allows for wireshark captures, which may eliminate the need for the 485USBBT-2W USB to MSTP adapter, saving $$$ (since the post listed wireshark captures as the reason for having it)

For an Oscilloscope, I am looking at the Digilent Analog Discovery 2 - $$$
https://store.digilentinc.com/analog...-power-supply/

I am pretty sure it will need some sort of adapter. I have a thread on the forum asking some questions.
It is also a power supply, voltmeter, signal generator (good for calibrating your Oscope leads) and a protocol analyzer that lists modbus and other UART RS485 protocols as ones it decodes. This may eliminate the need for the IF Tools msb-rs485 Serial Interface Analyzer, saving $$$

The Analog Discovery 2 also has Differential inputs on their Oscilloscope (instead of Single End inputs), so you can measure between + and - on the bus without shorting to each other or ground, or letting out the magic smoke on the electronics involved. Crap happens in mechanical rooms.
Also the Oscope is 14bit resolution and 30MHz bandwidth with 100MS/s sampling vs Pico 2205A at 8bit, 25MHz, and 200MS/s.

I was just looking at the possibility of needing a protocol analyzer when I found this thread with someone who has actually been using one. So my list of possible substitutions are all speculations of someone looking for input from someone who knows.

[orion242]

r letting out the magic smoke on the electronics involved. Crap happens in mechanical rooms.

+/-25v on the inputs, so if "crap" is 24vac on the line...she still might be a smoken. For a device you could toss in your bag, its pretty flexable for the price.

[numbawunfela]

+/-25v on the inputs, so if "crap" is 24vac on the line...she still might be a smoken. For a device you could toss in your bag, its pretty flexable for the price.

I noticed that too. Hard to cover ALL bases. But I did notice it had Arduino style mode boards that plug into it. Like this
https://store.digilentinc.com/analog...-2-pro-bundle/
Then I can use standard BNC Oscilloscope leads. Some of these allow for x10 measure, so I can read 250v power on my 25v inputs. And presumably if I smoke anything, it is the $$$ BNC mod board and not the $$$ main unit. Still getting clarity on that.

Again, I know I do not have all the answers, when the rubber meets the road weird things happen.
If Mr Phreak has any opinions on the protocol analyzer I would appreciate that.
I LOVE the idea of seeing Modbus traffic. I really hate the all or nothing nature of Modbus. That is super hot.

[kontrolphreak]

Guys, no pricing in the open forum. Please edit your posts or it will be done for you.

kontrol out

[kontrolphreak]

Will probably pick one of these up to play with, cost appears to be low enough to see it's capabilities. I also subscribe to "Two is one and one is none" philosophy. Was one a project where I couldn't bring my laptop into the building and this would have been okay if I needed an oscilloscope / protocol analyzer.

Xminilab Portable

kontrol out