So, the first thing I do when I have a troublesome MSTP (or even Modbus) is to throw an oscilloscope on the network to see the physical characteristics of the electrical signal. This will at a minimum let you know if you have a live wire (data packets being transported) as well as any interference or noise on it. I also generally do this when starting up new network segments to get a baseline on its physical characteristics. With Modbus I have had issues with getting devices to communicate, this initial test will at least let you know if there are data packets being transported on the wires. If there are, its more than likely a configuration issue.

I used to use a cheap Oscilloscope from another vendor which did the above but went to the PicoScope 2205A for its added software capabilities of math functions and serial decoding. These allow you to actually view the data packets bits and will let you know if you are getting clean data or corrupt packets. The easiest way to determine this is to look for the BACnet preamble of the packet if your oscilloscope has serial decoding capabilities.

Attached are some screen captures and a short video from both a live site and my test bench. The live sight has 24 controllers and the network is approximately a 1000’ feet in length, probably a couple of splices (one definitely that I know of). It was installed over a decade ago and was having some serious issues (latency) until all the previous BACnet controllers were removed and the wiring cleaned up a couple of years ago. It’s a crazy network (will do a drawing and upload in the future) that goes from IP to MSTP then in the field back to IP before going to MSTP sub-networks (IP to 38.4Kbs to IP to 76.8Kbs). The screenshots are of the primary MSTP network only. The second set of screenshots are on the test bench with just 3 controllers at 76.8Kbs and is about 5' in length. The names of the screenshots describe the how many EOLs are active or not. As you can see with the networks with EOL issues, all the data packets are still being transported with valid bits, the final byte is delayed on the networks with EOL issues. Also something to take note of is the condensed packet length on the 78.6Kbs data compared to the 38.4Kbs data. I run all my networks at 76.Kbs whenever possible.

Looks like I can't upload mp4 files (or zip them even though the upload manager indicates that zip is a valid upload type of file). Which sucks as some of my captures (IF Tools/Yabe/BAC-5051E) are only videos. If you want to see them send me an email and I'll shoot them over.

You can also use a standard VOM to do some electrical troubleshooting of IEA485 (MSTP/Modbus). I don't do this but here is a good video from KMC on the basics:

YouTube link to oscilloscope capture:
https://www.youtube.com/watch?v=P5N9vRKD4Vw&feature=youtu.be

[video=youtube;p1bK88CoOnw]https://www.youtube.com/watch?v=p1bK88CoOnw[/video]

kontrol out

 

Generally, the second step is using Wireshark to view and decipher the BACnet MSTP packets. To do this I use the B&B Electronics 485USBTB-2W USB to IEA485 devices with YABE’s MSTP capture application. This is installed when you install YABE and I like it due to the ability to suppress the capture of certain packets. It works with Wireshark but you have to go into Wireshark’s Capture Options and manually create the pipe (the path is displayed in the capture tool interface).

I’m competent at deciphering the packets but no guru by any means, there are a couple of good videos on YouTube on the subject (see below). Once you have done a capture for 5 to 15 minutes you can start looking at the packets yourself or if you have a Visual BACnet account you can upload it and have them create a report of your network for you (stay tuned for a future post on Visual BACnet).

As this and the previous post’s connections are passive there should be no concern of effecting the normal data flow on the wires.

YABE BACnet MSTP Capture Video on YouTube:

https://www.youtube.com/watch?v=XQLt0-MMwrI&feature=youtu.be

[video=youtube;14rJYjmxUcM]https://www.youtube.com/watch?v=14rJYjmxUcM[/video]

It will only allow one video to be inserted so the other two are links to the videos on YouTube.

https://www.youtube.com/watch?v=8asoKlP7iFA
https://www.youtube.com/watch?v=GZkSpAqgvck

I had wanted to attach the capture form the site but once again it is not an allowed type of attachable file.

Optigo and Greg Holloway have been kind enough to allow me to upload the pcapng files and his colorization rules (a must have if you're using Wireshark with BACnet) from the above video. You will need to change the extension from .pdf to .zip once downloaded. The Wireshark Display Filters Word document is something I had on my laptop, unfortunately I don't remember where I got it so I can't give credit where credit is due.

kontrol out

 

Another tool to have in your toolbag of tricks is the IF Tools msb-rs485 Serial Interface Analyzer. I feel it is a jack of all trades but a but haven't used it as much as an oscilloscope and Wirehark. It is costly but for what it does it is probably cheaper than buy a bunch of different devices.

It has the combined features of a basic oscilloscope and a protocol anaylser. I am a firm believer of the two is one and one is none rule, so this covers both the oscilloscope and USB/serial capture abilities of Wireshark for the two devices above. I have uploaded a basic video of all the features to YouTube (first video upload WooHoo! another first brought on by HVAC-Talk). And will upload a few more in the near future. Below I'll list some of the features of the MSB.

The first feature is a Protocol Scanner which will allow you to scan the IEA485 and determine your Buadrate and Protocol (Data Bits/Parity/Stop Bits). Once these have been determined you can use the configuration in you current session, or if you already know them you can manual configure the session.

The Data Monitor will give you the raw hex (or can be configured for Dec and ASCII) values of the data packet which if you understand how a BACnet packet is constructed can tell you things like the type of packet (token/poll for master/etc.) as well as the destination address and sending address. Once again with BACnet MSTP packets I look for the preamble (55 and FF) to start decoding the packets.

Not really sure what the Event Monitor does, but the help files is extremely detailed and thorough (those Germans are engineering masters, they'll write a 50 page document on how to tie your shoelaces).

The Protocol Monitor displays the raw data in a human friendly format - Preamble/Frame Type/Destination Address/Sending Address/Length/Header CRC/Data. It has a filter but once again haven't really messed with it.

The final one is the Signal Monitor which shows you the electrical characteristics of the signal like an oscilloscope. It only shows you the packets and will not show any noise, interference or floating voltages like a true oscilloscope.

Multiple windows can be open at the same time and synced to show the data currently.

The session can be saved and reviewed at a later time.

Most of the data can be exported as a .csv file as well.

[video=youtube;eENBQw_6YNM]https://www.youtube.com/watch?v=eENBQw_6YNM&feature=youtu.be[/video]

https://www.youtube.com/watch?v=fOfuKxsSK_I&feature=youtu.be

https://www.youtube.com/watch?v=9d3JdMmvaWk&feature=youtu.be

kontrol out

 

only shows you the packets and will not show any noise, interference or floating voltages like a true oscilloscope.

That's because this guy is a logic analyzer. LAs typically only see signals as 1s or 0s but measure precise timing over a load of channels. So noise, ringing, etc will not show up like it would on a scope. Simple way to think about it, scope is an analog view, LA is a digital view of the signal. Screens may look similar but they are different tools.

 

Another great tool is Optigo Network's Visual BACnet. It is a web based service that you can manually (or a scheduled automatic upload) of captured pcapng formatted files to be analyzed and presented in a format that is easier for humans to comprehend. I have used it on and off and have found that it quickly enables you to drill down and diagnose problems on your network. The BACnet Browser pages for network source and destination will really quickly show your "noisy" devices to take a look at.

Device 1 in the screenshots and detailed report is the primary BACnet/IP to MSTP router of the convoluted network shown in the post below. As all traffic to the front end passes through it you can see that it is the most active controller on the network by far. It also routes all the data packets from the sub-networks hanging off the controllers on it's buss (not the optimal network design but it was how the previous contractor had installed the original system and the client did not want to install wiring to convert the units with sub-nets to BACnet/IP).

The screenshots and attached report have shown me a couple of issues on the network I was scanning that I will have to resolve in the near future. There are too many views/pages to go into here from their website, but you can generate reports that are downloadable as .pdf files which I have attached to this post as well. The are both of the same capture, one being a basic one and the other more in depth.

kontrol out

 

Placeholder 5

kontrol out

 

Placeholder 6

kontrol out

 

Thanks! This is an awesome idea, MS/TP issues are a major time sink for me.

 

I just got a Pico and have used it a couple of times. I've seen some really weird things but I have no idea what it exactly means, what to look for, what could be causing it, etc.

 

Most vendors will have MSTP wiring guidelines that may contain images of oscilloscope screenshots and what they are depicting. I know Reliable Controls and ALC both do. My suggestion is set up a testbench and create some of the common misconfigurations in both the hardware (wiring/EOL/Elect noise) and firmware (MAC address/DEV#/Buad Rate).

Also getting you oscilloscope parameters set up for you application is key. After using an oscilloscope for the past couple of years I'm just beginning to use it confidently enough to start this thread, but I'm still only a level 1 scope wizard.

Basic rule of thumb is if it doesn't look right it probably isn't.

Will get the third post up this afternoon/night.

kontrol out

 

You should be able to put the videos on Vimeo or YouTube and link them here, if you want.

 

Thanks for putting this together, looks great so far. May it should become a "Sticky" post so that it is always easy to find.

Kevin

 

So, I thought we would make this a little more interesting. I have attached a network riser of the project I was doing the scans of for this thread and want to throw a challenge out. If professional member can produce a more convoluted single site out there with a Wireshark capture to show stable communications I will give the lucky winner a Contemporary Controls BASRTP-B for their laptop bag.

https://ccontrols.com/basautomation/basportable.php

I will be updating the posts above with the Visual BACnet upload of the primary BACnet/MSTP@38.4Kbs network, which though isn't perfect for what it is it's working well. I currently have a bad MSTP controller that needs to be replaced on the primary network as well as a couple of "ghost controllers" that have been removed from the system but still have controllers sending"Who-Is" to them that I need to clean up. Visual BACnet's report has help greatly in tracking these down.

The second pdf attached is actually the pcapng file. To view download it and change the .pdf back to .pcapng and open with Wireshark.

kontrol out

 

I see the BASRTB-P with 3.0 firmware now also has diagnostics built in. The one I am offering currently doesn't have 3.0 firmware but I have contacted CC to see if I can get it updated to have these new features.

kontrol out

 

Hmm, looks like the BASRT-B does on the info page too. I have some older ones, anyone know if their firmware update gets you similar functionality?

 

Yep, that is what I was saying in non technical terms. It's a jack of all trades not a master of one.

kontrol out

 

Awesome post! Thanks for all the info in one spot. Was going to mention the kmc multimeter one, very cool for the install techs and checkout.
Nice job. Got some watching to do.


Sent from my iPhone using Tapatalk

 

Interesting post you have going here.
I don't mean to complain, but I've been troubleshooting for over 20 years.
Over the years, pounded into my head is the one most important thing missing on your convoluted network diagram.
I have a Picoscope as well, but long before I pull it out the first thing I'll do now is map out a network.
I like to know from the top down all the Device IDs, network connections (ethernet, UDP, MS/TP) with network numbers, MAC address of each device and locations of terminators.

Many times a problem network will have just one or two devices configured in way that can trash a perfectly wired network. Every now and then I do run across a malfunctioning controller, or some dumb ass that connects a new device in a star wired to the closest controller and puts a terminator on the end of the wire.

Other issues I've seen tend to do with improper grounding, wrong type of cable, and in one case over 400 feet of cat5 cable with 2 pairs of wires twisted in parallel for both Tx and Rx used for MS/TP.

I've seen allot of stupid $hit and go on and on, but the point is if the network will or won't work on paper there's little chance of getting it to work in the wild.

just my $0.02

keep up the good work

 

I like to know from the top down all the Device IDs, network connections (ethernet, UDP, MS/TP) with network numbers, MAC address of each device and locations of terminators.

Wouldn't we all. Reality coming into an unknown network, there is a snowball's chance in hell your going to have all that information. What's your plan B?

You could spend days getting all the details or hours finding real issues.

Most issues are stupid simple. That said if trouble shooting goes past one day, its time to whip out more advance techs or tools. With the right tools its a quick diagnosis.

 

I hear you on the network documentation. I have all of that in Excel but there was no graphical representation so I drew up the network diagram real quick. I'll update it for the client in the future and upload it here then.

kontrol out

 

Wireshark kinda reminds me of fiddler, same idea but I take it wireshark is tracking a particular wire?

 

Yep that's it. Used to track wires, like DiddleDiddle.

kontrol out

 

I firmly believe a Building Owner or Building Management should keep all the system documentation for his/her building handy for the system users.

Now I'm sure most control guys on this board have come across the PC in the corner closet covered with dust and fur balls, neglected and seriously in need of upgrade.

Or when the building management decides to promote the head janitor to chief engineer (this saves them considerable coin) the skill sets required tend to be lacking a bit. They go from users to Lusers.

Plan B you ask. I have and use tools similar to those described in this thread for BACnet systems. LON is a different story. If the customer doesn't have the documentation or information needed to solve HIS/HER problem, we call that billable time and quote our hourly rate.

We're very busy so we tend to charge a little more than our competition. When the competitors fail they call back.
Around here it happens more than you think.

In most cases by the time we're done, we know all the political, mechanical, electrical, DDC, programing and system problems. Give them a report on all the issues they need to fix. Then allow them to decide tear down, renovate, upgrade or abandon what ever the problem is they have.

Our biggest problem is finding good technicians. Kids these days are lazy in my opinion. They don't have anywhere near the skills needed to understand the topics discussed in this thread. And they think they should get paid top dollar. All the while we're concerned they will get their ass electrocuted on a VFD some place and collect TDI for the rest of their life.

I think KontrolPhreak is doing a great job. I wish I had more time to spend doing something like this. But, I need to go back to work now.

 

Uploaded Optigo's pcapng used in the video files and more importantly Greg Holloway's colorization rules to the third post in this thread.

kontrol out

 

awesome , train time sorted for a few weeks

 

Unfortunately the older models require to be shipped back to CC to upgrade not a simple flash. I sent my contact in regards to sending in a LX model or just the portable ones I have.

kontrol out

 

I have the normal BASRT-B and I just reached out to them. They offered to upgrade it for about a third of the cost of the device. My firmware on this one is 2.7.2 and they are offering me 3.0.3, which should start shipping in July.

I asked and have also received the firmware updating instructions with a stern warning it might brick the device. My question is what's this new diagnostic ability look like and it worth taking the risk or expense of a flash update?

This really is nice, we got the same thing in the optiflex ALC routers. Ours captures straight to a file on the device for download later though, no live data. There is an option for a continuous capture too, keeps the last 5MB of data. You can basically leave it running all the time and if you are quick you can go in and pull logs after an event.

 

You can filter a live Wireshark feed but you cannot sort it by packet type.

Live is interesting but I learn more from the data once I've stopped the capture and can sort the data back and forth. Either way, click save before you restart the capture so you can more closely analyze it later.

 

With Greg's colorization rules, live captures are actually useful. When all the packets are colored the same (generally white) you can't really see much. Issues really pop out when color coded.

kontrol out

 

Maxburn,

The new cut sheet has some info and pictures in regards to the new features with rev. 3 firmware.

kontrol out

 

Thank you for spending time to put together so much useful information for everyone to see.

 

Colorization rules are really the key to deciphering things quickly. Also the Statistics - I/O Graph is another WS feature I find helpful. You can split out different traffic with the same style of rules leaving you with a nice timeline & number of occurrences of whatever you define.

One thing to note, if your working with large capture files applying a wad of colorization rules will bring WS to its knees even on a beefy machine. If I'm doing long term captures, I setup WS to roll the capture file at reasonable intervals to keep the files under ~100MB.

Another thing I have been playing with and finding handy is a raspi & network tap if I need longer term captures. Using a tap eliminates the need to find a hub, switch with span/mirror port and it doesn't pollute the capture with traffic from the device doing the capture. Slip it in the line and your off capturing. The PI is nice because is small enough to leave in a panel and can store many days worth of captures. You can remote in and use WS just as you would on a PC to check on how things are going and pull all the captures remotely. It cannot keep up with 1GB Ethernet and solid file transfers, but more than capable for even large BMS networks. I have it setup so once its powered up, it starts capturing rolling the log every hour. Very easy to just drop it in and pick it back up later or remotely without alot of dicking around.