Page 1 of 4 1234 LastLast
Results 1 to 13 of 44
  1. #1
    Join Date
    Dec 2006
    Location
    What? Who? Where?
    Posts
    2,220
    Post Likes

    BACnet Communications Troubleshooting Procedures

    For a while now I have wanted to do a thread on BACnet MSTP troubleshooting and decided today is the day to start it.
    This is going to be a long and detailed thread on what I do when I am working with a troublesome BACnet network. This thread will be mainly MSTP and I might do another in the future on IP or just add to this thread. Give me some time to get this completed as I have a bunch of screen captures, videos, a couple of reports and links to YouTube videos on the subject.

    This initial post will be about the hardware and software I use to diagnose and interrogate BACnet networks. I will also be reserving a few posts after this one to add the above mentioned attachments and a write up on each one.

    So lets gets this started......................................

    All items below are available from the uberwebz with no restrictions other than having the $$$ to purchase or download if freeware.

    Hardware:
    Lenovo Thinkpad Laptop
    KMC BAC-5051E BACnetIP/BACnetMSTP Router (with latest firmware / daignostics)
    IF Tools msb-rs485 Serial Interface Analyzer
    B&B Electronics 485USBBT-2W
    PicoScope 2205A USB Oscillascope
    Fluke 87 Multi Meter

    Software:
    Windows 10 64 Bit
    Wireshark
    YABE MSTP Capture
    KMC BAC-5051E Onboard Webpages
    IF Tools MSB
    PicoScope 6
    Optigo Network Visual BACnet Website
    Snagit Screen Capture Software

    That's about it for hardware and software.

    kontrol out
    Last edited by kontrolphreak; 05-18-2018 at 12:25 AM.
    "Good" - Jocko
    "Open is as open does." - Forrest Gump
    "Can't we all just get a Lon?" - Garry Jack
    "BACnet: integration or interrogation?" - The Janitor
    "Open protocols? You can't handle open protocols!" - Nathan R. Jessup
    “What’s that? Aaa… open protocols? Don’t talk about…. open protocols? Are you kidding me? Open protocols? I just hope we can hardwire an interface!” - Jim Mora Watch it here!

  2. Likes rjk_cmh, stanbyyourword liked this post
  3. #2
    Join Date
    Dec 2006
    Location
    What? Who? Where?
    Posts
    2,220
    Post Likes
    Thread Starter
    So, the first thing I do when I have a troublesome MSTP (or even Modbus) is to throw an oscilloscope on the network to see the physical characteristics of the electrical signal. This will at a minimum let you know if you have a live wire (data packets being transported) as well as any interference or noise on it. I also generally do this when starting up new network segments to get a baseline on its physical characteristics. With Modbus I have had issues with getting devices to communicate, this initial test will at least let you know if there are data packets being transported on the wires. If there are, its more than likely a configuration issue.

    I used to use a cheap Oscilloscope from another vendor which did the above but went to the PicoScope 2205A for its added software capabilities of math functions and serial decoding. These allow you to actually view the data packets bits and will let you know if you are getting clean data or corrupt packets. The easiest way to determine this is to look for the BACnet preamble of the packet if your oscilloscope has serial decoding capabilities.

    Attached are some screen captures and a short video from both a live site and my test bench. The live sight has 24 controllers and the network is approximately a 1000’ feet in length, probably a couple of splices (one definitely that I know of). It was installed over a decade ago and was having some serious issues (latency) until all the previous BACnet controllers were removed and the wiring cleaned up a couple of years ago. It’s a crazy network (will do a drawing and upload in the future) that goes from IP to MSTP then in the field back to IP before going to MSTP sub-networks (IP to 38.4Kbs to IP to 76.8Kbs). The screenshots are of the primary MSTP network only. The second set of screenshots are on the test bench with just 3 controllers at 76.8Kbs and is about 5' in length. The names of the screenshots describe the how many EOLs are active or not. As you can see with the networks with EOL issues, all the data packets are still being transported with valid bits, the final byte is delayed on the networks with EOL issues. Also something to take note of is the condensed packet length on the 78.6Kbs data compared to the 38.4Kbs data. I run all my networks at 76.Kbs whenever possible.

    Looks like I can't upload mp4 files (or zip them even though the upload manager indicates that zip is a valid upload type of file). Which sucks as some of my captures (IF Tools/Yabe/BAC-5051E) are only videos. If you want to see them send me an email and I'll shoot them over.

    You can also use a standard VOM to do some electrical troubleshooting of IEA485 (MSTP/Modbus). I don't do this but here is a good video from KMC on the basics:

    YouTube link to oscilloscope capture:
    https://www.youtube.com/watch?v=P5N9...ature=youtu.be



    kontrol out
    Attached Images Attached Images         
    Last edited by kontrolphreak; 05-18-2018 at 09:43 PM.
    "Good" - Jocko
    "Open is as open does." - Forrest Gump
    "Can't we all just get a Lon?" - Garry Jack
    "BACnet: integration or interrogation?" - The Janitor
    "Open protocols? You can't handle open protocols!" - Nathan R. Jessup
    “What’s that? Aaa… open protocols? Don’t talk about…. open protocols? Are you kidding me? Open protocols? I just hope we can hardwire an interface!” - Jim Mora Watch it here!

  4. Likes rjk_cmh, orion242 liked this post
  5. #3
    Join Date
    Dec 2006
    Location
    What? Who? Where?
    Posts
    2,220
    Post Likes
    Thread Starter
    Generally, the second step is using Wireshark to view and decipher the BACnet MSTP packets. To do this I use the B&B Electronics 485USBTB-2W USB to IEA485 devices with YABE’s MSTP capture application. This is installed when you install YABE and I like it due to the ability to suppress the capture of certain packets. It works with Wireshark but you have to go into Wireshark’s Capture Options and manually create the pipe (the path is displayed in the capture tool interface).

    I’m competent at deciphering the packets but no guru by any means, there are a couple of good videos on YouTube on the subject (see below). Once you have done a capture for 5 to 15 minutes you can start looking at the packets yourself or if you have a Visual BACnet account you can upload it and have them create a report of your network for you (stay tuned for a future post on Visual BACnet).

    As this and the previous post’s connections are passive there should be no concern of effecting the normal data flow on the wires.

    YABE BACnet MSTP Capture Video on YouTube:

    https://www.youtube.com/watch?v=XQLt...ature=youtu.be



    It will only allow one video to be inserted so the other two are links to the videos on YouTube.

    https://www.youtube.com/watch?v=8asoKlP7iFA
    https://www.youtube.com/watch?v=GZkSpAqgvck

    I had wanted to attach the capture form the site but once again it is not an allowed type of attachable file.

    Optigo and Greg Holloway have been kind enough to allow me to upload the pcapng files and his colorization rules (a must have if you're using Wireshark with BACnet) from the above video. You will need to change the extension from .pdf to .zip once downloaded. The Wireshark Display Filters Word document is something I had on my laptop, unfortunately I don't remember where I got it so I can't give credit where credit is due.

    kontrol out
    Attached Images Attached Images
    Attached Files Attached Files
    Last edited by kontrolphreak; 05-21-2018 at 01:45 PM.

  6. Likes orion242 liked this post
  7. #4
    Join Date
    Dec 2006
    Location
    What? Who? Where?
    Posts
    2,220
    Post Likes
    Thread Starter
    Another tool to have in your toolbag of tricks is the IF Tools msb-rs485 Serial Interface Analyzer. I feel it is a jack of all trades but a but haven't used it as much as an oscilloscope and Wirehark. It is costly but for what it does it is probably cheaper than buy a bunch of different devices.

    It has the combined features of a basic oscilloscope and a protocol anaylser. I am a firm believer of the two is one and one is none rule, so this covers both the oscilloscope and USB/serial capture abilities of Wireshark for the two devices above. I have uploaded a basic video of all the features to YouTube (first video upload WooHoo! another first brought on by HVAC-Talk). And will upload a few more in the near future. Below I'll list some of the features of the MSB.

    The first feature is a Protocol Scanner which will allow you to scan the IEA485 and determine your Buadrate and Protocol (Data Bits/Parity/Stop Bits). Once these have been determined you can use the configuration in you current session, or if you already know them you can manual configure the session.

    The Data Monitor will give you the raw hex (or can be configured for Dec and ASCII) values of the data packet which if you understand how a BACnet packet is constructed can tell you things like the type of packet (token/poll for master/etc.) as well as the destination address and sending address. Once again with BACnet MSTP packets I look for the preamble (55 and FF) to start decoding the packets.

    Not really sure what the Event Monitor does, but the help files is extremely detailed and thorough (those Germans are engineering masters, they'll write a 50 page document on how to tie your shoelaces).

    The Protocol Monitor displays the raw data in a human friendly format - Preamble/Frame Type/Destination Address/Sending Address/Length/Header CRC/Data. It has a filter but once again haven't really messed with it.

    The final one is the Signal Monitor which shows you the electrical characteristics of the signal like an oscilloscope. It only shows you the packets and will not show any noise, interference or floating voltages like a true oscilloscope.

    Multiple windows can be open at the same time and synced to show the data currently.

    The session can be saved and reviewed at a later time.

    Most of the data can be exported as a .csv file as well.



    https://www.youtube.com/watch?v=fOfu...ature=youtu.be

    https://www.youtube.com/watch?v=9d3J...ature=youtu.be

    kontrol out
    Attached Images Attached Images     
    Attached Images Attached Images
    Last edited by kontrolphreak; 05-19-2018 at 09:19 AM.
    "Good" - Jocko
    "Open is as open does." - Forrest Gump
    "Can't we all just get a Lon?" - Garry Jack
    "BACnet: integration or interrogation?" - The Janitor
    "Open protocols? You can't handle open protocols!" - Nathan R. Jessup
    “What’s that? Aaa… open protocols? Don’t talk about…. open protocols? Are you kidding me? Open protocols? I just hope we can hardwire an interface!” - Jim Mora Watch it here!

  8. Likes orion242 liked this post
  9. #5
    Join Date
    Dec 2006
    Location
    What? Who? Where?
    Posts
    2,220
    Post Likes
    Thread Starter
    Another great tool is Optigo Network's Visual BACnet. It is a web based service that you can manually (or a scheduled automatic upload) of captured pcapng formatted files to be analyzed and presented in a format that is easier for humans to comprehend. I have used it on and off and have found that it quickly enables you to drill down and diagnose problems on your network. The BACnet Browser pages for network source and destination will really quickly show your "noisy" devices to take a look at.

    Device 1 in the screenshots and detailed report is the primary BACnet/IP to MSTP router of the convoluted network shown in the post below. As all traffic to the front end passes through it you can see that it is the most active controller on the network by far. It also routes all the data packets from the sub-networks hanging off the controllers on it's buss (not the optimal network design but it was how the previous contractor had installed the original system and the client did not want to install wiring to convert the units with sub-nets to BACnet/IP).

    The screenshots and attached report have shown me a couple of issues on the network I was scanning that I will have to resolve in the near future. There are too many views/pages to go into here from their website, but you can generate reports that are downloadable as .pdf files which I have attached to this post as well. The are both of the same capture, one being a basic one and the other more in depth.

    kontrol out
    Attached Images Attached Images        
    Attached Images Attached Images
    Last edited by kontrolphreak; 05-21-2018 at 01:16 PM.
    "Good" - Jocko
    "Open is as open does." - Forrest Gump
    "Can't we all just get a Lon?" - Garry Jack
    "BACnet: integration or interrogation?" - The Janitor
    "Open protocols? You can't handle open protocols!" - Nathan R. Jessup
    “What’s that? Aaa… open protocols? Don’t talk about…. open protocols? Are you kidding me? Open protocols? I just hope we can hardwire an interface!” - Jim Mora Watch it here!

  10. #6
    Join Date
    Dec 2006
    Location
    What? Who? Where?
    Posts
    2,220
    Post Likes
    Thread Starter
    Placeholder 5

    kontrol out
    "Good" - Jocko
    "Open is as open does." - Forrest Gump
    "Can't we all just get a Lon?" - Garry Jack
    "BACnet: integration or interrogation?" - The Janitor
    "Open protocols? You can't handle open protocols!" - Nathan R. Jessup
    “What’s that? Aaa… open protocols? Don’t talk about…. open protocols? Are you kidding me? Open protocols? I just hope we can hardwire an interface!” - Jim Mora Watch it here!

  11. #7
    Join Date
    Dec 2006
    Location
    What? Who? Where?
    Posts
    2,220
    Post Likes
    Thread Starter
    Placeholder 6

    kontrol out
    "Good" - Jocko
    "Open is as open does." - Forrest Gump
    "Can't we all just get a Lon?" - Garry Jack
    "BACnet: integration or interrogation?" - The Janitor
    "Open protocols? You can't handle open protocols!" - Nathan R. Jessup
    “What’s that? Aaa… open protocols? Don’t talk about…. open protocols? Are you kidding me? Open protocols? I just hope we can hardwire an interface!” - Jim Mora Watch it here!

  12. #8
    Join Date
    Dec 2014
    Location
    Vancouver, BC
    Posts
    29
    Post Likes
    Thanks! This is an awesome idea, MS/TP issues are a major time sink for me.

  13. #9
    Join Date
    Mar 2018
    Posts
    32
    Post Likes
    I just got a Pico and have used it a couple of times. I've seen some really weird things but I have no idea what it exactly means, what to look for, what could be causing it, etc.

  14. #10
    Join Date
    Dec 2006
    Location
    What? Who? Where?
    Posts
    2,220
    Post Likes
    Thread Starter
    Quote Originally Posted by BillyJoeJimBob View Post
    I just got a Pico and have used it a couple of times. I've seen some really weird things but I have no idea what it exactly means, what to look for, what could be causing it, etc.
    Most vendors will have MSTP wiring guidelines that may contain images of oscilloscope screenshots and what they are depicting. I know Reliable Controls and ALC both do. My suggestion is set up a testbench and create some of the common misconfigurations in both the hardware (wiring/EOL/Elect noise) and firmware (MAC address/DEV#/Buad Rate).

    Also getting you oscilloscope parameters set up for you application is key. After using an oscilloscope for the past couple of years I'm just beginning to use it confidently enough to start this thread, but I'm still only a level 1 scope wizard.

    Basic rule of thumb is if it doesn't look right it probably isn't.

    Will get the third post up this afternoon/night.

    kontrol out
    "Good" - Jocko
    "Open is as open does." - Forrest Gump
    "Can't we all just get a Lon?" - Garry Jack
    "BACnet: integration or interrogation?" - The Janitor
    "Open protocols? You can't handle open protocols!" - Nathan R. Jessup
    “What’s that? Aaa… open protocols? Don’t talk about…. open protocols? Are you kidding me? Open protocols? I just hope we can hardwire an interface!” - Jim Mora Watch it here!

  15. #11
    Join Date
    Oct 2014
    Location
    Columbus, Ohio
    Posts
    1,113
    Post Likes
    You should be able to put the videos on Vimeo or YouTube and link them here, if you want.

  16. #12
    Join Date
    Sep 2002
    Location
    Hampton Roads, Virginia
    Posts
    1,895
    Post Likes
    Thanks for putting this together, looks great so far. May it should become a "Sticky" post so that it is always easy to find.

    Kevin
    "There is plenty of room at the top because very few people care to travel beyond the average route. And so most of us seem satisfied to remain within the confines of mediocrity." -- Benjamin Nnamdi Azikiwe, first president of Nigeria

    "It's not the customer's job to know what they want." -- Steve Jobs

  17. #13
    Join Date
    Dec 2006
    Location
    What? Who? Where?
    Posts
    2,220
    Post Likes
    Thread Starter
    So, I thought we would make this a little more interesting. I have attached a network riser of the project I was doing the scans of for this thread and want to throw a challenge out. If professional member can produce a more convoluted single site out there with a Wireshark capture to show stable communications I will give the lucky winner a Contemporary Controls BASRTP-B for their laptop bag.

    https://ccontrols.com/basautomation/basportable.php

    I will be updating the posts above with the Visual BACnet upload of the primary BACnet/MSTP@38.4Kbs network, which though isn't perfect for what it is it's working well. I currently have a bad MSTP controller that needs to be replaced on the primary network as well as a couple of "ghost controllers" that have been removed from the system but still have controllers sending"Who-Is" to them that I need to clean up. Visual BACnet's report has help greatly in tracking these down.

    The second pdf attached is actually the pcapng file. To view download it and change the .pdf back to .pcapng and open with Wireshark.

    kontrol out
    Attached Images Attached Images  
    Attached Images Attached Images
    Last edited by kontrolphreak; 05-18-2018 at 09:54 PM.
    "Good" - Jocko
    "Open is as open does." - Forrest Gump
    "Can't we all just get a Lon?" - Garry Jack
    "BACnet: integration or interrogation?" - The Janitor
    "Open protocols? You can't handle open protocols!" - Nathan R. Jessup
    “What’s that? Aaa… open protocols? Don’t talk about…. open protocols? Are you kidding me? Open protocols? I just hope we can hardwire an interface!” - Jim Mora Watch it here!

Page 1 of 4 1234 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Related Forums

Plumbing Talks | Contractor MagazineThe place where Electrical professionals meet.