bacnet internetwork - using private Vlans for segmentation

[martinsbt]

hi

Its a Huge campus with about 200 buildings. We have a mix of Bacnet Ip from different vendors in each buildings (average 2500 controllers so far jncluding the mstp.devices). We want tomfind a way to isolate bacnet ip traffic to each building and use bbmd's (one in each building, and one at the BAS)

IT will never allow to have a single VLan in each building, as it is a waste of ip adresses. So my question is, can we use the feature avail on Cisco switches called Private vlans to accomplish this ? Anyone tried it ?

Thanks in advance

[Frozenlock]

IT will never allow to have a single VLan in each building, as it is a waste of ip adresses.

Uh?
IT might be reticent to make you 200 VLANs because it requires a little work, but I highly doubt it's because they are so limited in their IP address space.

In addition, if you are building networks touching 200 buildings, you really should let the IT department know about it, or even handle it.

For your original question, if you are referencing something like this, I don't see why it wouldn't work.

[rjk_cmh]

Each vlan gets its own IP space unless you are connecting them with some routing trickery. Similarly, each vlan requires its own DHCP server because it is a completely separate network segment, if you use any dynamic addressing. Most networking equipment (especially Cisco) supports the full 4096 or however many maximum vlans per physical connection, but using them all might tax the memory of the routers and switches in the network - but it's probably just because IT doesn't want to make such a huge change to their configs.

[CraziFuzzy]

How many actual IP devices do you have per building? What is your reasoning for actually segregating the buildings? Also, if you have IT involved, you would be far better off working a way to NOT use BBMD's if you can help it. BBMD's were developed as a way to work around complex network topologies, but you are far better simply ensuring that the devices you want to talk to each other, can talk to each other directly (whether that's via a common VLAN address space, or VPN's connecting your equipment on a large 'private' network devoted to just the controls equipment).

I, personally, have started transitioning all our BBMD connected buildings to VPN connections. The biggest advatnage of this, is it makes us more immune to IT changes. As long as the VPN client (out in the remote building) can find the VPN server (in our central plant), then then the network stays fully connected. the individual clients can have their IP addresses changed, can be moved to other locations, etc, and it doesn't matter, because the VPN connection is initiated by the client TO the server. The server doesn't ever need to know where the clients are, or what their addresses are. I've actually been very successful using consumer grade routers for this (with 3rd party firmware), making the equipment costs extremely low.

I've even seen performance improvements (subjective observation, only). I'm guessing this is attributed to the VPN's compression.

[kontrolphreak]

I have used BBMD routing at a school district with more than 50 schools by the company I was associated with at the time, with little to known issues (other than one highscholl where another vendor also had a controller set up as a BBMD). The district has more than 200 schools with 60-70% being BACnet from multiple vendors (each on a separate network) all tied into a Niagara front end. With new BBMD routers having tables with a lot more entries the hardware side has been decreased as well. There are even over the counter BACnet routers with BBMD capability for a reasonable price.

kontrol out

[CraziFuzzy]

I was certainly not saying that BBMD doesn't work, I just find it is an unnecessary step if you can put everything on the same subnet.

[kontrolphreak]

Definitely having everything on the same subnets simplifies the process, but on larger sites/multiple sites it's not always feasible.

kontrol out

[MaxBurn]

I, personally, have started transitioning all our BBMD connected buildings to VPN connections. The biggest advatnage of this, is it makes us more immune to IT changes. As long as the VPN client (out in the remote building) can find the VPN server (in our central plant), then then the network stays fully connected. the individual clients can have their IP addresses changed, can be moved to other locations, etc, and it doesn't matter, because the VPN connection is initiated by the client TO the server. The server doesn't ever need to know where the clients are, or what their addresses are. I've actually been very successful using consumer grade routers for this (with 3rd party firmware), making the equipment costs extremely low.

DD-WRT or something similar? So you place a router on each remote sites and another with the front end to connect to?

[CraziFuzzy]

DD-WRT or something similar? So you place a router on each remote sites and another with the front end to connect to?

similar, using Tomato, but any firmware that supports openvpn should work. For my server end, I actually have openvpn running on the windows pc that's running the BMS server, though it could be done with a router on that end as well. I am not traversing the internet at all with my configuration, so I've made the security configuration pretty simple (using the same key for each client). Over the internet, I would definitely use a more robust security mechanism. OpenVPN certificates are not an incredibly simple process if you haven't dealt with them before, and sadly, many IT departments don't even 'get' them.

[kontrolphreak]

So does your communication between networks rely on the computer being online and Tomato running?

kontrol out

[orion242]

Raspberry pi can run openvpn also, been testing them out lately. Its not for the faint of heart to setup, but they have metal din rail enclosures for them. Thinking about the best way to manage all the certs yet. I can't say I like the fact a file is all that is needed to log into the VPN. Seems some malware on a techs laptop could be the start of a $hit storm.

[CraziFuzzy]

So does your communication between networks rely on the computer being online and Tomato running?

kontrol out

If you set it up as I described, then yes, that would be the case. In my situation, there's only a single point that is shared from one building to another for control, and it has always had a failsafe fallback implemented, so the only time I ever need to send data across buildings is when the data is getting from the field TO the server anyway, so if the server is down, so is the need for the data - but there are other alternatives. The same routers I'm using as my clients can also run as servers at the same time, and you can set up a list of servers to connect to in order, so i could set it up so that the primary VPN server is the main server, but on fallback, it would connect to the router in the building it needs its point from.

Raspberry pi can run openvpn also, been testing them out lately. Its not for the faint of heart to setup, but they have metal din rail enclosures for them. Thinking about the best way to manage all the certs yet. I can't say I like the fact a file is all that is needed to log into the VPN. Seems some malware on a techs laptop could be the start of a $hit storm.

Pi's only have a single network interface though, making it more difficult to use a gateway between your 'private' controls network and the corporate LAN. Also, by the time you've purchased the Pi, the enclosure, and a power supply, you're running real close to a router.

[orion242]

only have a single network interface though, making it more difficult to use a gateway between your 'private' controls network and the corporate LAN.

We don't normally have separate networks for the controls, so that's a non-issue in most cases. I like the fact its a small din rail mounted device. Mounting one of these home routers in the panel is not really a treat. I tried tomato and DDWRT at home years back and those linksys routers where about as reliable as a crack head. I'm sure it was amount of traffic and/or connections that sent them sideways which wouldn't be an issue for just BMS traffic. In the back of my head I still wonder how reliable they are. With the pi, once you have a working setup, its easy to image the SD card and clone it to another for an almost instant setup. I'm sure backing up the router settings is about as easy, but I think cert management may be easier one something I can run a file syncing program on. Someone leaves the company, pull their cert and it propagates to all the devices.

[supers_5]

I like the rPi, or other more powerful SoC devices. (same physical size) From experience, the old linksys WRT routers, now less then a McD lunch, were rock solid with ddwrt or tomato. The only trick was keeping them coolish and getting a non crippled revision. Usually a fan was all it needed in hot environments. I still have some running after nearly a decade.

[orion242]

The only trick was keeping them coolish

Yea like in a control panel in the boiler room, steam...not hot water. Both will crap out in this kind of location.

[CraziFuzzy]

We've had been using the consumer level routers for years in this facility as simple routers anyway, with the BBMD's behind them. Most of my effort was simply reconfiguring them, not installing anything.

We don't normally have separate networks for the controls, so that's a non-issue in most cases. I like the fact its a small din rail mounted device. Mounting one of these home routers in the panel is not really a treat. I tried tomato and DDWRT at home years back and those linksys routers where about as reliable as a crack head. I'm sure it was amount of traffic and/or connections that sent them sideways which wouldn't be an issue for just BMS traffic. In the back of my head I still wonder how reliable they are. With the pi, once you have a working setup, its easy to image the SD card and clone it to another for an almost instant setup. I'm sure backing up the router settings is about as easy, but I think cert management may be easier one something I can run a file syncing program on. Someone leaves the company, pull their cert and it propagates to all the devices.

I'm not sure I'm seeing how the Pi is fitting into your scheme here.

[supers_5]

Yea like in a control panel in the boiler room, steam...not hot water. Both will crap out in this kind of location.

That would be "slightly" over it's rating. I can see why you had issues. That's when you start looking at various forms of active cooling. However you got your stuff to keep ticking would be an interesting story.

The rPi is like a computer and can be configured in nearly infinite ways. Almost Whatever you need, you'll be able to make it do, or other hobby boards. Such as VPN. The only downside is that its not plug and play. You have to be an enthusiast to use one. They are also dirt cheap before you count your time.

[orion242]

I'm not sure I'm seeing how the Pi is fitting into your scheme here.

I open the single VPN port to the internet, tunnel in, auth, and I have bacnet, web, and any other service (and only those services) I need to the BMS via a secure tunnel. I hate having to deal with a wad of different ports with IT and bacnet (pretty much any BMS protocol) has no real security. I don't want any controller/software bare to the internet outside of the VPN.

So if the Pi can act as a VPN endpoint (w/ DDNS support) and get me access to the BMS services I need and have the flexibility I need, its a long ways to what I want. If tomato and ddwrt can do the same reliably, I will certainly would look at them. Once the that is covered, it becomes a scalability/management issue. I'm not looking for a one off solution, but something that is secure, easy to manage, and scalable to most sites. Is that a pi, soho router, industrial pc, cyberbull$hit box, not sure yet. What does seem clear is nothing on the market today meets my needs and I will have to likely roll up my sleeves to get it done on my time frame.

[CraziFuzzy]

What I'm not getting is how you are connecting your equipment over the VPN via the Pi, with it only having a single ethernet port/interface.

[orion242]

Well this is a pet project and I don't have the pi completely setup and tested. Getting openvpn running and certs setup on the pi2 was all the excitement I could stand in a single setting. Haven't got back to it yet and with summer coming it may be a bit. The single nic may be a gotcha, but from what I could find its not. I use openvpn at home on windows with a single hardware nic and it works just fine. Windows uses a virtual nic bridged to a physical.

In the pi world it looks like the key is enabling ipv4 packet routing in sysctl.conf and adjusting iptables.