The Ultimate Guide to Systems Integration

 

Lesson #4: How to Use a Gap Analysis to Detail out the New Systems

 

What is a gap analysis

A gap analysis is a key part of the integration process but what is it?

In a gap analysis you take the functional specifications you created in your use case and you compare your existing system(s) to the functional specifications. In laments terms you look at the capabilities required for the use case and the capabilities you currently have.

The missing capabilities become the gap.

 

Why is it critical to perform a gap analysis?

Think about it, imagine you go and decide to buy some really cool video game… Ok, I am admitting to one of my really bad hobbies here.

But, you go and buy this game and you don’t check if your computer can run it. You come home and all you have is a $60 dollar brick because the game won’t run on your computer.

Now, for most $60 dollars isn’t the end of the world. Imagine though, if you had went and purchased a $300,000 analytics package.

But…

You didn’t go and check if your BAS could provide the required information for the analytics software to run. Now, here you are with this $300,000 software package and you don’t have the points in your system that the analytics software needs! So, now you have two choices, cancel the contract and absorb any termination fees OR go and spend the time, money, and effort to add all the necessary points to your BAS.

Chose your poison.

 

How do you perform a gap analysis?

Ok Phil, you’ve sold me on the need to perform a gap analysis but how do you do one?

Quite simply you take the requirements of your use case and you put them next to your current capabilities.

Let’s see this in action, by continuing our scenario, laid out at the beginning of this series, where we are integrating two BAS into a single graphical user interface (GUI).

 

Step 1: Identify your use case requirements

In the beginning of this series we created a use case for our single GUI integration. While we laid out several use cases, we did not identify the technical requirements of our single GUI. Let’s do that now.

Our ultimate goal is to have a single GUI that the two existing BAS can tie into to provide full access and control of real-time systems and historical data. This GUI shall use existing logins and not require new logins.

Let’s break this into bite size chunks.

 

Real-Time Systems

We need a way to capture and control the existing data points from the two BAS. The new GUI supports BACnet/IP

 

Historical Data

We need a single database or a way to access the existing two databases and pull them into the new GUI. The new GUI supports SQL

 

Existing Logins

We need to allow the users to continue to use their current logins and passwords. If the user changes their login or the user is removed then the GUI needs to reflect this. The new GUI stores its user name and passwords in a SQL database but does not support external user names and passwords.

 

Step 2: Identify your current capabilities

Currently our two BAS systems support BACnet/IP, have SQL databases, and they have SQL databases for user accounts.

The two BAS’s however, do not have a common data model.

 

Step 3: Map out the present and missing requirements

Because of these requirements we know that we have:

• Database connectivity (SQL)
• Protocol connectivity (BACnet/IP)
• User Interfaces in SQL in the two BAS and new GUI

The big gap here is, how do we get and manage the new user logins into the new GUI? 

 

Step 4: Identify a solution to close the gap

Ok, so we have a gap. We need to find a way to transfer the user name and logins to the new GUI.

Simple right?

Problem is, we need to maintain the state of the logins.

What does maintain the state mean?

In any database you have what is called the state, the state is the current condition, or snapshot if you will of the database. In order to achieve our scope of using the user accounts and passwords that exist in the BAS we need to make sure that the user database for the GUI reflects, is consistent with the accounts that exist in the new BAS.

Minor problem, what if two users accounts with the same name exist? 

This is quite easy to solve, we can simply import both user names and using some SQL magic we can make them unique. When someone tries to login the GUI will check both passwords to see if that is correct. Naturally this assumes that Bob on system A is the same Bob on system B.

Because of this, it’s often a good practice to pull all user names look for identical names and request the user change their name. This is done in the beginning of the integration process.

So far we have identified 2 issues:

• Maintaining the database state
• Dealing with identical user names

Now we need to document our resolution for these two issues. Let’s say that in order to maintain the database state it will take 8 hours a week of a Database Administrator at $60/hr.

The facility manager doesn’t want to carry this cost.

What can he/she do?

This is where alternate solutions come in play. The facility manager will ask the integrator or rather the integrator will present alternate options to the facility manager.

The facility manager can now go and chose between accepting the cost of managing database state OR going and looking at a second option, like making folks create new user names and passwords on the single GUI…

 

 

Lesson #5: Detailing out the physical and logical integration points

Don’t misunderstand. Without a use case and gap analysis you can get your butt into a whole heaping mess of trouble.

Oh, those were supposed to be BACnet/IP cards on the rooftop…

I’ve lived in that world, I’ve got the battle scars.

But, even if you do the perfect use case and identify every single gap in your design, you still haven’t created anything until the wires are connected and the code is written..

So, what is our next step? How can you go from concept to working systems integration? 

In order to do this you need to detail out your physical and logical integration points.

But what are physical and logical integration points? And what exactly does detailing them out mean?

Well, my friend, I’m going to answer those exact questions below.

 

What are Physical Integration Points

When folks think of systems integration, their minds usually don’t go straight to the physical integration points and that’s a shame! I would argue that without physical integration, nothing happens.

You can write the best code in the world. You can take two completely disparate data models and make them one. But none of that “stuff” matters if the two systems can’t talk to one another.

And that is why I start with physical integration points.

What are physical integration points you ask?

Physical integration points are the physical connections between the systems you identified earlier in your gap analysis and use case. These physical connections can be direct cabling, networking, wireless connections, the list goes on and on.

The important point is that physical connections, physically connect the systems…

 

What are Logical Integration Points

Logical integration points on the other hand, don’t physically exist. You can’t go to Home Depot and pick up 20ft of logical wiring, well if you can let me know and we can make a killing on Amazon…

Logical integration points are the connections between the systems that utilize the physical connections.

For example, you may physically connect a lighting system to a building automation system. But, without a logical connection to tell the two systems how to communicate and to resolve data and all sorts of other “programming” problems, your integration may be a dud.

 

How do you “detail” them out…

So then, how do you make all this integration “stuff” work and what the heck is detailing something out…

Well, here’s the deal, every systems integration job is unique, can we all agree on that?

Ok, I’ll take that virtual head nod as agreement 😀

So, if every job is unique it makes sense that we would need to plan out each project and “detail” out the plans for each step.

Well, in this section I am going to teach you my process for “detailing” out physical and logical integrations. You might want to go grab a pen and pencil for this, don’t worry I’ll wait for you…

 

The 3 Step Process for Detailing Out Physical and Logical Integrations

 

Step 1: Identify Your Physical Connections

Ok, well that was obvious…

But was it?

Using our use case from the previous posts, we can map out our connections.

If you don’t remember our original use case, or you haven’t read my other posts (naughty, naughty…) then here it is.

We want to integrate two building automation systems into a single graphical user interface (GUI) on a new system.

Which systems connect to what? 

Do you connect the two building automation systems directly to the new GUI?

You need to look at your use case and follow how the systems are supposed to interact. Then, you need to draw your physical connections on a piece of paper. Specifically you should cover:

• Which systems are there?
• What system connects to another system?
• What physical connection is used?

 

Step 2: Identify your communication flow

Before we get to the logical connections and interfaces we need to identify how the data flows.

Using the use case from step 1 maybe we are routing some information from the first building automation system into the second building automation system.

Why would we do that?

Maybe the first building automation system controls the air handlers which then send data to the second building automation system that controls the central plant. The important part of this step is that we show that connection.

Could you imagine the chaos if we broke that connection and just simply directly connected the building automation systems to the single GUI?

You will want to know:

• Which way should data flow?
• Is a process in a system dependent on another system?

 

Step 3: Logically map your logical connections

Well shucks, I could only fit the word logical into that heading twice… Oh, well.

It’s important that we figure out how our data is going to be connected to the other system(s). In the case of our single GUI, one of our design constraints is to have a new single database.

Well, what if the data is completely different in both BAS? 

We might have to go and create an intermediary database to normalize that data.

You know what that means right?

We’ve got to map out how and what data flows. In order to that we need to identify what data we are looking to communicate and diagram out where that data goes to.

Questions you want to be asking yourself here are:

What priority is the data at?

Are there any Quality-of-Service (QoS) issues?

Where does the data come from and where does it go to?

 

Lesson #6: How to create an Integration map

 

What is An integration map?

So what exactly is an integration map?

In lesson number five I discussed the concept of physical and logical integration connections, man is that ever a mouthful.

I described that in any systems integration project you will have either physical, logical or both connections.

I also discussed how you need to go and map out the connections between your integrations. However, I did not show you the exact process for doing this.

That is what I’m covering here.

An integration map, is a method, for mapping out the connections, dataflow, and process flow. Usually this is done by use case.

What do I mean this is done by use case?

In lesson two I discussed, how you can create a use case for your integrations. I also showed you how to list out the systems for this use case.

Since each use case is specific systems you’re going to want to create an integration map for each use case. In any systems integration project you want each integration to be as standalone as possible.

This will help you and troubleshooting as well as supporting and upgrading the use case because it will eliminate any interdependencies between use case

Now if you send yourself what is on interdependency?

An interdependency is where one use case depends on logical and/or physical systems from another use case.

Now it’s not possible to eliminate all interdependencies because some use cases require a previous use case before they can take place.

For example, if you were to go and say that you wanted to have use case on creating a schedule. That scheduling use case may be dependent on a login logout use case.

Ok, so back to the question “what is an integration map?”.

As I mentioned an integration map is a method or process for mapping out your connections, data flow, and use case flow. I also mentioned that an integration map usually created for each individual use case.

 

What is an integration map used for?

Okay, now that you understand what integration map is.

What exactly is it used for?

I mean, saying it’s a process for mapping out your connections, data flow, and use case flow, isn’t exactly the clearest way of defining what an integration map is used.

When you go to execute an integration project, you’re going to have to perform some specific tasks. These tasks commonly are:

• Connecting the systems
• Normalizing the data
• Prioritizing the data flows
• Creating any sequence or trigger logic (basically if this happens then that happens)

 

Now if you only had two systems to integrate these tasks would most likely be quite simple.

 

However, with the data center example I provided, I had to perform the following tasks:

• Data normalization on each individual system
• Quality of service (basically making sure the messages get prioritized over other network traffic)
• Physically connecting the systems on the network
• Creating an integration gateway to make all the systems communicate via OPC
• Triggering use case is based on specific points, alarms and commands

 

Could you imagine trying to do this on the fly?

I had over a dozen different use cases, and several dozen data centers, could you imagine the chaos of trying to map out all of those systems in my head?

I’m hope that you see, why I would want to create an integration map.

So quite simply, an integration map is a way of structuring your physical connections, logical connections, and data flow in a way that allows you to execute and document your projects step-by-step.

 

When and when not to use an integration map?

At this point you may be wondering when you should be using an integration map?

I’m not going to tell you that you should use an integration map on every project because it’s simply not true.

If you are connecting two systems, then as long as you identify the physical and logical connections and any data normalization that has to take place you should be fine.

So the general rule of thumb, is whenever you have to connect more than two systems or your project involves multiple use cases, with interdependencies, you should look at creating an integration map.

 

How do you create an integration map?

Okay, we’ve come to the point of the post that most of you been waiting for!

So you been following along, and you now agree that you should create an integration map, but how? If you go and scour the Internet, you can find resources on systems integration but it’s focused on IT systems.

How do you create an integration map for building automation systems?

I’ve come up with a step-by-step process for mapping out your integrations by use case...

 

6 Steps for Creating an Effective Integration Map

1. Identify the use case that mapping out the integrations for (I described this in lesson 2)
2. Identify your physical connections, logical connections, systems, and data flow (by the way I went through how to do this in lessons 3, 4 and 5)
3. Create an integration drawing to list out your systems
4. Draw the physical and logic connections between the systems and number those connections
5. Draw the direction, priority, and type of data that flows across the physical and logical connections, number your data flows.
6. Take your numbered connections and data flows and list them in a spreadsheet

 

Step 1: Identify the use case that you are mapping out the integrations for (I described this in lesson 2)

As I mentioned in lesson 2, during the use case creation process you identify the systems that will be used. It is important now that you identify the use case that you will be mapping out integrations for.

Otherwise you may not identify the proper systems and connections that are required.

 

Step 2: Identify your physical connections, logical connections, and data flow between your systems (by the way I went through how to do this in lessons 3, 4 and 5)

Speaking of systems and connections, it is important now that you identify the physical connections, logical connections and data flow between your systems.

You will use this information to create your integration.

You will need to identify:

• The physical connection type
  • What systems the physical connection is between
• The logical connection type
  • What systems the logical connection is between
  • The protocol being used for the logical connection
• The direction of the data flow
• The priority of the data flow
• The size of the data flow

Once you have this information you’ll be able to perform the next step.

 

Step 3: Create an integration map to list out your systems

Now, I’m not gonna walk you step-by-step how to create a system drawing. However, for those of you that are interested I will be creating a systems integration course in the future that will go through this process.

That being said, the important thing is to pick the format and software that you’re most comfortable with using.

I personally use Microsoft Visio and would recommend that to others.

Once you’ve picked the software that you’re going to create your system drawing in, you will need to place each individual system on the integration map.

 

Step 4: Draw the physical and logic connections between the systems and number those connections

Next, using the information you collected in step two, you will want to add the physical and logical connections to your system integration map.

I personally recommend that, physical connections be solid lines and logical connections be dashed lines ( - - -).

In most cases I recommend using a color code for the connections to show what protocol, physical medium (Ethernet, RS-485, RS-232, etc.) is being used.

You would typically list this in the upper right-hand corner as a legend.

 

Step 5: Draw the direction, priority, and type of data that flows across the physical and logical connections, number your data flows

Now that you have the physical and logical connections added to the integration map, I recommend that you go and add the direction in type of your data flows.

For data flows I recommend using dots ( ● ● ●), instead of dashes or solid lines.

You will want to list out the direction that your data flows in, the priority level of the data (four commands and quality of service), the size of the data being sent, and the type of data that is being sent.

You will also want to number your data flows.

I will be talking more about data flow, data models, and system requirements in lesson 7.

 

Step 6: Take your numbered connections and data flows and list them in a spreadsheet

Now that you have your connections in data flows numbered, you will want to list them out in a spreadsheet. In this spreadsheet you will want to show, what physical connections, logical connections, and data flows connect between what systems.

You will also want to list out the following information...

 

For your physical connections you will want to list out:

• What systems are connected by the physical connection
• The media type of the physical connection

 

With the logical connections you will want to list out:

• What systems are connected by the logical connection
• The protocol type of the logical connection
• Any software and/or services that the logical connections originate from or and at (often times these are called endpoints)

 

Finally, with your data flows will want to list out:

• The system(s) that the data flows from
• The system(s) that the data flows to
• The priority level at which the data flows
• The size of the data flow (typically defined in bytes per second)

 

With all of this information defined you’re now ready to move on to laying out your data model and system requirements. Some of you may be asking why I waited to have the data model after the integration.

I have found that if you don’t map out should your data flows and logical connections prior to creating the data model you can often miss or add data that you do not need.

 

Lesson #7: Detailing out the data model and system requirements

 

A brief segue on data

Data is everywhere. Some have said, “Data is the new oil”. Having the right “data” can provide you insight into how your building operates. With this information, you are able to make operational decisions that will impact the efficiency and effectiveness of your building automation system.

 

What is a data model and how do you create one?

A data model organizes data and details out how that data is formatted. When you are exchanging data between systems it is important to understand the format that the data is in. This is represented visually below:

As you can see I created a simple data model for a VAV box. This data model describes the primary key (which is how we will find the data) and two properties, zone temperature (ZN-T) and damper output (DPR-O).

You will notice that the primary key, controller name, is defined as the data type string and the two other properties are defined as the data type float.

In the programming world, a string, is a string of text and a float, is typically a number, with decimal precision.

Now, if I had, another system, that I was going to be integrating the VAV box data into and that system had a different data type then there would be a conflict. This is illustrated below:

As you can see, the VAV box data is not of the same data type as the third-party BAS data. This will cause a data conflict.

What is happening is the third-party BAS, has a primary key of device ID, which is of type int. An INT, stands for an integer. Unlike a float, an integer, does not have decimal precision. This means that the zone temperature and damper positions would lose their “precision” or accuracy.

But the bigger issue, is that the primary keys conflict both in type and in name. This will cause a mismatch. In order to resolve this the data needs to be normalized.  You can see how data is normalized in the image below:

What I just did in the above image, is I created a higher level data set that bridges the two data sets utilizing what is called foreign keys. This allows the third-party BAS and VAV box data sets to access the data from the normalized data set.

When you purchase a device like a field server gateway, this is what is happening. Inside that gateway there is software that is performing this data normalization and then allowing the two data sets to connect.

When you look at an individual system you will want to go and create the data sets for each system. Now you may not know the data types that each system has. This is where you will have to turn to the system vendor to provide you that information.

Now I take a quick second the make sure I’m clear. If you’re performing a protocol integration using similar protocols, you most likely will not have to go this deep. The examples I am providing are geared more towards dissimilar protocols, for example integrating a LON system into a BACnet/IP system.

You can ask the vendor to provide you the data model for their specific system. Then you need to understand the points that you want to share between the systems. Once you have defined the points you will need to map up the data types.

Once you have done this you will need to determine if a normalization schema is needed. For some of you the word schema may be unfamiliar. When you hear the word schema, think format.

Schema is another way of saying the format of the data.

 

What are system requirements and how do you define them?

System requirements list out the physical and logical requirements of the systems that are part of the integration project. You will typically want to understand:

• CPU
• Memory
• Storage
• Bandwidth
• Network Throughput

 

In an earlier post we discussed how to identify your systems. Now that you know what your systems are its time to formally document the system requirements. This step is not very complex but it is time-consuming.

If you do not have an IT person on staff than is important that you retain the services of someone with an information technology background.

 

The 4 Step Process for Defining Your System Requirements

 

Step 1: Define the network, storage, and compute requirements for each system

Using the list of systems that you created in (earlier post), you will need to go and work with your IT staff and vendors to determine the network (also known as bandwidth), storage, and compute (also known as processing) requirements of each individual system.

 

Step 2: Identify where the systems will be hosted

Based on the information from step one, you will want to work with your IT representative to determine where the systems will be hosted and what part of the network they will be connected to.

 

Step 3: Create an overall architecture diagram

Now that you have listed out where the systems will be hosted and the requirements you will need to create an overall architecture diagram. This architecture diagram will visually represent where the systems reside on the network and how they are connected to one another.

 

Step 4: List out any system, software, requirements

With all of this information defined, your final step is to work with your IT representative and your vendor to determine any software that is required to be installed on the systems in order for them to run.

 

Lesson #8: Developing a sequence of operations

Developing a sequence of operations will help you or your integrator to effectively execute a systems integration project. Without this document it is very likely that your project won't work. I know, I know.

It seems like I've said that about every post in this series. Well folks, I only say that, because it is true!

Look, I'm walking you through the mine field but it's up to you if you want to stay by my side or run ahead and play with your projects life.

 

What a sequence of operations is

So what is a sequence of operations. Chances are, if you are working on a systems integration project you are familiar with a sequence of operations. But, I'm going to assume for a second that you aren't familiar with a sequence of operations.

A sequence of operations is like a use case on steroids. Whereas a use case walks you through the how the sequence of operations is the how, the what, the when and even more.

For example, I'm going to describe a use case where the main success scenario is the user being able to use credentials from another BAS login into a system, hint, hint this may look familiar to a use case I described in an earlier post.

 

Use Case Text

The user will be able to use login credentials from an existing BAS to log into the new graphical user interface.

 

Sequence of Operations Text

A user will enter their credentials from the existing BAS to log into the new graphical user interface. Upon entering their credentials the graphical user interface will perform an MD5 hash algorithm against the users password. The graphical user interface will then communicate to the back-end database and will perform a select query where the username and hashed password match. If a matching data set can be located then the user will be granted access to the graphical user interface.

Do you see the difference? The sequence of operations is the end result of information from the physical and logical integration map, the data model and the use case.

 

How to create a sequence of operations

Creating a sequence of operations is actually quite easy once you get the hang of it. I used to suck at writing sequence of operations and then one day it all just clicked for me.

This happened because I came up with a step-by-step process that empowered me to take what was in my head and put it on paper.

 

The 7 Step Process for Creating a Sequence of Operations

I am going to walk you through the steps using the example of tying in a lighting system.

 

Step 1: Gather the main actions from the use case

For this step what you want to do, is to gather the main actions from the use case that you created. You will then take these main actions and expand upon them in order to create a sequence of operations.

A use case may say, the lighting system will be integrated to the building automation system. The building automation system will turn the lights on when the building is occupied.

The main actions for this use case are:

• The lighting system turning on
• The building automation system going occupied
• The building automation system commanding the lighting system

 

Step 2: Identify the systems from your integration map

Next you want to go and take the systems that you identified as part of your integration map. These systems will then expand upon the sequence of operations you started in step one.

From the integration map I will be able to identify that the building automation system and the lighting system are connected via BACnet/IP. I will also know that the building automation system will be commanding the lighting system have priority level 10.

I will also be able to identify, that the building automation system syncs its timeclock with the building's local time-server.

 

Step 3: Clarify any settings and/or timings

Here I will decide that I want the lighting system to turn on within 30 seconds of the building automation system switching to occupied mode.

 

Step 4: Chunk out your actions and build out a flow map

I will then take each one of these actions and build a flow map. This flow map will visually represent how and in what order the different actions will take place.

Step 5: Capture the wording for the actions

Once I'm sure that the actions I want the integration to execute are in the right order. I will break them out and begin to add details around the wording.

At first this will look like:

• The buildings time-server sends a time synchronization message to the building automation system
• Building automation system is occupied
• Lighting system is commanded to occupied
• Lights are commanded on by the lighting system

 

Step 6: Add your settings and timings

Now I will take those four bullets that I created in step five and I will add details around each bullet.

• The buildings time-server will send a time synchronization message to the building automation system every 15 minutes (adj). The time synchronization message will be sent using the NTP protocol to the building automation server located in the data closet.
• The building automation system will have a set schedule that will command the building to occupied. This schedule will be from 7AM CST to 5pm CST (adj).
• The building automation system will command the lighting enable object to on (adj) within 30 seconds (adj) of the building automation system occupancy signal switching to the occupied state. This command will be sent via the BACnet/IP protocol from the BAS server to the lighting server at priority level 10.
• Upon receiving the on command from the BAS, the lighting system will command all of the buildings interior lights to on.

 

Step 7: Consolidate into a sequence

Now for the easy part. All I need to do is simply combine the bullets together and I will have my sequence of operations.

The buildings time-server will send a time synchronization message to the building automation system every 15 minutes (adj). The time synchronization message will be sent using the NTP protocol to the building automation server located in the data closet.The building automation system will have a set schedule that will command the building to occupied. This schedule will be from 7AM CST to 5pm CST (adj). The building automation system will command the lighting enable object to on (adj) within 30 seconds (adj) of the building automation system occupancy signal switching to the occupied state. This command will be sent via the BACnet/IP protocol from the BAS server to the lighting server at priority level 10. Upon receiving the on command from the BAS, the lighting system will command all of the buildings interior lights to on.

 

How to use a sequence of operations to execute your systems integration project

Once you have the sequence written what do you do? How do you take your sequence and execute a project.

Well, everyone has a different way of executing their systems integration projects. If you have a way that you like and you have followed the steps up to this point then I encourage you to use your method.

However, if you want a proven process that has helped me to execute some crazy complex projects, then check out my 5 step process below:

 

The Proven 5 Step Process for Executing Your Systems Integration Project

 

Step 1: Build out the physical architecture

In this first step you will want to install the systems that you identified as part of the to be architecture.

Once you have the systems installed you will then want to make sure that any software associated with the systems set up including, drivers, databases, and dependencies.

Using our lighting example, you would want to install the lighting and building automation servers, controllers, and end devices.

 

Step 2: Build out the logical architecture

Next you will want to go and set up logical architecture for the integration project. As part of this process you will want to configure any network, database, API, and other logical pieces of the system.

Continuing our lighting example, you would want to set up any IP addresses, device instances, VLANs, databases, and system names.

 

Step 3: Add any programming

Next you will want to go and develop any programming that is required in order to execute the systems integration sequence. This would include data adapters, interlocks, dependency logic, and any control logic.

For our lighting solution, you would want to build out, the  interlock logic along with the prioritization scheme for the integration points.

 

Step 4: Insert settings and timers

Once all of the code is written you will want to put in the initial settings and timers for the sequence. This would include any schedules, set points, on or off delays, etc.

You will want to set the timers and schedules for the lighting integration.

 

Step 5: Test the integration

Finally, you will want to test the integration to verify that it works.

In order to test the lighting integration you would command the building automation system from occupied to unoccupied and verify that the lights changed state.

You would also want to change the master time-server and verify that the building automation system adjusts accordingly.

Finally, you would want to make sure that the lighting system cannot override the building automation system.