Middleware sits "in the middle" between application software that may be working on different operating systems. It is similar to the middle layer of three-tier single system architecture, except that it is stretched across multiple systems or applications. Examples include EAI software, telecommunications software, transaction monitors, and messaging-and-queuing software.

The distinction between operating system and middleware functionality is, to some extent, arbitrary. While core kernel functionality can only be provided by the operating system itself, some functionality previously provided by separately sold middleware is now integrated in operating systems. A typical example is the TCP/IP stack for telecommunications, nowadays included in virtually every operating system.

In simulation technology, middleware is generally used in the context of the high level architecture (HLA) that applies to many distributed simulations. It is a layer of software that lies between the application code and the run-time infrastructure. Middleware generally consists of a library of functions, and enables a number of applications – simulations or federates in HLA terminology – to page these functions from the common library rather than re-create them for each application.

In the computer industry, middleware is a general term for any programming that serves to "glue together" or mediate between two separate and often already existing programs. A common application of middleware is to allow programs written for access to a particular database to access other databases.

Typically, middleware programs provide messaging services so that different applications can communicate. The systematic tying together of disparate applications, often through the use of middleware, is known as enterprise application integration (EAI).

Okay, I think that the previous section define the term middleware from academic view, maybe we can be a bit more specific about what middleware is by describing what it does. First, though, we need a basic model to show where middleware fits among the other pieces of the client/server puzzle.

We'll divide our discussion into three areas:

• Basic client/server component model
  
• Basic middleware services
  
• Types of middleware
  

Basic Client/Server Component Model

We've created a simple network application model to help explain the concept of middleware, and we'll use it throughout this guide. Instead of the OSI network model, we have basic chunks on servers and clients, including operating system, application program, and the connectivity pieces like network protocols and middleware software.



Here's our model which is defined in the context of database networking middleware:



The middleware is right there in the middle (called database connectivity middleware in our diagram). Notice that middleware components are instantiated on both client and server platforms. Since some definitions of middleware include code that isolates operating systems from hardware platform differences (like the hardware abstraction layer or HAL in Windows NT), let's keep our discussion focused on middleware for distributed systems. In some cases, there will be more middleware services provided by an intermediate device like a database gateway. The next diagram provides a more specific example using Oracle and PowerBuilder on common platforms:



What does the middleware do? Simply put, it provides a set of services to applications like a client/server database as depicted in our diagram. What are those services? Read on.

Basic Middleware Services

The basic set of middleware services are offered by most products today. We'll refer to Microsoft's SQL Server RDBMS (Relational Data Base Management System or Relational) and its DB-Library data access middleware as an example. Services offered include:

Client/Server Connectivity -Middleware provides the mechanism by which network applications communicate across the network. This includes in the case of database networking for example the service of putting packages of query results data into network transport packets. Microsoft SQL Server, for example, uses Sybase's Tabular Data Stream (TDS) protocol to handle formatting of data for transport across the network. This session layer interaction may also have its own timers and even error control to handle automatic retransmission of lost packets. One feature common is the ability for the client to interrupt the current operation on the server to cancel a large query response download. (We'll discuss the various mechanisms provided in depth when we discuss the typology of Middleware later.)

Platform Transparency -Client and server don't have to have intimate knowledge of each other in order for work to get done. Differences between platform specific encodings like big-endian and little-endian or EBCDIC and ASCII are typically hidden by middleware. Middleware often runs on a variety of platforms, letting the organization utilize all its existing desktop and server hardware as applications require. Still, some middleware products find it hard to look beyond Windows clients and UNIX or Windows NT servers. Make sure the middleware you're buying handles all the platforms you really have deployed. Microsoft SQL Server DB-Library middleware provides access only to Windows NT servers (since that's the only supported SQL Server host platform), but does so from DOS, Windows (3.1, 95, NT), Mac and UNIX clients (though not all this client software ships with SQL Server itself).



Network Transparency and Isolation -Middleware often makes networking choices transparent to application programmers. Actually, though, every middleware product we've ever heard of runs on TCP/IP, with all the other protocols coming in a distant second. If you want to be more prepared to run tomorrow's middleware, get on the TCP/IP bandwagon. Then again, don't let application programmers become too divorced from networking decisions; it can lead to poorly designed applications.



SQL Server supports multiple protocols between clients and servers, though some are specific to a given platform. From Macs, the choices are TCP/IP and AppleTalk. From PCs, there are more choices: TCP/IP, NetWare IPX/SPX, and NetBIOS/NetBEUI (Named Pipes). In some cases, the client and server don't even have to run the same network protocol between them. An intermediate device which might best be called a database relay can get the two end nodes talking to each other. But, they'll need the same middleware product everywhere to make this happen.



Application and Tool Support (APIs) -Before middleware can be used, it must present its own API to client applications that might use it. For shrink-wrapped tools like a database query tool, the API support can be critical. While ODBC has provided some level of transparency across multiple proprietary database APIs, many RDBMS vendors still encourage using their own proprietary APIs. Be sure you know what APIs your middleware offers as well as what APIs your tools can use. Hopefully there's a match! SQL Server offers both ODBC standard and DB-Library proprietary APIs on the client. For more generic middleware, the API on the server must be available as well; for RDBMS middleware, the server side is typically hard-coded to support an RDBMS.



Language Support -Middleware often provides transparency across different SQL database dialects. Even when coding in embedded SQL in a 3GL, the middleware mig ht allow the use of a single SQL dialect across a variety of RDBMS back ends. Outside of the database specific middleware products, generic middleware products often allow different programming languages to be used to create the distinct pieces of an application (pieces that reside on different machines). Since SQL Server's DB-Library only supports SQL Server RDBMSs, the SQL dialect supported is Transact SQL, a superset of ANSI 89 SQL created by Sybase and Microsoft.



RDBMS Support -When we focus on database networking middleware (also called data access middleware), middleware may also provide a level of transparency across different data storage formats. It will make different RDBMSs look like the same RDBMS. ODBC is one way of hiding RDBMS differences, but middleware products often provide multiple RDBMS support from both proprietary and standard APIs. SQL Server's DB-Library middleware does support ODBC interfaces, but still natively gets users to SQL Server RDBMS only.



Much database networking middleware is closely tied to the RDBMS of that same vendor. To get to even more data sources, database gateway products are needed. Third party products like TechGnosis SequeLink or IBI EDA/SQL offer more variety in RDBMSs. Even Microsoft has recently allied with IBI to have their multi-RDBMS connectivity solution connected to the DB-Library network so that DB-Library clients can get to RDBMSs other than SQL Server.



For non-data access products, RDBMS functionality can still be provided using extensions to make accessing that kind of data directly easy over the middleware solution deployed. NetWeave is one messaging vendor that also has database access software options.



Clearly we've run through quite a number of useful kitchen appliances already, way beyond just the sink. Let's examine the more advanced services middleware might be providing.



Types of middleware

For many programmers, middleware is like a black box. Middleware helps get the job done more quickly and keeps the programmer from having to worry about even more things. Most application programmers prefer to keep that black box closed. They'll learn how to make calls to the box and get results back, but they won't try to understand what goes on in the box.

Middleware is sometimes called plumbing because it connects two sides of an application and passes data between them. Common middleware categories include:

1. Message Oriented Middleware. This is a large category and includes asynchronous store and forward application messaging capabilities as well as integration brokers that perform message transformation and routing or even business process coordination.
   
2. Object Middleware. This category consists largely of Object Request Brokers that were mentioned on one of the earlier definitions.
   
3. RPC Middleware. This type of middleware provides for calling procedures on remote systems, hence the name Remote Procedure Call. Unlike message oriented middleware, RPC middleware represents synchronous interactions between systems and is commonly used within an application.
   
4. Database Middleware. Database middleware allows direct access to data structures and provides interaction directly with databases. There are database gateways and a variety of connectivity options. Extract, Transform, and Load (ETL) packages are included in this category.
   
5. Transaction Middleware. This category as used in the Middleware Resource Center includes traditional transaction processing monitors (TPM) and web application servers. One could make the case for splitting the category.
   
6. Portals. We include enterprise portal servers as middleware largely because they facilitate what we refer to as "front end" integration. They allow interaction between the user's desktop and back end systems and services.
   

WSCF.blue is a Visual Studio .NET 2008 add-in that provides the following features:

· A WSDL Wizard that allows the developer to step through the creation of a WSDL from one or more XSDs.

· A Data Contract Generator (similar to XSD.exe, XSDObjectGen.exe and SvcUtil.exe) that generates the .NET equivalent classes of the XSD types.

· A Service/Endpoint Stub (support for self-hosted and web-hosted) Generator and

· A Client Proxy Generator.

· A Generate Data Contract Code feature that supports the selection of multiple XSD/WSDL source files.

· A Paste XML as Schema option that generates a schema for a block of XML in the clipboard.

· Support for C# and VB.NET code generation.

· You can choose if operation methods on your service class will throw a NotImplementedException, call an implementation method in a partial class, or will be defined as abstract methods.

· Force the SOAP actions (Action and ReplyAction) applied to each operation contract follow the standard WCF format

· Errors found while processing the WSDL are reported in a WSCF.blue pane in the Visual Studio Output window.

Please verify that the receive location exists, and that the isolated adapter runs under an account that has access to the BizTalk databases.

This error occurs because the web service is running on the application pool which security account user is not a member of the BizTalk Isolated Host Users group. in order to solve this error, you need to add the user that runs the web-service into the BizTalk Isolated Host Users group. Or change the application pool to the pool which user is already a member of the BizTalk Isolated Host Users Group. :)

I've received this error while deploying a MSI application; although, I've found a solution of this error, which is:

"when you run a 32-bit program on a 64-bit Windows Server 2003-based computer"

Welcome to machine again.

In this section we will discuss some of EAI patterns, and how these could be implemented in BizTalk.

• Aggregator:
  

  Aggregator is the pattern of receiving information from multiple sources and consolidating it into a single message
  

• Splitter:
  

  How can we process a message if it contains multiple elements, each of which may have to be processed in a different way?
  

  Use a Splitter to break out the composite message into a series of individual messages, each containing data related to one item
  

  In BizTalk, there is a concept known as Envelope splitting. This is a useful feature in BizTalk for splitting messages.
  

  The Splitter pattern takes a single message and splits it into multiple messages
  

• Parallel Convoy:
  

  A parallel convoy enables multiple single messages to join together to achieve a required result. The set of related messages can arrive in any order, but BizTalk Server must receive all of them before starting the process.
  

• Sequential Convoy:
  

  The Sequential Convoy pattern enables multiple single items to ‎join together to achieve something that an individual item cannot ‎accomplish by itself. A sequential convoy is a set of related items ‎that have a predefined order. Although the items do not have to ‎be exactly the same, BizTalk Server must receive the items in a ‎sequential order.‎
  

In order to understand better what business requirements require convoy messaging, consider the three business scenarios outlined in the following section.

Scenario 1: Hospital Admits a New Patient

Business Overview: A hospital admits a new patient. The hospital requires several pieces of information from the new patient including insurance information, past medical history, and contact information. Several different people collect this information including an insurance specialist, a nurse, and a receptionist. Several different systems process this information. The order in which collection and submission of this information occurs is not guaranteed. This is because some information collectors may be busy with other patients, the medical records department may be behind in their schedule, or the insurance system may be not functioning correctly. Assembling this information for the patient in an organized manner must occur throughout the time the patient spends at the hospital. This guarantees that the patient receives accurate care and correct billing.

Technical Overview: This process involves receiving several different pieces of information about a new patient in any order. All the pieces of information are required before the business process continues and admits the patient. This reduces the number of non-payments by guaranteeing the admission of only those patients with all the requirements. This also creates a race condition in which any one of the various messages can be the first to arrive and start the business process for the new patient. Because this is a new patient, no patient number is available for correlation or reference. Accomplishing correlation in this case occurs by using date of birth, social security number, or a combination of other demographics. Any one of the three messages may be required to start a new orchestration; this creates a race condition inside the message box.

Scenario 2: Standardized Test Taking and Grading

Business Overview: A student is getting ready to take a standardized test. As part of the testing process, the student registers for the test. After registered the student checks in at the testing center on the correct date and takes the test. Upon check out, checking the student's identity occurs again and the student leaves the testing center. Finally, the student's test arrives at the grading center. Only after the completions of all these steps in order, can the student's test be graded and a score recorded.

Technical Overview: Completion of this business process requires receipt of several different types of messages. In this case, there is a predefined order to follow in which to process these messages. After receipt of the four single messages, sending a single notice occurs to allow the student's test to be graded.

Convoy ‎ means multiple single items must be related together in order to achieve something that the individual item cannot accomplish by itself

Convoy processing can be additionally broken out into two separate categories as shown in the following list:

• Sequential convoy - Sequential convoys are a set of related items that have a predefined order. The items do not have to be exactly the same. Grouping together requires that all they have to have is a purpose to group.
  
• A real world example of a sequential convoy is a group of friends headed to a party traveling in separate vehicles. The overall goal is for all of them to reach to the party but in this case, only the first person has the directions. It is up to the lead vehicle to make sure they drive to the correct location and that all the other vehicles following are not lost. In order to make sure they all stay together, the leader instructs each following vehicle to put a blue flag on the vehicle's antenna. There are many other vehicles on the road going home, to work, or to another party but only the vehicles going to this party in this group have the blue flag.
  
• Parallel convoy - Items that are in any order, but must all be present before something occurs, Parallel convoys are a set of related items that can be in any order. BizTalk Server 2004 must receive all the items before achieving the final goal. Shown here is this basic pattern. Another way of thinking of parallel convoys is like a door with many locks. The locks can be unlocked in any order. Only when the last key is turned will the door open. The key ring may have many different keys on it, but only one set matches this combination of locks.
  

• Message Broker:
  

  Message Broker is the pattern of determining the destination of a message and still maintaining control over the message flow the goal of our generic message broker will be to dispatch any type of message to its destination decided at runtime. So, by avoiding type dependency and early routing binding (basically, hard-coding) we would get single very flexible orchestration which can easily handle requirement changes - one of the top goals of good application design.
  

  
  

  
  

  So, the question is how to implement these patterns in BizTalk?
  

  The answer is very simple, check this link J
  http://www.codeplex.com/PatternWizard
  

  And enjoy BizTalking
  

Always design a thing by considering it in its next larger context — a chair in a room, a room in a house, a house in an environment, an environment in a city plan.

Introduction:

In the last post http://mohammad-yousri.blogspot.com/2009/06/happy-biztalk-to-you-part-2.html

I asked a very innocent question: Which integration approaches best address which of these criteria?

And I talked about the different approaches; do you remember the last approach? It was the Messaging, thus, we need to explain the different types of message channels. Are you ready? Go.

Let's check the following conversation between three messages; they met each other in the channel.

Message #1: Hello

Message #2: Hey, where are you going?

Message #1: I am going to System A, you?

Message #2: Don't know, I am waiting the admin to decide.

Message #3 3 3 3 3 3 3 3 3 3 3 3's…. :

Here We Go Again, Here We Go Again. the original is still there. the original is still there

Message #2: Wow, where are these 3's going?

Message #1: They are copy of 3's, and they are going to the subscribers.

• Point-to-Point Connection
  

  A point-to-point connection ensures that only one receiver receives a particular message. For this to work, the sending system must know the location of the receiving node. The sending system often must translate the message into a format that the receiving system understands.
  

  When you use point-to-point connections, each system determines the address of all the other nodes that it needs to communicate with. When target addresses or protocol details change, all the systems that communicate with the target server must be updated. As the size of your integration network grows and as the frequency of change increases, the operational cost associated with this approach becomes significant.
  

  The strength of the Point-to-Point Connection pattern is how simple it is to implement. The weakness of the Point-to-Point Connection pattern is the duplication of transformation and routing code between systems, and the high configuration cost of endpoint address changes. To minimize these weaknesses, you can add another layer of indirection between endpoints that contains a broker. (Message #1)
  

• Message Broker
  

  A message broker is a physical component that handles the communication between applications. Instead of communicating with each other, applications communicate only with the message broker. An application sends a message to the message broker, providing the logical name of the receivers. The message broker looks up applications registered under the logical name and then passes the message to them. Communication between applications involves only the sender, the message broker, and the designated receivers. The message broker does not send the message to any other applications. Use a central Message Broker that can receive messages from multiple destinations, determine the correct destination and route the message to the correct channel. (Message #2)
  

• Publish/Subscribe
  

  At a high level, the Publish/Subscribe [Buschmann96] pattern helps keep cooperating systems synchronized by one-way propagation of messages because one publisher sends a message to any number of intended subscribers.
  

  A Publish-Subscribe Channel works like this: It has one input channel that splits into multiple output channels, one for each subscriber. When an event is published into the channel, the Publish-Subscribe Channel delivers a copy of the message to each of the output channels. (Message #3)
  
  

Today I am going to explain the application integration options, by the way there's more than one approach for integrating applications, each approach ‎addresses some of the integration criteria better than others. The various ‎approaches can be summed up in four main integration styles: ‎

Wait !!!!!!!!! What are the integration criteria?

What makes for good application integration? If integration needs ‎were always the same, there would only be one integration style. Yet ‎like any complex technological effort, application integration involves a ‎range of considerations and consequences that should be taken into ‎account for any integration opportunity. So, find below the integration criteria:

• Application coupling — Even integrated applications should minimize their dependencies on each other so that each can evolve without causing problems for the others. Tightly coupled applications make numerous assumptions about how the other applications work; when the applications change and break those assumptions, the integration breaks. The interface for integrating applications should be specific enough to implement useful functionality, but general enough to allow that implementation to change as needed.
  

  
  

• Integration simplicity — when integrating an application into an enterprise, developers should strive to minimize changing the application and minimize the amount of integration code needed.
  

  Yet changes and new code will usually be necessary to provide good integration functionality, and the approaches with the least impact on the application may not provide the best integration into the enterprise.
  

  
  

• Integration technology — Different integration techniques require varying amounts of specialized software and hardware. These special tools can be expensive, can lead to vendor lock-in, and increase the burden on developers to understand how to use the tools to integrate applications.
  

  
  

• Data format — integrated applications must agree on the format of the data they exchange, or must have an intermediate traslator to unify applications that insist on different data formats. A related issue is data format evolution and extensibility—how the format can change over time and how that will affect the applications.
  

  
  

• Data timeliness — Integration should minimize the length of time between when one application decides to share some data and other applications have that data. Data should be exchanged frequently in small chunks, rather than waiting to exchange a large set of unrelated items.
  

  Applications should be informed as soon as shared data is ready for consumption. Latency in data sharing has to be factored into the integration design; the longer sharing can take, the more opportunity for shared data to become stale, and the more complex integration becomes.
  

  
  

• Data or functionality — integrated applications may not want to simply share data, they may wish to share functionality such that each application can invoke the functionality in the others.
  

  Invoking functionality remotely can be difficult to achieve, and even though it may seem the same as invoking local functionality, it works quite differently, with significant consequences for how well the integration works.
  

  
  

As you can see, there are several different criteria that must be considered when choosing and designing an integration approach. The question then becomes:

Which integration approaches best address which of these criteria? ‎

• File Transfer — Have each application produce files of shared data for others to consume, and consume files that others have produced. In the File Transfer pattern, one application produces a file and transfers it so that other applications can consume it. Because files are a universal unit of storage for all enterprise operating systems, this method is often simple to implement.
  

  The applications must agree on file naming conventions and the directories they appear in. The writer of a file must implement a strategy to keep the filenames unique. The applications must agree on which one will delete old files, and that application will have to know when a file is old and no longer needed. The applications will need to implement a locking mechanism or follow a timing convention to ensure that one application is not trying to read the file while another is still writing it. If all of the applications do not have access to the same disk, then some application must take responsibility for transferring the file from one disk to another.
  

  
  

  While this approach seems simplistic, it does have some advantages. Using files affords good physical decoupling between the different processes. If the target system is not immediately available to receive the file, it can be stored until the system becomes ready. Also, files are essentially platform and language independent as long as they use a common character set.
  

  However, batch file transfers also present a number of challenges. Data changes in one system may not be available in the other system until the next day. This may be confusing to users, and cause data integrity problems if the (outdated) data is updated in the other system. Additionally, batch exchanges tend to extract all available information from one system and replicate it to the other system. If only few changes were made, this approach will cause large amounts of unnecessary data transmission.
  

• Shared Database — Have the applications store the data they wish ‎to share in a common database.‎
  

  In an attempt to eliminate data synchronization issues and the ‎replication of huge amounts of data, some enterprises went to ‎create a shared database resource that would be shared by all ‎systems. Having the data in a single place would eliminate the ‎need for duplication of data. Synchronization issues and ‎contention can be addressed by the locking and protection ‎mechanisms provided by most database systems. The key ‎drawback of this solution lies in the difficulty of defining a data ‎model that fits all applications. Many applications are built or ‎purchased with proprietary data models and many attempts to ‎morph all these models into a single, universal data model turned ‎out to be a never ending task. Also, accessing a single database ‎may cause a serious performance bottleneck. In addition, this ‎solution still simply exchanges data items, rather than managing ‎business functions across systems.‎
  

• Remote Procedure Invocation — Have each application expose ‎some of its procedures so that they can be invoked remotely, and ‎have applications invoke those to run behavior and exchange data.‎ Remote Procedure Call (RPC) mechanisms isolate the application ‎from the raw data exchange mechanisms through an additional ‎layer. This layer manages the marshalling of complex data types ‎into byte streams as required by the transport layer. As a result, ‎RPC mechanisms allow an application to transparently invoke a ‎function implemented in another application. The marshalling ‎mechanisms rely on stubs generated from the interface ‎specification in a language-neutral Interface Definition Language ‎‎ (IDL).‎
  

  While RPC-based systems made dealing with the integration much ‎simpler, these systems still share the drawbacks of unreliable, ‎synchronous communication with the raw data exchange ‎methods. RPC-based systems also imply brittle point-to-point ‎connections between senders and, receivers that become difficult ‎to maintain as the number of participants increases. Furthermore, ‎most vendor-provided RPC implementations are not interoperable ‎with other products.‎
  

• Messaging — Have each application connect to a common ‎messaging system and exchange data and invoke behavior using ‎messages.‎
  

  Messaging systems try to overcome the drawbacks of the previous ‎solutions by providing reliable, asynchronous data transfer. An ‎application can publish data to the integration layer and can be ‎assured that the data will be delivered to the recipient(s). The ‎originating system is not required to wait for an acknowledgement ‎and can continue the primary process flow. Messaging systems ‎also incorporate addressing schemes that avoid the difficult to ‎maintain point-to-point connections that are typical in RPC ‎systems. At this time, it appears that messaging systems provide ‎the best basis for an integration solution. However, there are still ‎some challenges to be overcome. Messaging systems are still ‎comparatively new and the body of knowledge around architecting ‎these systems is still evolving. Asynchronous systems require a ‎different approach to systems architecture and design than ‎synchronous systems do. Furthermore, the asynchronous nature ‎of the interaction can make testing and debugging more difficult as ‎well.‎
  

Ummmm….Messaging, seems that we are talking about BizTalk.
J

Conclusion:

Today, we talk about the integration criteria which make your integration application good, according to these criteria; you have to choose the best approach that matches the criteria is.

In the next article, I will explain the different between messaging channels, so, be there.