Microsoft BizTalk Server has been around for a few years now and apart from some hands on labs at a Microsoft Tech Ed conference, I’ve not had much opportunity to get involved with it. As BizTalk Server 2004 is the product’s third incarnation and it seems to be gaining momentum in the marketplace, I attended a presentation given by Nick Page (a Technical Evangelist at Microsoft UK) and the following are my notes from the event.

Today’s IT environments are complex with multiple servers, operating systems and protocols, many disparate applications (ERP, HR, Finance) and many programming models (e.g. COM, J2EE, .NET, CICS). At the same time, organisations are demanding integration between internal systems, and integration with business partners.

Tightly-coupled systems impose many dependencies, requiring lots of commonality between systems (e.g. those of an organisation and its business partners), whereas loosely-coupled systems lend themselves to a service orientated architecture (SOA). In this context, a service is a reusable business process or business function, that can be accessed through public and private networks.

In the past, attempts to standardise integration technologies have been hampered by a lack of industry acceptance and adoption but today we have a ubiquitous network (the Internet), a ubiquitous data format (XML) and all major vendors support XML web services.

The web services dream is for each line of business (LOB) system to be connected to all the applications from which it needs to consume services.

This is all very well, but once more than one system is involved the mesh of connections becomes difficult to manage.

The answer lies in the use of an integration server, and Microsoft’s enterprise application integration (EAI) product is BizTalk Server, although BizTalk offers more than just EAI.

BizTalk Server 2000 and 2002 separated messaging and orchestration. BizTalk Server 2004 re-integrates the two, using an architecture which is summarised in the following diagram.

BizTalk Server includes an adapter framework to build adapters (which are actually Microsoft .NET components) and all BizTalk Server 2004 adapters are configured in the same manner (unlike previous versions). Examples of adapters may be:

• A file system adapter, which looks for the presence of a new file in a particular location.A SQL adapter, which understands the database schema.
• An HTTP adapter, for transport.
• An API adapter (e.g. SAP), mapping XML messages to native application programming interfaces (APIs).

Out of the box, BizTalk Server includes EDI, flat file, FTP, HTTP, SOAP and SQL adapters but many more are available from ISVs.

As messages (with context) pass through the pipeline, components are used to translate/process the message resulting in and XML output. Sometimes, this processing takes place within the adapter – the decision where to put the processing depends on whether they will be reused or not (e.g. if there are many adapters working with the same transport, such as flat file .CSV, then this processing may be put in the pipeline). An example use for a component would be to validate messages against a schema. Standard components included with BizTalk include XML receive and pass-through receive.

With BizTalk Server 2004, all components act directly on the messages, whereas previous BizTalk Server versions used the Microsoft Message Queue (MSMQ).

By promoting part of a message into context (i.e. putting it into the property schema), then messages can be routed on content, avoiding the requirement to load the whole message at each stage in the pipeline.

The business rules portion of the process is where the orchestration takes place, defining processes that interact with systems and apply business logic to the XML messages.

The message box is a SQL Server database, used to store the XML messages, which have rich properties, allowing them to be linked to services through a publish/subscribe model. Unlike in previous versions of BizTalk Server, from 2004, all messages pass via the SQL Server and a single message passed into the message box database work queue could result in multiple outgoing messages (each one a copy of the original), based on the subscriptions of other services.
The configuration database contains details of all the ports in the integration, whilst the tracking database tracks the progress of the message.

BizTalk Server 2004 ports are intended to be stateless so as to allow for scaling in/out (e.g. using network load balancing), with SQL Server stored procedures carrying out the routing so that many processing servers may be provided for one common database; however there are some exceptions – persisted (long running) transactions, correlation messages and session-based protocols all need to be routed to the correct server. Storage can also be scaled out using multiple SQL Servers but if clustering is used, then there will be a 30-60 second delay during a cluster failover.

Just as pipeline components are used to parse and serialise messages, message security is provided using components in the pipeline, for example decrypting/encrypting/signing messages or resolving parties (i.e. mapping to a certificate). It is also possible to segment security with multiple logical hosts using different service accounts. This is particularly useful if multiple suppliers are involved in a process, where BizTalk will reject incoming messages with no party ID and the message box will not allow messages from untrusted hosts. The incoming and outgoing ports may use separate service accounts, as may the orchestration element.

BizTalk Server is designed for throughput (scalability) and not latency (sub-second point-to-point responses), so orchestration processes are not left running and waiting for a response indefinitely as this will consume system resources unnecessarily. Instead, processes are dehydrated to the database and correlation (a collection of message properties) is used to about matching incoming messages to orchestration processes and rehydrate processes as necessary.

For developing solutions, BizTalk Server 2004 includes the BizTalk project system (hosted within Microsoft Visual Studio), to provide an environment for designing, organising, and building the various elements of BizTalk applications.

When a BizTalk project is created, it generally includes one or more of the following file types which play specific roles in the creation of the business solution, and the BizTalk project system provides a corresponding graphical design tool for each of them.

• Orchestrations model the business process. Orchestration Designer simplifies the process of creating orchestrations (models of business processes that are compiled into executable code).
• Schemas describe the data that is sent and received. BizTalk Editor simplifies the process of defining schemas. Schemas are used to describe the format of data that is processed within organisations and between trading partners.
• Maps transform data from one format to another. BizTalk Mapper presents source schemas and destination schemas side-by-side and enables the definition of transformations between data elements of different messages.
• Pipelines perform a variety of operations to prepare incoming or outgoing messages for further processing. Pipeline Designer allows the preparation of incoming and outgoing messages for further processing by implementing such operations as encryption and decryption, compression, reformatting, and validation.

Other BizTalk terms include:

• Functoids – used to map two schemas and transform data (e.g. cumulative sum, database lookup, or run a script, calling out to external or in-line code). The transformation can occur wherever the messages are in XML format (i.e. after the pipeline or during orchestration).
• Vocabulary – mapping English terms/descriptions to elements within a message.Policy – a set of rules used to work on vocabulary or message elements. These rules are deployed to BizTalk servers but by extracting the rules from the business process, the rules are not hard coded and so allow for modification in line with business process changes.
• Envelopes – used to wrap the core data when receiving (disassembling) or sending (assembling) messages.
• Long-running transactions – when business processes last longer than a few seconds (e.g. send a request for assessment and expect a response after a few days). The converse is a short-running transaction, which uses a locking mechanism but this is too resource-intensive to scale so BizTalk Server 2004 has the concept of flexible delay.
• Compensation – if a process aborts and affects the results of a previously completed transaction, it cannot be undone. Compensation processes are used for actions which cannot be undone.
• Exception block – the method by which BizTalk Server 2004 deals with exceptions when external calls go wrong.

Finally, some of the other features provided by BizTalk Server 2004 include:

• Single sign on (SSO) capabilities – mapping user credentials from Windows to non-windows systems allowing services to authenticate once and then securely access target systems. An encrypted secret store contains the user mappings. BizTalk Server’s SSO integrates with Windows SharePoint Services (WSS) to provide SSO for a portal and has a web services front end for any client to any platform.
• Business activity monitoring (BAM) – allowing users to query an OLAP cube from within Microsoft Excel and ask near real-time questions such as “how long is production taking right now?” or aggregation questions such as “how much money did we make last month?”. Because the OLAP cube is extracted from the database, it is not quite real time. This can use data from documents or process and complements the existing SQL Server business intelligence (BI) offerings.
• Health and activity monitoring – this includes the Orchestration Debugger, which allows a message to be replayed through a process to follow calls into other Orchestrations, view exceptions thrown and examine any compensation code executed; or alternatively to attach to live orchestrations, setting breakpoints, watching variables and retrieving message contents before resuming processing until the next breakpoint.
• Human workflow services (HWS) – business analysts and developers can use standard orchestration processes for interaction, or HWS for more dynamic requirements. Each orchestration becomes a step in the workflow with permissions; however HWS is all about background processing and has no user interface, so third party products are needed to expand on this.