I’m not a BizTalk Server expert (by any stretch of the imagination), but I do know the concepts behind the product. Just before I left Conchango, I had the opportunity to attend a session delivered by Sue MacDermott (a technical pre-sales specialist with Microsoft UK) where she outlined the new features in the next release – BizTalk Server 2006 – currently scheduled for en early 2007 release. Some of what I was told is covered by a non-disclosure agreement (NDA), but the information below is in the public domain.

BizTalk Server 2006 is not a major release – Microsoft’s current cycle is for a major release every four years and an incremental release in between, so from that we can expect to see the next major BizTalk release, making use of the Windows communication foundation (codenamed Indigo), to be released in early 2008.

It is a common misconception that BizTalk Server 2006 will be released this November, at the same time as Visual Studio 2005 (codenamed Whidbey) and SQL Server 2005 (codenamed Yukon). In fact, it is expected that BizTalk Server 2006 will be officially launched at the same November 7th event, but the only product available at that time is expected to be beta 2. At the time of writing, Microsoft expect to provide a release candidate in the new year, before the product is finally released in the spring.

The reason for the delay (and the 2006 moniker, whereas SQL Server and Visual Studio are both 2005 products) is that there is a dependency on some of the 2005 technologies that are being released in November – namely Visual Studio 2005 and the Microsoft .NET Framework v2.0. There are no hard dependencies on SQL Server 2005, and BizTalk Server 2006 can used either SQL Server 2000 or 2005, but Microsoft did say that initial testing has indicated significant performance improvements when run on the latest SQL Server build (one of my former colleagues at Conchango indicated this may be as much as 30% faster).

Detailing all of the enhancements in BizTalk Server 2006’s is too much for a single blog post, (and in any case, much of the information should soon be available from Microsoft) but the main improvements are across the following areas:

• Management and operations, introducing the concept of a BizTalk application which groups related components such that the administrator’s view can match the application architecture.
• Business user empowerment with real-time alerting and notification, a business activity monitoring (BAM) portal and deeper Windows SharePoint Services (WSS) integration.
• Windows server system integration (with support for SQL Server 2005, Visual Studio 2005 and the Microsoft .NET Framework 2.0, Virtual server 2005 the 64-bit versions of Windows Server 2003).
• Setup, upgrade and deployment, with a new installer which checks for dependencies (split into mandatory “T1” items, which will block installation if they are missing, and “T2” items such as MSXML where a .CAB file may be downloaded if necessary to ensure that the latest versions are available at installation time), simplified configuration (through the application paradigm), and improved orchestration deployment (down from 74 clicks in BizTalk server 2004 to just a few operations within the new BizTalk Administration Console).

Other improvement areas are the core engine, with improvements around:

• Handling large messages during a transformation – writing out to disk rather than running out of memory, albeit with a corresponding performance hit.
• Handling bad messages with out having to roll back all related messages.
• Ordered delivery to ensure that sequenced messages arrive in sequence.
• More granular performance counters.
• A new flat file schema wizard.
• Engine throttling.

There are also new adapters, with MSMQ and MQSeries adapters now available out of the box (for BizTalk Server 2004 these were separate downloads), as well as new e-mail receive (POP3) and Windows SharePoint Services (WSS) adapters. In addition, existing adapters are enhanced (e.g. e-mail compose within the SMTP adapter, usability improvements and performance counters for adapter troubleshooting). Other new features include the ability to connect to UNC file shares using alternate credentials, SOAP array support and an ability to call web services without orchestrations (i.e. messaging only scenarios) using content based routing (CBR) send ports, and the ability to suspend failing HTTP requests.

On the development front there are new redeployment tools, support for zooming in/out of large orchestrations, and collapsed shapes are preserved in the orchestration designer (OD).

Overall, administration is simplified so that most operations are controlled through the BizTalk Administration Console; although health and activity tracking (HAT) is still available and Microsoft Operations Manager (MOM) is recommended for monitoring not only BizTalk (with an updated management pack) but also all of the related components (IIS, SQL Server, etc.).

The BizTalk Administration Console is a Microsoft management console (MMC) snap-in, with a new group hub concept which allows the overall status to be viewed at a glance, as well as improvements for analysing the root cause of issues, isolating errors, grouping and filtering of queries, and bulk operations (e.g. resume all, terminate all, suspend all).

Administration can also be performed via scripting APIs, or the command line (a number of sample scripts are available).

Microsoft are making a great play on the ordered processing functionality in BizTalk Server 2006 and the demonstration I saw showed a graphical application with sending and receiving components whereby the presenter wrote her name in the sending application the vectors for the pixelated data were sent to make it appear (albeit a bit jumbled), in the receiving application. Once ordered delivery was enabled, the sending and receiving copies were identical. This ordered processing can be handled in a number of ways and send-side order processing is available for any adapter; but if implemented on the receiving end, it requires an adapter with appropriate support (e.g. MSMQ or MQSeries) or for sequential data, an HTTP or SOAP adapter can be used. Orchestrations can use the ordered delivery setting on the orchestration receive port and a orchestration convoy to get the stream of ordered messages.

Looking at the new adapters, the WSS adapter features:

• Receipt of documents from (and posting documents to) a SharePoint document library.
• Filter inbound documents based on views.
• Archival of documents to another document library.
• Promotion of document properties.

The new POP3 adapter features:

• Polling for e-mail and attachments via a POP3 receive location.
• Population of e-mail header properties within the message context.
• POP3 over SSL.
• Configurable TCP port number.

Line of business adapter choices are also enhanced with Microsoft’s purchase of the iWay adapters for:

• Clarify.
• JD Edwards.
• Oracle Applications.
• Oracle DB.
• PeopleSoft.
• SAP.
• Siebel.
• TIBCO Rendezvous.
• TIBCO JMS (EMS).

(iWay customers that have purchased licenses for the .NET-based adapters will receive a license for the corresponding Microsoft adapter with the purchase of Software Assurance).

A major enhancement in BizTalk Server 2006 is the flat file schema wizard – used to accept messages from a flat file, for example a comma separated variables (.CSV) file, an EDI document, or a text file produced by a custom legacy application. To enable processing of this format using BizTalk Server developer needs to define a flat file schema (an XSD with additional flat file annotations).

Also improved is the interchange processing related to flat file document interchange. In BizTalk Server 2004, one bad document will result in the whole interchange being suspended whereas with BizTalk Server 2006’s recoverable interchange processing, only the “bad” elements are suspended and most of the processing is carried out as normal.

For business users, BizTalk Server 2006 has improved alerting capabilities, finally providing real time information via a new out-of-the-box portal and native integration with BizTalk messaging. There is also a software development kit with a new dynamic web part generator for WSS as well as integration with the Microsoft Office Business Scorecards Accelerator and SQL Server 2000 Reporting Services.

Finally, for those who want to have a look at BizTalk Server 2006, Owen’s blog has details of where to download the latest beta.

Microsoft BizTalk Server has been around for a few years now and apart from some hands on labs at a Microsoft Tech Ed conference, I’ve not had much opportunity to get involved with it. As BizTalk Server 2004 is the product’s third incarnation and it seems to be gaining momentum in the marketplace, I attended a presentation given by Nick Page (a Technical Evangelist at Microsoft UK) and the following are my notes from the event.

Today’s IT environments are complex with multiple servers, operating systems and protocols, many disparate applications (ERP, HR, Finance) and many programming models (e.g. COM, J2EE, .NET, CICS). At the same time, organisations are demanding integration between internal systems, and integration with business partners.

Tightly-coupled systems impose many dependencies, requiring lots of commonality between systems (e.g. those of an organisation and its business partners), whereas loosely-coupled systems lend themselves to a service orientated architecture (SOA). In this context, a service is a reusable business process or business function, that can be accessed through public and private networks.

In the past, attempts to standardise integration technologies have been hampered by a lack of industry acceptance and adoption but today we have a ubiquitous network (the Internet), a ubiquitous data format (XML) and all major vendors support XML web services.

The web services dream is for each line of business (LOB) system to be connected to all the applications from which it needs to consume services.

This is all very well, but once more than one system is involved the mesh of connections becomes difficult to manage.

The answer lies in the use of an integration server, and Microsoft’s enterprise application integration (EAI) product is BizTalk Server, although BizTalk offers more than just EAI.

BizTalk Server 2000 and 2002 separated messaging and orchestration. BizTalk Server 2004 re-integrates the two, using an architecture which is summarised in the following diagram.

BizTalk Server includes an adapter framework to build adapters (which are actually Microsoft .NET components) and all BizTalk Server 2004 adapters are configured in the same manner (unlike previous versions). Examples of adapters may be:

• A file system adapter, which looks for the presence of a new file in a particular location.A SQL adapter, which understands the database schema.
• An HTTP adapter, for transport.
• An API adapter (e.g. SAP), mapping XML messages to native application programming interfaces (APIs).

Out of the box, BizTalk Server includes EDI, flat file, FTP, HTTP, SOAP and SQL adapters but many more are available from ISVs.

As messages (with context) pass through the pipeline, components are used to translate/process the message resulting in and XML output. Sometimes, this processing takes place within the adapter – the decision where to put the processing depends on whether they will be reused or not (e.g. if there are many adapters working with the same transport, such as flat file .CSV, then this processing may be put in the pipeline). An example use for a component would be to validate messages against a schema. Standard components included with BizTalk include XML receive and pass-through receive.

With BizTalk Server 2004, all components act directly on the messages, whereas previous BizTalk Server versions used the Microsoft Message Queue (MSMQ).

By promoting part of a message into context (i.e. putting it into the property schema), then messages can be routed on content, avoiding the requirement to load the whole message at each stage in the pipeline.

The business rules portion of the process is where the orchestration takes place, defining processes that interact with systems and apply business logic to the XML messages.

The message box is a SQL Server database, used to store the XML messages, which have rich properties, allowing them to be linked to services through a publish/subscribe model. Unlike in previous versions of BizTalk Server, from 2004, all messages pass via the SQL Server and a single message passed into the message box database work queue could result in multiple outgoing messages (each one a copy of the original), based on the subscriptions of other services.
The configuration database contains details of all the ports in the integration, whilst the tracking database tracks the progress of the message.

BizTalk Server 2004 ports are intended to be stateless so as to allow for scaling in/out (e.g. using network load balancing), with SQL Server stored procedures carrying out the routing so that many processing servers may be provided for one common database; however there are some exceptions – persisted (long running) transactions, correlation messages and session-based protocols all need to be routed to the correct server. Storage can also be scaled out using multiple SQL Servers but if clustering is used, then there will be a 30-60 second delay during a cluster failover.

Just as pipeline components are used to parse and serialise messages, message security is provided using components in the pipeline, for example decrypting/encrypting/signing messages or resolving parties (i.e. mapping to a certificate). It is also possible to segment security with multiple logical hosts using different service accounts. This is particularly useful if multiple suppliers are involved in a process, where BizTalk will reject incoming messages with no party ID and the message box will not allow messages from untrusted hosts. The incoming and outgoing ports may use separate service accounts, as may the orchestration element.

BizTalk Server is designed for throughput (scalability) and not latency (sub-second point-to-point responses), so orchestration processes are not left running and waiting for a response indefinitely as this will consume system resources unnecessarily. Instead, processes are dehydrated to the database and correlation (a collection of message properties) is used to about matching incoming messages to orchestration processes and rehydrate processes as necessary.

For developing solutions, BizTalk Server 2004 includes the BizTalk project system (hosted within Microsoft Visual Studio), to provide an environment for designing, organising, and building the various elements of BizTalk applications.

When a BizTalk project is created, it generally includes one or more of the following file types which play specific roles in the creation of the business solution, and the BizTalk project system provides a corresponding graphical design tool for each of them.

• Orchestrations model the business process. Orchestration Designer simplifies the process of creating orchestrations (models of business processes that are compiled into executable code).
• Schemas describe the data that is sent and received. BizTalk Editor simplifies the process of defining schemas. Schemas are used to describe the format of data that is processed within organisations and between trading partners.
• Maps transform data from one format to another. BizTalk Mapper presents source schemas and destination schemas side-by-side and enables the definition of transformations between data elements of different messages.
• Pipelines perform a variety of operations to prepare incoming or outgoing messages for further processing. Pipeline Designer allows the preparation of incoming and outgoing messages for further processing by implementing such operations as encryption and decryption, compression, reformatting, and validation.

Other BizTalk terms include:

• Functoids – used to map two schemas and transform data (e.g. cumulative sum, database lookup, or run a script, calling out to external or in-line code). The transformation can occur wherever the messages are in XML format (i.e. after the pipeline or during orchestration).
• Vocabulary – mapping English terms/descriptions to elements within a message.Policy – a set of rules used to work on vocabulary or message elements. These rules are deployed to BizTalk servers but by extracting the rules from the business process, the rules are not hard coded and so allow for modification in line with business process changes.
• Envelopes – used to wrap the core data when receiving (disassembling) or sending (assembling) messages.
• Long-running transactions – when business processes last longer than a few seconds (e.g. send a request for assessment and expect a response after a few days). The converse is a short-running transaction, which uses a locking mechanism but this is too resource-intensive to scale so BizTalk Server 2004 has the concept of flexible delay.
• Compensation – if a process aborts and affects the results of a previously completed transaction, it cannot be undone. Compensation processes are used for actions which cannot be undone.
• Exception block – the method by which BizTalk Server 2004 deals with exceptions when external calls go wrong.

Finally, some of the other features provided by BizTalk Server 2004 include:

• Single sign on (SSO) capabilities – mapping user credentials from Windows to non-windows systems allowing services to authenticate once and then securely access target systems. An encrypted secret store contains the user mappings. BizTalk Server’s SSO integrates with Windows SharePoint Services (WSS) to provide SSO for a portal and has a web services front end for any client to any platform.
• Business activity monitoring (BAM) – allowing users to query an OLAP cube from within Microsoft Excel and ask near real-time questions such as “how long is production taking right now?” or aggregation questions such as “how much money did we make last month?”. Because the OLAP cube is extracted from the database, it is not quite real time. This can use data from documents or process and complements the existing SQL Server business intelligence (BI) offerings.
• Health and activity monitoring – this includes the Orchestration Debugger, which allows a message to be replayed through a process to follow calls into other Orchestrations, view exceptions thrown and examine any compensation code executed; or alternatively to attach to live orchestrations, setting breakpoints, watching variables and retrieving message contents before resuming processing until the next breakpoint.
• Human workflow services (HWS) – business analysts and developers can use standard orchestration processes for interaction, or HWS for more dynamic requirements. Each orchestration becomes a step in the workflow with permissions; however HWS is all about background processing and has no user interface, so third party products are needed to expand on this.