I recently found myself working with a customer to troubleshoot issues with their Exchange Servers. The servers were losing contact with each other periodically and then doing what Exchange is designed to do in this circumstance, failing databases over to other servers. Unfortunately, whilst I’m not going to go into the exact details here for reasons of confidentiality, it did completely underline my opinion (backed up by Microsoft’s Preferred Architecture for Exchange) that virtualised Exchange solutions using SAN storage are overly-complex and will cause issues at some point…

After going through the basics (are all the servers patched up to date, including consistent BIOS and firmware for all disks, controllers, NICs, etc.; has anti-virus software been disabled on the servers, especially host intrusion protection features; is the virtual infrastructure correctly configured, especially virtual networks; are there any tell-tale issues in the event logs?), I contacted my former colleague, Fujitsu Distinguished Engineer and Exchange Master, Nick Parlow (who fixes people’s broken Exchange solutions for a living).  Nick gave me some very useful advice for troubleshooting Exchange that I’m going to share here…

• Experfwiz (the Exchange Performance Data Collection tool) can be run to collect information on how the servers are performing.  There should be a lot of information in the Daily Performance Logs too (as described by Tony Redmond as long as no-one has deleted the scheduled task to save the logs from being created and taking up disk space!)
• Take the output from Experfwiz (or the Daily Performance Logs) and run it through the Performance Analysis of Logs (PAL) Tool, which will produce an HTML report that can be used to identify issues based on recognised thresholds (and version 2.7.3 includes Exchange 2013 thresholds).

A few more points that might be useful:

• The Exchange Best Practices Analyzer (ExBPA) is back. Sort of. It’s now positioned as a tool to check Exchange before integrating with Office 365 and needs an Office 365 logon (as MVP Damian Scoles explains).
• Exchange DAG database failover is logged in Microsoft/Exchange/HighAvailability/Debug (look for Event IDs 326/7/8).  These could be used as a trigger to stop the Microsoft Network Monitor (NetMon) from running… at which point you should have a network trace showing what was happening at the point when the databases failed over. (You could try something similar with Wireshark too.)
• MVP Nuno Mota has lots of advice on monitoring DAG failover on MSExchange.org.
• There’s a useful Exchange 2013 Performance Health Check Script in the TechNet Gallery. This helped to flag some minor issues that were worth fixing – like that Hyperthreading was enabled on the servers, that an out of date version of the .NET Framework was installed and that the power plan was balanced, rather than high performance (this will kill performance on a virtual machine – and I can’t see why any server would use anything other than high performance).
• Test-ReplicationHealth is a useful cmdlet to know (MVP Paul Cunningham has a post about it too).
• Test-ServiceHealth is another one (and here’s Paul’s post on that cmdlet).
• http://servername/virtualdirectory/healthcheck.htm can be used to test the response from a given server for a given protocol (and test any load balancing).

In a previous role, I managed a team that was responsible for Microsoft Exchange design. Working with Microsoft, we established a set of design principles, some template designs and a rule-book for any changes made to those designs. Soon afterwards, Microsoft published their Preferred Architecture for Exchange and the similarity was striking (to be honest, I’d have been concerned if it was different)!

As that Preferred Architecture is publicly available, I think it’s fine to talk about some of the principles we applied. They seem particularly pertinent now as I was recently working with a customer whose Exchange design doesn’t follow these… and they were experiencing a few difficulties that were affecting the stability of their systems…

Physical, not virtual

Virtualisation has many advantages in many circumstances but it is not a “silver bullet”. It also brings complexity and operational challenges into Exchange design, with few (if any) advantages that would not be already provided by Exchange out of the box. Exchange is designed to make full use of the available hardware and Microsoft is able to provide large, low cost mailboxes within Office 365 (Exchange Online), without a requirement to virtualise their Exchange 2013 platform. In addition to the operational and supportability complexities that virtualisation brings, virtualising the Exchange deployment requires more Exchange design effort.

Deploy multi-role Exchange servers

Microsoft’s current recommended practice is to deploy multi-role Exchange 2013 servers (i.e. client access and mailbox roles on the same server) for the following reasons:

1. Reduced hardware. Multi-role servers make best use of processor capacity given the more powerful server specifications which are now available.
2. Reduced operational and capital expenditure. Fewer servers to deploy and manage.
3. Building block design which is simple to deploy and scale. Automated deployment of standard server builds.

The mailbox server role must be designed not to exceed the maximum processor capacity guidance for multi-role servers; this provides confidence that the hardware deployed can co-host all roles on a single server. This is where the Exchange 2013 Server Role Requirements Calculator comes in…

Use direct attached storage – not a SAN

Microsoft designed Exchange 2013 to run on commodity hardware and believes this is the most cost effective way to provide storage for the Exchange mailbox databases.  Changes to the Exchange 2013 storage engine have yielded a significant reduction in I/O over previous versions of Exchange, allowing customers to take advantage of larger, cheaper disks and reduce the overall solution costs. In general, Direct Attached Storage (DAS) should be used in a Just a Bunch of Discs (JBoD) configuration although there are some circumstances where a Redundant Array of Inexpensive Devices (RAID) configuration may be used.

Microsoft uses a commoditised email platform with DAS and JBoD architecture to provide and support large, low cost mail mailboxes within Office 365. There are many more solution elements to consider with a SAN (Host Bus Adapters (HBAs), fibre channel switches and SAN I/O modules) as well as additional software for managing the infrastructure and firmware to keep up-to-date. Consequently, there is an increased likelihood of technical integration issues using a SAN and, once installed, a SAN infrastructure has to be carefully monitored and managed with appropriately skilled staff. In stark contrast, the costs of direct-attached JBoD solutions is falling as larger disks become available.

Native resilience

Database availability groups (DAGs) were introduced in Exchange Server 2010 to replicate databases between up to 16 servers. A DAG with multiple mailbox database copies, can provide automatic recovery from a variety of server, storage, network and other hardware failures. Auto-reseed functionality in Exchange Server 2013 allows for automatically bringing spare disks on line in the event of failure and creating new database copies.

If four highly available copies of each database are deployed, Exchange native resilience can be used without the requirement for third party backup solutions. Only specific requirements (i.e. ability to recover to an offline datacentre; recovery of deleted mailbox outside the deleted mailbox recovery retention time; protection against operational immaturity; protection against security breaches etc.) drive a requirement for adoption of a third party backup solution

Exchange Online uses the Exchange native resilience to protect against database failures, without resorting to the use of third party backup solutions.

Whilst a DAG can support 16 servers, it may be prudent to artificially limit the number of DAG members (e.g. to 12) in order to provide flexibility in upgrade scenarios.

Site resilience

DAGs can be extended between sites and copies of databases replicated across sites to provide additional redundancy. Each member of the DAG must have a round trip latency no greater than 500ms to contact the other members, regardless of their physical location. In general, Exchange DAGs should span at least two physical sites and Microsoft also recommends that separate Active Directory sites are used.

Mailbox distribution

With multiple sites in use, the next consideration is whether both are active (i.e. providing live service) or whether one is a secondary, passive, datacentre (i.e. invoked for disaster recovery purposes).  If all active mailboxes are hosted in a single site, and all passive copies of the mailboxes reside in a secondary site, the user distribution model is referred to as active/passive. If there are active mailboxes in both primary and secondary datacentres then the user distribution model is known to be active/active.

This should not be confused with the databases within the DAG, where only one copy of each database is active at any time.

Exchange 2013 simplified the client access model and with all clients connecting using HTTPS an active/active architecture is simple and spreads the client load across all Client Access servers, making best use of the deployed hardware.

This also facilitates a simplified SMTP namespace and allows automatic site failure (assuming the File Share Witness is located in a tertiary datacentre).

Archiving

With today’s storage capabilities, large mailboxes are becoming normal.  The use of a native Exchange 2013 archive or a third party archiving solution is only required where there is a defined need for a user experience that warrants the management of email data (the ‘personal archive’ user experience, e.g. auto archive functionality) or by legal/policy requirements regarding the retention and discovery of email data (‘the regulatory archive’).

There is a common misconception that using a third party archive solution will provide a cost effective, single instance storage solution by differentiating ‘hot’ and ‘cold’ data and providing the ability to store ‘colder’ data on cheaper, slower disks. In fact, introducing a secondary system increases costs and complexity (in design and management) as well as reducing the flexibility of the solution.

Many organisations are electing to leave behind their archives with browser-only access as they migrate to larger online mailboxes in the cloud, e.g. using Exchange Online.

Conclusion

Whilst Exchange is supported in a virtualised environment, with SAN-attached storage, third party backup and making use of email archive solutions, deviating from the Preferred Architecture is a huge risk. The points in this blog post, combined with Microsoft’s advice linked above highlight the reasons to keep your Exchange design as simple as possible. Whilst a more complex design will probably work, identifying issues when it doesn’t will be a much bigger challenge.

For the last couple of years, I’ve had the privilege of leading a team of talented Exchange and Lync subject matter experts (either directly or more recently as a virtual team).  I’ve tried to keep up my technical skills but inevitably I’m not at the level of detail I once was and, as I switch to a more technical role, I’m expecting to have to re-learn a lot.

Thankfully, I won’t be alone – the team I’m joining is also full of talented subject matter experts – but I will need to stand on my own two feet.

That’s why I thought I’d write this post, with a few notes that are based on a recent conversation when my former colleague Mark Bodley checked over the Exchange calculations I’d used to create a guide price for a customer solution… [I’ve since made a couple of extra edits based on advice from Fujitsu’s resident Exchange Master, Nick Parlow]

Inputs

To get started, some educated guesses can be made to adjust the defaults:

• Exchange environment configuration:
  • 64-bit GCs, multi-role servers, non-virtualised, HA deployment (in line with our design principles).
  • Set number of servers per DAG and number of DAGs according to size of solution. Start DAG count low to keep hardware down – revise up later if required based on guidance elsewhere in the tool.
• Site resilience configuration:
  • Single active/active resilient DAG (i.e. no passive servers).
  • Watch the RPO value – it won’t affect the server count but could be a factor in inter-datacentre bandwidth calculations.
• Mailbox database copy configuration:
  • Increased HA database copies to 4; decreased lagged copies to 0 (lagged copies require more storage as transaction logs are retained for longer).
  • Increased number of HA database copies in secondary datacentre to account for non-lagged copies (in this case I had a 50/50 split between DCs).
• Lagged database copy configuration: not used.
• Exchange data configuration: left at defaults.
• Database configuration: left at defaults.
• Exchange I/O configuration: left at defaults
• Transport configuration:
  • Set Safety Net expiration to 2 days (unless using lagged copies).
• User mailbox configuration:
  • One tier for each logical group of users.
  • Days in work week set to 5 – unless there are significant groups of people working 7 days a week (affects log replication).
  • Mailbox configuration values will depend on organisation but the advice given for a starting point was to set:
    • Total send/receive capacity per mailbox per day to 100 (down from 200) messages.
    • Average messages size to 150KB (affects storage).
    • Mailbox size limits are used for capacity growth planning – split between main and personal archive should not affect the calculations.
    • Deleted item retention window may need to be increased to meet requirements (14 days is low if not using backups).
    • Multiplication factors come into play with mobile devices.
    • Desktop search engine value is more significant in Citrix environments than with Outlook cached mode clients.
• Backup configuration:
  • Assumed software VSS via SCDPM, daily full backups.
  • Calculator will take highest value of backup/truncation failure tolerance and network failure tolerance (so only one of these really matters!).
• Storage options:
  • JBOD with multiple databases per disk (in line with our design principles).
  • Main thing to watch here is the number of auto reseed volumes per server – align to failure rate of the disks.
• Disk configuration: will depend on servers in use. For reference, with our Fujitsu PRIMERGY servers, the available options led us to:
  • 900GB system disks (10K RPM SAS 2.5″).
  • 4000GB database plus log disks (7.2K RPM SATA 3.5″).
  • 4000GB restore volume (7.2K RPM SATA 3.5″).
• Processor configuration:
  • Core/server is of secondary interest but helps the calculator to advise on Global Catalog cores.
  • SPECint2006 value set to the lowest that will allow a suitable server utilisation number. In practice there will be a balance between servers/processor options and price. It’s useful to have a lookup of SPECint2006 values for a range of servers (e.g. Fujitsu PRIMERGY RX300) or you can search the SPECint site and take the value from the result column.
•   Log replication configuration:
  • Will vary according to environment (and international spread of users) but rule of thumb used was (this should add up to 100%):
    • 1: 0.33%.
    • 2: 0.24%.
    • 3: 0.24%.
    • 4: 0.2%.
    • 5: 0.82% (I’m not sure why we have a night-time peak – automated emails sent overnight? Or this could be spread across others…)
    • 6: 0.31%.
    • 7: 0.34%.
    • 8: 1.46%.
    • 9: 6.46%.
    • 10: 10.08%.
    • 11: 10.55%.
    • 12: 11.06%.
    • 13: 9.48%.
    • 14: 9.61%.
    • 15: 10.52%.
    • 16: 10.41%.
    • 17: 8.31%.
    • 18: 5.07%.
    • 19: 1.94%.
    • 20: 0.95%.
    • 21: 0.84% (I might be tempted to up this slightly for the evening email check in many organisations…).
    • 22: 0.36%.
    • 23: 0.21%.
    • 24: 0.21%.
  • Network configuration: latency will depend on available datacentre connectivity.
• Environment customisation: only used to generate naming configuration.

Check the results

With the main inputs in place, some fine tuning is probably required:

• Role requirements:
  • Watch out for errors (e.g. over-utilised servers).  This is where the server count may need to be adjusted, or a higher-specification server used (SPECint2006 values). Ideally, servers should be close to 80% utilised [in a failure scenario] but not over. Getting the right CPU is more critical than memory as RAM can normally be added later! [also bear in mind the impact of other software on utilisation – for example file level antivirus, and any message hygiene software running on the box.]
  • Also check the recommended transport database location (we’ll come back to that in a moment).
• Volume requirements:
  • Watch out for disk sizes/configurations that can’t be met!  If the solution is storage-bound, then it may be necessary to add additional servers (we use direct attached storage as a design principle) or, if available, then larger commodity disks would help. One clue to a potential issue is if the DB and log volume design/server table has DBx as a database copy name, rather than DB1-DB4, etc.
• Storage design: confirms (or otherwise) that the solution can be deployed in a JBOD configuration (as per our design principles, although RAID is also an option).
  • This worksheet will also give the number of disks required in each server. Note that this is the count for database and log disks, restore volumes and auto reseed volumes – don’t forget the system disks for operating system/application binaries… and then think about the transport database.
  • In my solution, the recommended transport database location (from the role requirements) was the system disk but we were using 900GB 10K RPM SAS 3.5″ disks. With OS (150GB), Anchor LUN (2GB), Exchange binaries (50GB), and consideration for Exchange Logs (maybe 600GB) those disks are already pretty full, so it’s worth considering an extra RAID 1 volume for the transport database (over-ruling the calculator) and possibly hot spares for the RAID volumes too (depending on your attitude to risk).

A couple of other points to consider: public folders and unified messaging. I didn’t need public folders but don’t overlook them in your plans if they are in use.  As for unified messaging, it’s no longer a server role so is included in the calculations, up to a point – UM is likely to hit the CPU load, so it might be prudent to factor in some additional headroom with the SPECint2006 of the servers (to keep the utilisation well below the 80% mark).

YMMV

Of course, there are just my notes on some of the things I checked to fit our reference design and they are not exhaustive. As the saying goes, your mileage may vary. For reference, I was using Exchange 2013 Server Role Requirements Calculator v6.6 but you should always use the latest available version and there is a very useful readme file which should be referenced when working with the calculator. You should also factor in this guidance on sizing Exchange 2013 deployments and consider the Exchange 2013 performance recommendations too.

[Edited 16 May 2015 to clarify comments re: server utilisation]