SharePoint: High Availability

Recently, I have been asked to develop a plan to make SharePoint 2010 farm Highly Available.
This short post will outline the main phases that farm needs to undergo in order to be Highly Available.

In my case, we have 2 datacenters. One of them is Primary, and another - Secondary. In case primary goes down, we should provide a seamless switch to the Secondary without loss functionality and preferably performance. 


In my personal opinion, I prefer to give performance boost in case I have resources, even it might mean that end-users will notice some performance degradation in case of failover.

Here is a schema to implement for High Availability:

Here are key notes:

1. Web Front Ends (WFEs) - When Primary is up, all 4 WFEs (primary and secondary) are serving the requests. In case the Primary is out, the Secondary WFEs will be getting all requests.
Here I see, that it may impact performance, since in the usual scenario end-users uses all 4 WFE (that load-balanced via ISA).
One of the option to keep WFE performance steady, is to keep only 2 WFEs available in case Primary is up.
The seamless switch will be provided via ISA

2. SharePoint application server - both of them engaged in case Primary is up. In case of failover, the second application server should have the same services running as in Primary to maintain the same functionality as before. 
The weakest point is a timer job. I can imagine that in some scenarios WFs that been served by primary server at the time of failover will never get back.
One more note on this- plan your search architecture: Search Service Application: Architecture in one page


3. SSRS servers - in case Primary is active, engage both. In case of failover the second SSRS will get all requests. The seamless switch will be provided via ISA


4. Often enterprise SharePoint solution interact with external system via BDC (Trying to figure out what's the difference between 2007 BDC and 2010 BCS?)
We need to plan how we can provide access to these external system in case of failover.
That means extensive communication with teams who support such system.
In my example, we accumulate all external systems calls via web services developed on BizTalk server. From our side, we need to configure ISA to have additional BizTalk availabled on the Secondary datacenter.


5.On SQL side - we are implementing async mirroring. In case primary goes down, we don't have any loss and ready to switch to the Secondary.
I prefer to have a witness on the Secondary based on assumption that we use Secondary in case Primary is down, not vice versa.

In case report db servers fail in the Primary, we need to have extra work from front-ent side. Report connection files have connection information inside them. We need to make sure that all connection files have alias name instead of the actual. In the case of failover, we just modifing alias on Report server side. Keep in the mind that SSAS type of connection file won't work with SQL alias and we can do it via host file.

We need to setup failover settings for SharePoint databases.
Refer to this post for what can be mirrored in SharePoint 2010:
The 2010 SharePoint databases, purposes and mirroring supportability
You will find that some of the SharePoint DBs are not required and are not design to setup for failover since they are not critical and easy to re-create.
Based on what recover time you have , adjust for yourself what needs to be configured with failover and what can be omitted. If a SharePoint db is not configured with failover, plan ahead what necessarily actions you should pefrom to put the bd back in case of failure, and what impact will be if the db is not ready right away. As an example, most likely the StateService will not be in high demand right after switching to the reserved (Secondary) datacenter.


Here is an outline plan how to introduce HA in SharePoint 2010 farm gradually:


1. Decide how many additional servers are needed, and their configuration. 
2. WFE. Test first without including in ISA. Then include in the prod farm.
3. App server. Build the server cautiously. I believe once the app server is joined, it will used by prod WFE. No ISA configuration is needed. All requests goes from the SharePoint farm configuration directly.
4. Work on external systems HA
5. Work on SQL servers, failover settings on SharePoint side 
6. SSRS servers

Wish you happy HA to you.

SSRS: Performance

SSRS Add-in installation:

http://technet.microsoft.com/en-us/library/aa905871(v=sql.100).aspx

SSRS Performance Optimizations:

http://sqlcat.com/sqlcat/b/technicalnotes/archive/2009/01/14/reporting-services-performance-optimizations.aspx

Monitoring SSRS Performance:

http://www.pertell.com/sqlservings/archive/2011/11/monitoring-ssrs-report-performance-part-4

SSRS Memory Pressure zone:

http://www.bidn.com/blogs/JeffRush/ssas/1330/ssrs-memory-pressure-zones

SSRS HTTP Errors:

http://msdn.microsoft.com/en-US/library/ms159778(v=sql.100).aspx

SSRS Authentication (why we do need just RSNegotiateNTLM)

http://msdn.microsoft.com/en-US/library/cc281253(v=sql.100).aspx

SQL Server 2008 Reporting Services for high-availability deployment

http://searchsystemschannel.techtarget.com/feature/SQL-Server-2008-Reporting-Services-for-high-availability-deployment 

The 2010 SharePoint databases, purposes and mirroring supportability

I have scrutinized the article Database types and descriptions (SharePoint Server 2010) and here is the result - a table which gives an overall understanding what datatables SharePoint 2010 uses and what for, plus their ability to be mirrored.
* Url to set Failover field sometimes is missing since I haven't tested by myself
*** "No??" Indicates the msdn documentation gives "No", but in CA UI I see an option to set Failover
***The easy way to set failover:
1. Make sure that you have configured a mirror
2. Run the script
2.1. suitable  in case that all your mirrors on the one box:

Param([string]$FailoverInstance = $(Read-Host "Enter the Mirror (Partner) SQL Instance(like server\instance)")) 
get-spserviceinstance -all | foreach-object { 
    if ($_.typeName -eq "Microsoft SharePoint Foundation Database") { 
        foreach ($Database in $_.Databases){ 
             
            write-host "Updating FailoverPartner on" $Database.Name "to" $FailoverInstance 
            $Database.AddFailoverServiceInstance($FailoverInstance) 
            $Database.Update() 
            write-host "Successfully Updated Failover Partner on" $Database.Name "to" $FailoverInstance 
        } 
         
    } 
}

2.2.
For individual database, run this simple script.
 $db = get-spdatabase | where {$_.Name -eq "db_name"}   
 $db.AddFailoverServiceInstance("db02");
 $db.Update() 
****In case of the issue with Failover, try to find an answer from SharePoint 2010 Failover doesn't work

	Default Database name	Purpose	Mirror supportability	Default recovery  mode	Url to set Failover
1	SharePoint_Config	contains data about SharePoint databases, Internet Information Services (IIS) Web sites, Web applications, trusted solutions, Web Part packages, site templates, and Web application and farm settings specific to SharePoint 2010 Products, such as default quota settings and blocked file types.	Yes	Full	PowerShell
2	SharePoint_AdminContent	It stores all site content, including site documents or files in document libraries, list data, and Web Part properties, in addition to user names and rights for the Central Administration site collection	Yes	Full	PowerShell
3	WSS_Content	Content databases store all content for a site collection, including site documents or files in document libraries, list data, Web Part properties, audit logs, and sandboxed solutions, in addition to user names and rights.	Yes	Full	_admin/CNTDBADM.aspx
4	WSS_UsageApplication	It stores health monitoring and usage data temporarily, and can be used for reporting and diagnostics.	Yes, but not recommended. Can be easily recreated in the case of failure	Simple	PowerShell
5	Bdc_Service_DB_	stores external content types and related objects	Yes	Full	_admin/ServiceApplications.aspx
6	Application_Registry_server_DB_	stores backward-compatible information that is used to connect to information that is used by the Microsoft Office SharePoint Server 2007 Business Data Catalog API.	Yes	Full	PowerShell
7	SubscriptionSettings_	stores features and settings for hosted customers	Yes	Full
8	Secure_Store_Service_DB_	stores and maps credentials, such as account names and passwords	Yes	Full	_admin/ServiceApplications.aspx
9	StateService	stores temporary state information for InfoPath Forms Services, the chart Web Part, and Visio Services.	Yes	Full	PowerShell
10	WebAnalyticsServiceApplication_StagingDB_	temporarily stores un-aggregated fact data, asset metadata, and queued batch data for the Web Analytics service application.	No	Full
11	WebAnalyticsServiceApplication_ReportingDB_	stores aggregated standard report tables, fact data aggregated by groups of sites, date and asset metadata, and diagnostics information for the Web Analytics service application.	Yes	Full	PowerShell
12	Search_Service_Application_DB_	hosts the Search service application configuration and access control list (ACL), and best bets for the crawl component. This database is accessed for every user and administrative action	Yes	Simple	searchfarmdashboard.aspx
13	Search_Service_Application_CrawlStoreDB_	stores the state of the crawled data and the crawl history	Yes	Simple	searchfarmdashboard.aspx
14	Search_Service_Application_PropertyStoreDB_	stores information that is associated with the crawled data, including properties, history, and crawl queues	Yes	Simple	searchfarmdashboard.aspx
15	User Profile Service Application_ProfileDB_	stores and manages users and associated information. It also stores information about a user's social network in addition to memberships in distribution lists and sites.	Yes	Simple	_admin/ServiceApplications.aspx
16	User Profile Service Application_SyncDB_	stores configuration and staging data for use when profile data is being synchronized with directory services such as Active Directory	No??	Simple	_admin/ServiceApplications.aspx
17	User Profile Service Application_SocialDB_	stores social tags and notes created by users, along with their respective URLs	Yes	Simple	_admin/ServiceApplications.aspx
18	Managed Metadata Service_	stores managed metadata and syndicated content types	Yes	Full	_admin/ServiceApplications.aspx
19	WordAutomationServices_	stores information about pending and completed document conversions	Yes	Full
20	PerformancePoint Service Application_	stores temporary objects, persisted filter values, and user comments	Yes	Full	_admin/ServiceApplications.aspx
21	ProjectServer_Draft	contains data for editing projects. This database also hosts the tables used by the Project Queue. Data in the Draft database is not directly accessible by end users.	Yes, but you must follow additional steps.	Full
22	ProjectServer_Published	contains a copy of all of the projects that have been published. The Published database also contains tables that are specific to Project Server (timesheets, resources, custom fields, security definitions, and other metadata). This database also hosts the tables used by the Timesheet Queue. Data in the Published database is not directly accessible by end user	Yes, but you must follow additional steps.	Full
23	ProjectServer_Archive_	stores the backup data of projects, resources, calendars, enterprise custom fields, the enterprise global Project Web Access view definitions, Project Web Access system settings, and category and group security settings as set up by the Project Web Access administrator. Data in the Archive database is not directly accessible by end users.	Yes, but you must follow additional steps.	Full
24	ProjectServer_Reporting	is the repository for the entire portfolio of projects in Project Server. These tables present stable snapshots of each project plan based on the last time a project was published to Project Server and include de-normalized time phased data, allowing for advanced reporting capabilities outside of the Project client. Data in the Reporting database is accessible by end users.	Yes, but you must follow additional steps.	Full
25	FASTSearchAdminDatabase	stores and manages data related to administration of FAST Search Server 2010 for SharePoint. This includes search setting groups, keywords, synonyms, document and site promotions and demotions, term entity extractor inclusions and exclusions, spell check exclusions, best bets, visual best bets, and search schema metadata.	No	Full
26	DefaultPowerPivotServiceApplicationDB	stores the location of cached or loaded PowerPivot data files, data refresh schedules, and PowerPivot usage data that is copied from the central usage data collection database.	Yes, but mirroring is managed solely through SQL Server: SharePoint Server 2010 is not aware of the PowerPivot database.	Full
27	RSDB	stores all report metadata including report definitions, report history and snapshots, and scheduling information.	No	Full
28	RSTempDB	stores all the temporary snapshots while reports are running.	No	Full