SharePoint: High Availability

Recently, I have been asked to develop a plan to make SharePoint 2010 farm Highly Available.
This short post will outline the main phases that farm needs to undergo in order to be Highly Available.

In my case, we have 2 datacenters. One of them is Primary, and another - Secondary. In case primary goes down, we should provide a seamless switch to the Secondary without loss functionality and preferably performance. 


In my personal opinion, I prefer to give performance boost in case I have resources, even it might mean that end-users will notice some performance degradation in case of failover.

Here is a schema to implement for High Availability:

Here are key notes:

1. Web Front Ends (WFEs) - When Primary is up, all 4 WFEs (primary and secondary) are serving the requests. In case the Primary is out, the Secondary WFEs will be getting all requests.
Here I see, that it may impact performance, since in the usual scenario end-users uses all 4 WFE (that load-balanced via ISA).
One of the option to keep WFE performance steady, is to keep only 2 WFEs available in case Primary is up.
The seamless switch will be provided via ISA

2. SharePoint application server - both of them engaged in case Primary is up. In case of failover, the second application server should have the same services running as in Primary to maintain the same functionality as before. 
The weakest point is a timer job. I can imagine that in some scenarios WFs that been served by primary server at the time of failover will never get back.
One more note on this- plan your search architecture: Search Service Application: Architecture in one page


3. SSRS servers - in case Primary is active, engage both. In case of failover the second SSRS will get all requests. The seamless switch will be provided via ISA


4. Often enterprise SharePoint solution interact with external system via BDC (Trying to figure out what's the difference between 2007 BDC and 2010 BCS?)
We need to plan how we can provide access to these external system in case of failover.
That means extensive communication with teams who support such system.
In my example, we accumulate all external systems calls via web services developed on BizTalk server. From our side, we need to configure ISA to have additional BizTalk availabled on the Secondary datacenter.


5.On SQL side - we are implementing async mirroring. In case primary goes down, we don't have any loss and ready to switch to the Secondary.
I prefer to have a witness on the Secondary based on assumption that we use Secondary in case Primary is down, not vice versa.

In case report db servers fail in the Primary, we need to have extra work from front-ent side. Report connection files have connection information inside them. We need to make sure that all connection files have alias name instead of the actual. In the case of failover, we just modifing alias on Report server side. Keep in the mind that SSAS type of connection file won't work with SQL alias and we can do it via host file.

We need to setup failover settings for SharePoint databases.
Refer to this post for what can be mirrored in SharePoint 2010:
The 2010 SharePoint databases, purposes and mirroring supportability
You will find that some of the SharePoint DBs are not required and are not design to setup for failover since they are not critical and easy to re-create.
Based on what recover time you have , adjust for yourself what needs to be configured with failover and what can be omitted. If a SharePoint db is not configured with failover, plan ahead what necessarily actions you should pefrom to put the bd back in case of failure, and what impact will be if the db is not ready right away. As an example, most likely the StateService will not be in high demand right after switching to the reserved (Secondary) datacenter.


Here is an outline plan how to introduce HA in SharePoint 2010 farm gradually:


1. Decide how many additional servers are needed, and their configuration. 
2. WFE. Test first without including in ISA. Then include in the prod farm.
3. App server. Build the server cautiously. I believe once the app server is joined, it will used by prod WFE. No ISA configuration is needed. All requests goes from the SharePoint farm configuration directly.
4. Work on external systems HA
5. Work on SQL servers, failover settings on SharePoint side 
6. SSRS servers

Wish you happy HA to you.

SharePoint: SharePoint 2007 Administration and PowerShell 2.0

Last months my work with SharePoint 2007, I felt a strong inclination to use PowerShell 2.0.
I envied a happy people who were already with SharePoint 2010 and could enjoy SharePoint 2010 Administration via SharePoint.ps1


PowerShell and SharePoint: What, Why and How
Simple concept: How to use SharePoint cmdlets in PowerShell ISE


I also wanted to have such beautiful commands that have been shipped by SharePoint.ps1 for SharePoint 2010:
Get-SPFarm,
Get-SPWeb,
Get-SPWebApplication


So, my problem was that we don't have SharePoint.ps1 for SharePoint 2007.
To easy up my desire, I have created my functions:
Get-SPFarmV3,
Get-SPWebV3,
Get-SPWebApplicationV3.


And some additional:
Get-SitesWithMissingTemplate, Get-SSProvider, Get-SPVersionV3.


All above I have packed in one SPv3Adapter.mdl module.


Windows PowerShell Module Concepts


Module Installation Best Practices:


Do not install modules for Windows PowerShell in the system location at %Windir%\System32\WindowsPowerShell\v1.0\Modules. Only modules included with Windows are installed to the system location.


http://msdn.microsoft.com/en-us/library/dd878350(v=vs.85).aspx


Module installation
http://msdn.microsoft.com/en-us/library/dd878350(v=vs.85).aspx




And some additional PowerShell files:
Helper functions: Start-CustomTranscript, Get-CustomAPPLog,Get-SolutionDeployed


I have used a lot Get-SolutionDeployed during upgrade SP2007 to SP2010:


The module Helper.mdl has dependency on SPv3Adapter.mdl

Please install  SPv3Adapter.mdl  module first in order to enjoy Sp2007 helper functions.

One of the favorable option for me is to use manifest (you are welcome to use attached customModulesLoader.psm1 and  Manifest.mdl)  to ship the PS modules.

SharePoint: Is there life after development? Transition plan for SharePoint support team

Finally, my 2 year project on SharePoint 2007\2010 has been finished!

We started with 9 brave people in the team.
Our tasks were:
1. Build SOA infrastructure;
2. Bring data from disparate data sources into one application for measurement and management;
3. Build SharePoint farm;
4. Organize development and deployment process;
5. Develop custom business solutions based on SharePoint technology;
6. Build a report system to make use of the data in SharePoint;
It took us roughly 8 months of initial and hectic work.

And in another 1 year we have been asked (with 7 people):
 7. Extend functionality;
 8. Maintenance;
6.  Develop mobile solution to work with custom SharePoint sites;
7.  Upgrade 2007 to 2010.

Finished all. Now, it's time to transition support to the support team.
So, here is my a suggested transition plan for SharePoint support team.


The plan should be outlined by a person who holds knowledge. But the content should be filled by support team.  Also support team needs to get an estimate for every phase, the estimate should be recommended by a knowledgeable person.
They need to collaborate with developers to get the document done. 
Once the document is ready, a team lead\ or some other guy who knows the picture, needs to verify the document and to correct if necessarily.

Here are steps for transition plan:

1. Knowledge holders 
A main holder can be a team lead in the project. He should provide contact information to support team.
The contact info should contain all major areas that is needed to fully support the SharePoint farm and a responsible\ knowledgeable person in this area.

Here is an example of areas:
SharePoint Developer
SharePoint Architect
External system (BDC) administrators
DBA
Report developer
AD Administrator
ISA Administrator
SharePoint trainers

2. Environment  awareness
Support team should become aware:
 - what environments were build (ex: Dev, QC, Training, Staging , Prod);
 - the purpose of the environment;
 - the environment topology (how many servers with what roles, load balancing (ex. ISA) configuration)

The access plan should be developed: when to ask access to where. 

It will be milestones for the support team get them up to speed.

3.  Code awareness
- what the custom solutions and where they are;
- solution dependencies;
- how to build and deploy;

4. Development and deployment approaches
- is it possible go with no-downtime deployment and how (SharePoint : Farm solution deployment with no downtime. Update-SPSolution -local)


Approaches to deploy:
Content-based and code-based deployment, hybrid ( refer to previous posts):
When it makes sense to use:
- Sandbox solution - SharePoint 2010: Sandboxed solution restrictions and considerations
- Hybrid Sandbox solution ;
- Farm solution;
- Content deployment

Each of these has own appropriate context and their advantages and disadvantages.

Note: This step is  for education purposes. You never know what knowledge a new member in SharePoint support team has.


5. Troubleshooting : common areas
As an example for our farm it is:
- SSRS
- External system check ( BDCM)

6. Backup and restore strategy


Happy supporting

SharePoint : Farm solution deployment with no downtime. Update-SPSolution -local

Wondering if it's possible to re-deploy farm solution wsp file without downtime? 

In some circumstances, it's possible.


You can do no-downtime deployment if following is true:
 - Your modified wsp file doesn't contain new files or features. The most common scenario, you want to change some code logic that is an existing dll. Once you deploy your modified wsp, the new dll will be placed in GAC (in case you don't have versioning enabled).


- You have load-balanced WFEs. (ISA with SharePoint Farm ( a little bit more complicated scenario with off-box SSL termination))

Here is how it works:

1. ISA test.

Test first if site is up with one server drained. Test for all severs in the ISA farm object for portal.

2. ISA switch.

Drain the first WFE.

3. WFE work. Update-SPSolution

Copy new wsp file on the local drive.

Run

Update-SPSolution -Identity {name}.wsp -LiteralPath "{path}\{name}.wsp" -Local –GACDeployment

Make sure that you specify the parameter –local.

It will deploy files locally and locally restart IIS.

After such update you will see following:

I have 2 boxes in QC env: SOA-MOSS01-QC, SOA-MOSS02-QC.
In the last operation result I see that operation is been performed only on one box SOA-MOSS01-QC.
The second box haven’t been updated yet and at this time actively  serving user requests with old functionality in place.

NOTE:

The Update-SPSolution cmdlet upgrades a deployed SharePoint solution in the farm. Use this cmdlet only if a new solution contains the same set of files and features as the deployed solution. If files and features are different, the solution must be retracted and redeployed by using the Uninstall-SPSolution and Install-SPSolution cmdlets, respectively.

Uninstall-SPSolution also has –local parameter.

But, The Install-SPSolution cmdlet deploys an installed SharePoint solution in the farm. Use the Add-SPSolution cmdlet to install a SharePoint solution package io the farm.

I haven’t tested this option.  And I can see a caveat here, if we need to add-spsolution , we need to remove-spsolution first. And this command doesn’t have –local option. Most likely in the scenario when we need to re-install spsolution instead of update, we have to bring the portal down.

4. Test the result locally on that drained and already updated WFE.

5. ISA Switch

Re-switch the servers.

6. Repeat steps from 3-4.

Done.

P.S.

That is how it looks from an end-user:

He is still accessing the portal while the specific WFE IIS is down.

Simple Concept: How to export wsp file?

[Question]: How can I export wsp file from farm solution?

[Answer]:
User PowerShell script:
PowerShell Script for SP2010 to export farm solution file

Networking: essential commands

I am getting deeper in networking while building topology for SharePoint 2010.
Here is a quick reminder for everyone who is not a network savy but has to survive.


The following list of the network commands will help to understand where you machine is and how it's routed.
So, run MS-Dos window and enjoy:


1. notorious   - ping - sends out a packet to a designated internet host or network computer and measures its response time. The target computer will return (hopefully) a signal. 


2.  ipconfig  (add /all to get a full picture) -  It is used to display the TCP/IP network configuration values (assigned IP, gateway and DNS server)

3.  tracert - The actual path between two computers on the Internet is not a straight line but consists of numerous segments or "hops" from one intermediate computer to another. Tracert shows each step of the path taken. 

4. pathping -   this command combines functions of Ping and Tracert. Pathping will first list the number of hops required to reach the address you are testing and then send multiple pings to each router between you and the destination. 

5. netstat - displays the active TCP connections and ports on which the computer is listening, Ethernet statistics, the IP routing table, statistics for the IP, ICMP, TCP, and UDP protocols. 

The command "netstat -a" will display all your connections. The command "netstat -b" will show the executable files involved in creating a connection. A figure showing all the switches and syntax is given below

5. nslookup - This command helps diagnose the Domain Name System (DNS) infrastructure.

For more detailed info, TCP/IP and Networking Tools

The 2010 SharePoint databases, purposes and mirroring supportability

I have scrutinized the article Database types and descriptions (SharePoint Server 2010) and here is the result - a table which gives an overall understanding what datatables SharePoint 2010 uses and what for, plus their ability to be mirrored.
* Url to set Failover field sometimes is missing since I haven't tested by myself
*** "No??" Indicates the msdn documentation gives "No", but in CA UI I see an option to set Failover
***The easy way to set failover:
1. Make sure that you have configured a mirror
2. Run the script
2.1. suitable  in case that all your mirrors on the one box:

Param([string]$FailoverInstance = $(Read-Host "Enter the Mirror (Partner) SQL Instance(like server\instance)")) 
get-spserviceinstance -all | foreach-object { 
    if ($_.typeName -eq "Microsoft SharePoint Foundation Database") { 
        foreach ($Database in $_.Databases){ 
             
            write-host "Updating FailoverPartner on" $Database.Name "to" $FailoverInstance 
            $Database.AddFailoverServiceInstance($FailoverInstance) 
            $Database.Update() 
            write-host "Successfully Updated Failover Partner on" $Database.Name "to" $FailoverInstance 
        } 
         
    } 
}

2.2.
For individual database, run this simple script.
 $db = get-spdatabase | where {$_.Name -eq "db_name"}   
 $db.AddFailoverServiceInstance("db02");
 $db.Update() 
****In case of the issue with Failover, try to find an answer from SharePoint 2010 Failover doesn't work

	Default Database name	Purpose	Mirror supportability	Default recovery  mode	Url to set Failover
1	SharePoint_Config	contains data about SharePoint databases, Internet Information Services (IIS) Web sites, Web applications, trusted solutions, Web Part packages, site templates, and Web application and farm settings specific to SharePoint 2010 Products, such as default quota settings and blocked file types.	Yes	Full	PowerShell
2	SharePoint_AdminContent	It stores all site content, including site documents or files in document libraries, list data, and Web Part properties, in addition to user names and rights for the Central Administration site collection	Yes	Full	PowerShell
3	WSS_Content	Content databases store all content for a site collection, including site documents or files in document libraries, list data, Web Part properties, audit logs, and sandboxed solutions, in addition to user names and rights.	Yes	Full	_admin/CNTDBADM.aspx
4	WSS_UsageApplication	It stores health monitoring and usage data temporarily, and can be used for reporting and diagnostics.	Yes, but not recommended. Can be easily recreated in the case of failure	Simple	PowerShell
5	Bdc_Service_DB_	stores external content types and related objects	Yes	Full	_admin/ServiceApplications.aspx
6	Application_Registry_server_DB_	stores backward-compatible information that is used to connect to information that is used by the Microsoft Office SharePoint Server 2007 Business Data Catalog API.	Yes	Full	PowerShell
7	SubscriptionSettings_	stores features and settings for hosted customers	Yes	Full
8	Secure_Store_Service_DB_	stores and maps credentials, such as account names and passwords	Yes	Full	_admin/ServiceApplications.aspx
9	StateService	stores temporary state information for InfoPath Forms Services, the chart Web Part, and Visio Services.	Yes	Full	PowerShell
10	WebAnalyticsServiceApplication_StagingDB_	temporarily stores un-aggregated fact data, asset metadata, and queued batch data for the Web Analytics service application.	No	Full
11	WebAnalyticsServiceApplication_ReportingDB_	stores aggregated standard report tables, fact data aggregated by groups of sites, date and asset metadata, and diagnostics information for the Web Analytics service application.	Yes	Full	PowerShell
12	Search_Service_Application_DB_	hosts the Search service application configuration and access control list (ACL), and best bets for the crawl component. This database is accessed for every user and administrative action	Yes	Simple	searchfarmdashboard.aspx
13	Search_Service_Application_CrawlStoreDB_	stores the state of the crawled data and the crawl history	Yes	Simple	searchfarmdashboard.aspx
14	Search_Service_Application_PropertyStoreDB_	stores information that is associated with the crawled data, including properties, history, and crawl queues	Yes	Simple	searchfarmdashboard.aspx
15	User Profile Service Application_ProfileDB_	stores and manages users and associated information. It also stores information about a user's social network in addition to memberships in distribution lists and sites.	Yes	Simple	_admin/ServiceApplications.aspx
16	User Profile Service Application_SyncDB_	stores configuration and staging data for use when profile data is being synchronized with directory services such as Active Directory	No??	Simple	_admin/ServiceApplications.aspx
17	User Profile Service Application_SocialDB_	stores social tags and notes created by users, along with their respective URLs	Yes	Simple	_admin/ServiceApplications.aspx
18	Managed Metadata Service_	stores managed metadata and syndicated content types	Yes	Full	_admin/ServiceApplications.aspx
19	WordAutomationServices_	stores information about pending and completed document conversions	Yes	Full
20	PerformancePoint Service Application_	stores temporary objects, persisted filter values, and user comments	Yes	Full	_admin/ServiceApplications.aspx
21	ProjectServer_Draft	contains data for editing projects. This database also hosts the tables used by the Project Queue. Data in the Draft database is not directly accessible by end users.	Yes, but you must follow additional steps.	Full
22	ProjectServer_Published	contains a copy of all of the projects that have been published. The Published database also contains tables that are specific to Project Server (timesheets, resources, custom fields, security definitions, and other metadata). This database also hosts the tables used by the Timesheet Queue. Data in the Published database is not directly accessible by end user	Yes, but you must follow additional steps.	Full
23	ProjectServer_Archive_	stores the backup data of projects, resources, calendars, enterprise custom fields, the enterprise global Project Web Access view definitions, Project Web Access system settings, and category and group security settings as set up by the Project Web Access administrator. Data in the Archive database is not directly accessible by end users.	Yes, but you must follow additional steps.	Full
24	ProjectServer_Reporting	is the repository for the entire portfolio of projects in Project Server. These tables present stable snapshots of each project plan based on the last time a project was published to Project Server and include de-normalized time phased data, allowing for advanced reporting capabilities outside of the Project client. Data in the Reporting database is accessible by end users.	Yes, but you must follow additional steps.	Full
25	FASTSearchAdminDatabase	stores and manages data related to administration of FAST Search Server 2010 for SharePoint. This includes search setting groups, keywords, synonyms, document and site promotions and demotions, term entity extractor inclusions and exclusions, spell check exclusions, best bets, visual best bets, and search schema metadata.	No	Full
26	DefaultPowerPivotServiceApplicationDB	stores the location of cached or loaded PowerPivot data files, data refresh schedules, and PowerPivot usage data that is copied from the central usage data collection database.	Yes, but mirroring is managed solely through SQL Server: SharePoint Server 2010 is not aware of the PowerPivot database.	Full
27	RSDB	stores all report metadata including report definitions, report history and snapshots, and scheduling information.	No	Full
28	RSTempDB	stores all the temporary snapshots while reports are running.	No	Full