SharePoint 2010 - Crawling stuck, crawl component in "Recovering" state

From past few days we started this issue out of now where. The crawler is stuck. When we browse to the Search Administration page, we see that the Crawler component is stuck in "Recovering" state and due to that Query component also reported a status as "Not Responding". Propogation status was shown as "Query server not responding"

We checked the SharePoint logs and found errors as below

file copy failed (error 0x80070005: Access is denied.   0x80070005, source c:\program files\microsoft office servers\14.0\data\office server\applications\GUID-crawl-0\projects\portal_content\indexer\cifiles\0001001A.ci, destination \\machinename\GUID-query-0\GUID-query-0\Projects\Portal_Content\Indexer\CiFiles\0000.0001001A.ci.cp)  [indexpropagator.cxx:403]  d:\office\source\search\native\ytrip\tripoli\propagation\indexpropagator.cxx

Exception thrown (0x80070005: [needs message] info 0)                           [indexpropagator.cxx:409]  d:\office\source\search\native\ytrip\tripoli\propagation\indexpropagator.cxx

In Event Viewer, we found Event ID: 2587 with the description that "The following conditions are currently affecting index propogation to this server for search service application "Service Application Name": 1. Query 0 has been disabled so that crawls can continue. It may be recovered via the Restart-SPEnterpriseSearchQueryComponent command in PowerShell

The code - 0x80070005 states that the access was denied and the account accessing the folder did not have permissions.

I granted permissions for the service and crawler account on c:\program files\microsoft office servers\14.0\data\office server\applications\
But this did not resolve the issue. This meant that apart from the location mentioned in the logs, the permission issue was somewhere else too. In such cases, process monitor does help. I took a process monitor and found that there were errors related to permissions on C:\ProgramData and few of its SharePoint subfolders. After granting the service and crawler account permission (Full Control) over the C:\ProgramData folder, the status of the components was not showing as "Online".

In case you face such errors, try giving permissions for the search service and crawler account (in case they are different) over the C:\ProgramData folder and check. If not, take help from Process Monitor to find out where the permission issue is.

Do not remove, add or restart any component unless you are absolutely sure that permissions or any of the below mentioned causes is not an issue.

Few other things to verify:

• Ensure that the search service is started. Open service manager (Start -> Run -> services.msc) and ensure that "SharePoint Server Search 14" is started.
• Re-enter the password for the service account for the service which you checked in the above step.
• Ensure that the search service instance is online by checking the same by using a powershell/stsadm command
• stsadm -o osearch -action list  This will show if the search instance is online or not on the servers.
• http://blogs.technet.com/b/patrick_heyde/archive/2009/12/07/sharepoint-2010-get-search-configuration-over-powershell.aspx

Update: Recently we noticed that restarting the server running the component would get the component back to "Online" state. Before restarting any services or running PowerShell cmdlet, try rebooting the server.