Auditing mailbox actions for security and accountability

I was facing with a challenge. A company VIP with a few secretaries, and a few mobile devices was facing a serious issue. Calendar items were changing by themselves. Updates were sent to attendees but something changed the meeting date / time on the owner mailbox.
I was asked to find out what is the root cause.

In order to do that, there are two tools:

1. The Microsoft Exchange Troubleshooting Assistant (Extra.exe) which will allow you to gather the information, but you will have to open a case with MS in order to get the results.
   The process requires to start the tool on the mailbox server of the effected user and target the user mailbox to perform a trace up until the issue occurs
   .
2. Enable Mailbox Audit on the Exchange Server (2010)
   
   This will be the path we will investigate
   
   Enabling mailbox auditing is something that you can do by yourself. The audit can be used for security reasons (see who is abusing his/her assigned permissions to do something that they are not supposed to do), this includes Admins as well as Delegates, but also you can configure to audit the mailbox owner actions which can help with troubleshooting.
   
   The Audit is enabled per user mailbox and is saved as a part of the mailbox.
   The audit log needs to be enabled, the amount of days to save the audit can be specified (90 days by default), and most importantly, you must configure what to audit.
   
   In my case, the default owner audit setting did not include all types of operations so it was difficult at first to track the problem. (which was discovered by the trace due to the lack of information). However, after enabling additional audit items on the mailbox owner, it enabled tracking all actions and associating specific calendar actions with a specific ActiveSync mobile device.
   This eventually was found to be the cause of the above problem.
   
   So... Lets start this thing:
   
   First, log on with a user with administrative rights to the Exchange server to enable auditing for the user mailbox you would like to investigate.
   
   Now, Enable auditing on the mailbox with the command:
   
   Set-Mailbox UserName -AuditEnabled $true
   
   This command will enable auditing for Admin actions as will as Delegate actions, but not for mailbox owner actions.
   
   If you run the command Get-Mailbox UserName | select *audit*
   you will see exactly what is audited and what is not:
   
   AuditEnabled     : True
   AuditLogAgeLimit : 90.00:00:00
   AuditAdmin       : {Update, Move, MoveToDeletedItems, SoftDelete, HardDelete, FolderBind, SendAs, SendOnBehalf, Create}
   AuditDelegate    : {Update, SoftDelete, HardDelete, FolderBind, SendAs, Create}
   AuditOwner       : {}
   
   You may notice that by default the mailbox owner is not edited.
   
   If you will run the command: Set-Mailbox Username -auditowner $true
   you will see that the AuditOwner setting changes to:
   
   AuditOwner       : {Update}
   
   This is nice, but may not be enough to get all the information you want.
   
   Lets enable auditing for all options for both Admin, Delegate and owner:
   
   Set-Mailbox UserName -AuditEnabled $true -AuditAdmin Update,Move,MoveToDeletedItems,SoftDelete,HardDelete,FolderBind,SendAs,SendOnBehalf,Create -AuditDelegate Update,SoftDelete,HardDelete,FolderBind,SendAs,Create
   -AuditOwner Update,Move,MoveToDeletedItems,SoftDelete,HardDelete,Create
   
   Of course this will consume more space, however you only need to enable this when you need to investigate an issue or on a regular basis for specific users.
   
   Now, allow some time for the information to be collected.
   
   When you want to investigate, you will need to view or export the information in order to filter them more easily.
   
   The command that is used to retrieve the data is:
   Search-MailboxAuditLog UserName -ShowDetails
   
   This will give you tons of lines if information on every every action performed on every item.
   You can either use powershell to filter the output sent to the screen, or you can more easily send everything (filter or unfiltered) to a CSV file later to be opened in Excel or your favorite spreadsheet.
   
   
   Search-MailboxAuditLog UserName -LogonTypes Admin,Delegate,Owner -ShowDetails | export-csv c:\UserName.csv -Encoding utf8
   
   Please note that you can specify all type of logon types (as shown in the command above) or just specific logon type such as Delegate if you only want to get data about actions performed by the delegates.
   The command -Encoding utf8 will allow exporting of characters that can be identified for non-English (in my case Hebrew) in the item subject.
   
   You can additionally filter based of available properties, for example, add the following pipe between the Search section and the Export section in order to filter audit items in the calendar folder only | where {$_.FolderPathName -like "\calendar"} | 
   If English is NOT your default folder language you will need to modify the folder name based on your folder language.
   
   In the CSV file you will get all the detailed information on the actions performed.
   The information also include the client type, such as Outlook and even ActiveSync.
   The ClientInfoString provides the full Device information so you will be able to tell which device performed the action (this is very helpful in case that a few devices are user to sync with the same mailbox).
   
   Another option to view and filter audit logs for a specific mailbox is using the Exchange Control Panel (ECP) on your exchange server. For example user https://yourserver/ecp
   
   
   
   However, based on available options, it doesn't seem you can view owner actions using this interface.
   
   I hope this article will provide you with valuable information for troubleshooting.
   If you like this article you are welcome to drop me a line at: liranzamir@gmail.com
   
    

Load Powershell Addins in your scripts

It is often that you want to write a script which uses Powershell cmdlets that are not the basic ones found in Powershell v2.0.

For example, you might want to perform actions on Active Directory using Windows 2008 built in AD commands, Exchange 2010 commands or even Quest Active Roles.

Option 1 - Importing the cmdlets from the batch
One option to run a script with Exchange 2010 commands (on a sever running Exchange 2010 or a server with the Exchange management tools installed) is to call the script from from a batch file which loads the Exchange powershell extentions from the batch file. For Example you .cmd or .bat file will look like this:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command ". 'C:\Program Files\Microsoft\Exchange Server\V14\bin\RemoteExchange.ps1'; Connect-ExchangeServer -auto; C:\Scripts\YourScriptName.PS1"


Option 2 - Importing the  cmdlets  from the script
This option - also running on an Exchange server or a server with the Exchange management tools installed, use the batch to load the script using the "Normal" Powershell with this command:

powershell -command "& 'C:/scripts/YourScriptName.ps1'"

and

In the script file, use the following commands to load the needed PSSnapIn:

For Exchange 2010 PSSnapIn
Add-PSSnapIn Microsoft.Exchange.Management.PowerShell.E2010

For Microsoft Active Directory commands (2008)
Import-Module activedirectory

For Microsoft Microsoft Lync 2010
Import-Module Lync

For Quest Active-Roles commands (Free download from Quest software Click Here )
Add-PSSnapin Quest.ActiveRoles.Admanagement


Example:

createmailboxlist.cmd
powershell -command "& 'C:/scripts/Mailboxlistgenerator.ps1'" 


Mailboxlistgenerator.ps1
Add-PSSnapIn Microsoft.Exchange.Management.PowerShell.E2010 
$list = get-mailbox -resultsize unlimited | select name,database,primarysmtpaddress
$list | export-csv c:\scripts\Mailboxlist.csv


Option 3 - Remoting commands to Exchange server
This option allows you to run Exchange Powershell commands from a computer running Powershell with
no Exchange snap-ins by creating a remote session to the Exchange server.
This requires an http connection between the the server or workstation and the Exchange server.
Use the following commands:

$Session = new-pssession -configurationname microsoft.exchange -connectionUri http://ExchangeServerName/powershell
Import-PSSession $Session

From this step you can run Exchange Powershell commands.


Now all you need is to schedule the command by running the batch or run it manually

The mailbox database size is much larger than the mailboxes.. What's up with that ???

You may notice that the database is much larger than the actual mailbox content.
It is important to know that the database contains additional data.. and that's the Dumpster data.

The database dumpster contains:
- Deleted mailboxes on retention (keep deleted mailboxes for X days in the mailbox database limits tab)
- Deleted items in the mailbox (Keep deleted items for X days in the mailbox database limits tab)
The deleted items in retention include items that were removed then a user empties the "Deleted Items" folder (this is known as "Soft Delete") or when the user Shift+Del an item ("Hard Delete").

All of those items are not counted as a part of the regular mailbox size limit, the dumpster limit is used instead.

Note! Deleted items can be recovered by using Outlook client or using Outlook Web App as long as the number of days that passed since the soft or hard deletion did not exceed the configured retention time that was set on the database Limits properties tab.

Exchange 2010 set a limit to the dumpster by default to the amazing size of 30 GB Per mailbox !
This means that a user with a limited mailbox size of 100 MB can potentially get large Emails, delete them, empty the deleted items folder (or use hard delete: Shift+Del), and than the content will be stored in the dumpster and will not longer be counted for the mailbox size limit.
A user might have a 50 MB mailbox size and up to 30 GB of dumpster items... all of which takes up space in the mailbox database. That is a lot !

important note !
The default dumpster quota size of 30 GB mentioned above is only applied to a mailboxes if a mailboxes are configured to inherit their size limit from the database limit settings.
If you decided NOT to use the size limit settings from the database limits tab, and assign size limits individually, the 30 GB dumpster limit will not take effect at all, in fact user will actually have unlimited dumpster size... This may mean troubles.
The dumpster limits will be ignored when a mailbox is configured on various type of legal hold. If you set one, you probably know about it..

Do large dumpsters effect your Exchange environment ?
Before you decide if and what action to take, wouldn't you like to know what is the actual effect of the dumpster usage in your Exchange databases ? Of course you do.

You can collect this data from the user mailboxes with the following EMS (Exchange Powershell) command:

$MBX = get-mailbox -resultsize unlimited | Get-MailboxStatistics | select displayname,totaldeleteditemsize,database 

The above command will query every mailbox in the Exchange organization and will fetch the name of the owner of the mailbox, the mailbox total dumpster size, and the database containing the mailbox. The results are than stored in an array inside the $MBX variable (Which we can later manipulate). 

Now we would like to view the results easily. We will take the data and sort it by the recoverable items size (dumpster size), largest results first:

$MBX | Sort-Object TotalDeletedItemSize -Descending
Now you can take the sorted results, and save them back to the variable:

$MBX = $MBX | Sort-Object TotalDeletedItemSize -Descending

Would you like to see only the top five mailboxes ? try this:
(make sure you used the previous command to save the sorted list) :

$mbx | select -first 5

Or get only the list of mailboxes where the dumpster size occupy more 1 GB or more.

$MBX | where {$_.TotalDeletedItemSize -ge 1GB} | Sort-Object TotalDeletedItemSize -Descending

You might also want to export the entire thing to a CSV file to analyze later:

$MBX | export-csv c:\mailboxrecoverables.csv -NoTypeInformation

What about getting the report to your Email ? (Change the second command based on your organization).

$HtmlBody = $mbx | convertto-html | out-string

Send-MailMessage -from Dumpster@yourdomain.com -to YourName@yourdomain.com -Subject "Dumpster Report" -BodyAsHtml $HtmlBody -smtpserver YourExchangeServerName

Not interesting enough ?!?!? Lets put it all into a script which can be scheduled.
The script will fetch the information for all mailboxes as we did before, sort the data based on largest mailbox dumpsters, and Email us only if there are mailboxes with dumpster size over 1GB.

Lets call this script (Click the script name to download) DumpsterReporter.ps1 or DumpsterReporter.zip

#Variables
$filtersize = 300MB
$fromAddress = "DumpsterReport@yourdomain.com"
$ToAddress = "YourEmailAddress@yourdomain.com"
$SMTPServer = "YourExchangeServerFQDN"

# Lets get all the information first
$MBX = get-mailbox -resultsize unlimited | Get-MailboxStatistics | select displayname,totaldeleteditemsize,database

# Find only the mailboxes with dumpster size larger or equal to the limit we chose to look for, save to $results
$results = $mbx | where {$_.totaldeleteditemsize -ge $filtersize} | sort totaldeleteditemsize -descending

# Send the Email only when there are valid results  
If ($results -notlike $null ) 
{
$HtmlBody = $results | convertto-html | out-string
Send-MailMessage -from $fromAddress -to $ToAddress -Subject "Dumpster Report" -BodyAsHtml $HtmlBody -smtpserver $SMTPServer 
}

In order to run the script on schedule on your exchange server, first create a folder for your scripts.
For example: c:\scripts

Save the DumpsterReporter.ps1 script to this folder.
In the folder create a batch / cmd file which will execute the powershell script with the Exchange cmdlets:

for example save the following content in DumpsterReporter.cmd

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command ". 'C:\Program Files\Microsoft\Exchange Server\V14\bin\RemoteExchange.ps1'; Connect-ExchangeServer -auto; C:\Scripts\DumpsterReporter.PS1"

Now.. all you need to do is to edit the variables in DumpsterReporter.ps1 to march your environment and use the Task Scheduler in control panel to schedule the batch file to run.

To make sure the batch is working fine, first try to run it by double clicking it.

Note that the scheduled task should be set to run with an account which have Exchange permissions, and also select the option "Run whether user is logged on or not"

 Now I know.. what should I do next ?

OK. Now you know if the dumpster size is an issue in your organization or not. It may not be an issue today but it might become one later on so you should consider allocating the dumpster size yourself.

As stated earlier on, setting the Recoverable items Quota can be done on a per Mailbox Database basis, or per Mailbox basis.
The per Database setting will apply to all mailboxes which inherit the mailbox size limit from the database.
The per Mailbox setting is required only when a mailbox is configured with it's own set of limits which are not inherited from the mailbox database.

The behavior of the dumpster quota is FIFO. The oldest item in the dumpster will be removed, even if it did not reach the age to be removed, if a newer item was deleted which caused the quota to exceed the limit.

To set the mailbox database level dumpster warning quota and limit quota use the following command:


Set-MailboxDatabase DatabaseName -RecoverableItemsWarningQuota 3gb -RecoverableItemsQuota 4gb

To set the  dumpster warning quota and limit quota on a mailbox, use the following command:

set-mailbox MailboxName -RecoverableItemsWarningQuota 2GB -RecoverableItemsQuota 3GB

Of course it is up to you to set the appropriate sizes based on your environment. 
What should you consider ?

• What will be a "Logical" dumpster allocation for a user ?
  This could be impacted by the number of days you set to keep deleted items.
  A user is more likely to have a larger dumpster size for larger number of days.
• What is the user mailbox size limit ?
  A user with a really limited mailbox size may be forced to repeatedly delete incoming mail.
  This can cause a large dumpster usage, and even more deletion mistakes.
  Consider setting those type of users with at least twice the mailbox size in the dumpster quota.
  
  For users with large mailboxes, they may not delete items as often, but may accidentally delete an entire folder or folder root with it's sub-folders. Allowing two the mailbox size in dumpster quota may also be useful.
• What are your storage limits ?
  If storage is tight... get a larger one and enjoy what Exchange 2010 can offer :-).
  But if you cannot do this right now, first consider limiting the number of days that deleted items are retained for the general users. Also use the above script to locate cases of misuse of the dumpster quota either as a result of a regular usage or due to a problem.
• Which users are more important ?
  Some employee Email are less critical than other. Provide regular mailbox with less days of deleted item retention and smaller dumpster quota, and provide management with more retention days and dumpster quota. This may save you from using tape to restore from backup.
  You could also set a different SLA (Service Level Agreement) for Email recovery for Management and regular users.

How to remedy a full dumpster ?

Well, I did not get to see the FIFO process for myself, yet. However, if your report state that a mailbox contains 20GB of data and it seems highly illogical or unacceptable in the current storage situation, you can ask the user permission to purge part or all of the Dumpster by using Outlook or Outlook Web App.

In Outlook 2010, go to the Folder ribbon, select Recover Deleted Items, and in the list of items, select a large bunch of Emails and permanently delete them with the X icon.

If the dumpster size is really really bug, you might want to look at the items before deleting them in order to find out if a specific Email caused the dumpster to fill, and what was the cause.

There might be other methods of exporting or deleting items in the recoverable items of a mailbox.. I may look into it in a later date.

Cheers

Liran Zamir