Google Docs Backup Documentation

DOWNLOAD THE LATEST DOCUMENTATION

Version 1.0 - Please download the latest documentation

The document backup package consists of a command line application, and a Windows application that enables users to backup their Google Apps domain's documents using two legged OAuth.

This document defines how to properly run and configure both the command line and Windows application utilities.


Platform Requirements

  • Windows Based PC (XP, Vista, Server) with .NET 3.5 Installed (Available through Windows Update or directly from MS)
  • Google Apps Premier or Education Edition

  • System Requirements (for .net 3.5)
    • Supported Operating Systems: Windows Server 2003; Windows Server 2008; Windows Vista; Windows XP
    • Processor: 400 MHz Pentium processor or equivalent (Minimum); 1GHz Pentium processor or equivalent (Recommended)
    • RAM: 96 MB (Minimum); 256 MB (Recommended)
    • Hard Disk: Up to 500 MB of available space may be required
    • CD or DVD Drive: Not required
    • Display: 800 x 600, 256 colors (Minimum); 1024 x 768 high color, 32-bit (Recommended)

Tested On

  • Windows XP Pro with .NET 3.5 Installed
  • Google Apps Premier

Installation

The Google Docs Backup utility is delivered a Windows MSI Installer which includes product documentation. The provided installer package will add Start Menu items, and an entries under Add/Remove Programs.


Licensing

The Docs Backup Utility requires a license to be installed to be fully functional. Without a license file, the application will be in "Trial Mode", allowing you to perform actions on only one user at a time. To obtain a license, please contact LTech for pricing.

License installation instructions will be included with your license.


Google Apps Configuration

Google Apps must be configured to use Two-Legged OAuth. This will allow Google Apps services to use OAuth to retrieve domain documents without requesting user permission. The domain must also have an admin user for the utility. An existing admin user account is all that is necessary but it may be beneficial to create a separate user admin account.

OAuth configuration for Google Apps domains is available on the administrator dashboard under the section titled Advanced Tools. The OAuth consumer key and secret is available under this menu as well.

The Provisioning API for Google Apps must also be enabled to operate the utility. This option is enabled under the 'Users and groups' section.

The Document Backup utility has three configuration modes. Configuration options my be passed in by command line, application configuration file or manually.

By default the utility is set to accept parameters via command line. To enable loading from a configuration file, the command line parameter '--config' should be passed in. This will direct the application to look at the app.config for configuration options. If the parameter '--setup' is supplied, the application will prompt the user for the parameters as required and whether to save the configuration.

Configuration settings loaded from the application configuration file can be overridden by supplying additional options on the command line. These additional settings can be made permanent by using the save (--save) parameter.

Backup Modes

Google Docs Backup offers two different storage options for document backups.

File System Backup

This is a traditional file based backup option. To enable via the command line the --backuppath parameter should be supplied with a Windows file system path.

Amazon S3 Backup

Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. Amazon S3 is a product offering under Amazon's Web Services platform (AWS).

To enable Amazon S3 Backup the parameters --awsid (AWS Access Key Id), --awssecret (AWS Secret Access Key) and --awsbucket (AWS Bucket) should be supplied over the command line. Encryption options on Google Docs Backup are available to encrypt sensitive information.

Command Line Utility

Configuration Mode: Command Line

The following parameters are the command line configuration options that are available:

Required Fields

  • domain - This is the source Google Apps domain to use for the export.
  • admin - This is the admin user that will be used to query Google Apps for user information.
  • pass - The admin user's password.
  • all - This argument does not require a value to be supplied, instead the key only needs to exist and will tell the utility to process all domain users.
  • user - If '--all' is supplied this parameter is ignored. This parameter represents a single user to process.
  • key - OAuth Consumer Key.
  • secret - OAuth Consumer Secret.

Backup Options

  • backuppath- Enter a folder to enable local file system backup.
  • awsid - Amazon S3 Access Key Id. Use with --awssecret and --awsbucket to enable Amazon S3 backup.
  • awssecret - Amazon S3 Secret Access Key
  • awsbucket - Amazon S3 Bucket to backup documents to. This bucket must exist before start of backup.

Saving/Encryption/Setup

  • save - This will save the current configuration options to the application configuration file. If the configuration has previously been encrypted any changes will also be saved encrypted.
  • encrypt - This will save and encrypt the current configuration settings to the application configuration file. It is not necessary to supply both --save and --encrypt.
  • setup - The application will prompt the user for the parameters as required and whether to save the configuration.
  • config - uses the values from the config file first.

Optional Fields

  • outpath - This is the directory that output will be exported to. In the case of the backup utility this folder will contain a folder for each user's documents. This field is not required and will default to 'C:\export' which will be created if it doesn't exist in the folder where the utility is being run.
  • silent - This argument does not require a value. This argument indicates that the utility will disable all output to the console. All output will be be stored in log files.
  • interval - By default the document backup utility will sleep between a specified amount of requests. The default value is 100 requests, but can be changed by specifying a different numerical value here.
  • sleep - By default the document backup utility will sleep between a specified amount of requests. The default value for this parameter is 10 seconds, but can be changed by specifying a different numerical value here. If the value 0 is supplied it will effectively disable sleeping between requests.
  • folders - This enables the document utility to respect folder structure for user documents. This means that documents will be placed in a folder structure on the file system that mimics the folders on Google Apps. Since Google Apps allows documents to be in multiple folders this means that documents may also be downloaded to multiple folders.
  • filter - This optional allows for documents to only be downloaded if they are contained in a folder.

Document Formats

  • --docfrmt - The format document items will be exported as. Valid parameters are any of the following: DOC, HTML, PDF, PNG, RTF, TXT, ZIP
  • --prfrmt - The format presentations will be exported as. Valid parameters are any of the following: PPT, PDF, PNG, SWF, TXT
  • --ssfrmt - The format spreadsheets will be exported as. Valid parameters are any of the following: XLS, CSV, PDF, ODS, TSV, HTML

Example Use Cases

Use Case Command
Basic Domain Backup
 DocumentBackup.exe --domain domain.com --admin login --pass password --all --key domain.com --secret DdhDnX8921d
Basic User Backup
 DocumentBackup.exe --domain domain.com --admin login --pass password --user exampleuser --key domain.com --secret DdhDnX8921d
Override Export Directory
 DocumentBackup.exe --domain domain.com --admin login --pass password --user exampleuser --key domain.com --secret DdhDnX8921d --outpath C:\export\googleapps\domain.com\documents
Override Sleep Intervals
 DocumentBackup.exe --domain domain.com --admin login --pass password --user exampleuser --key domain.com --secret DdhDnX8921d --sleep 20 --inteval 250
Override Document Formats (Document and Spreadsheet)
 DocumentBackup.exe --domain domain.com --admin login --pass password --user exampleuser --key domain.com --secret DdhDnX8921d --docfrmt RTF --ssfrmt XLS
Override Document Formats (Document and Presentation) DocumentBackup.exe --domain domain.com --admin login --pass password --user exampleuser --key domain.com --secret DdhDnX8921d --docfrmt RTF --prfrmt SWF
Run Silently
 DocumentBackup.exe --domain domain.com --admin login --pass password --all --key domain.com --secret DdhDnX8921d --silent
Folder Filter
 DocumentBackup.exe --domain domain.com --admin login --pass password --all --key domain.com --secret DdhDnX8921d --filter example
Amazon S3 Backup
 DocumentBackup.exe --domain domain.com --admin login --pass password --user exampleuser --key domain.com --secret DdhDnX8921d --awsid awsid --awssecret awssecret -awsbucket --awsbucket

Configuration Mode: App.Config Configuration

Configuration file mode is enabled by supplying the argument '--config' in at the command line.

The configuration file for the application is named 'DocumentBackup.exe.config', and the following parameters are the configuration file options that are available. By default configuration options are commented out of the file, but if the configuration file option is used the comments should be removed and sample placeholder information should be replaced as appropriate.

Required Fields

  • GoogleAppDomain -- This is the source Google Apps domain to use for the export.
  • GoogleAppsAdminUser- This is the admin user that will be used to query Google Apps for user information.
  • GoogleAppsAdminPassword - The admin user's password.
  • ProcessDomain - This specified whether the  utility will process the entire domain or whether to look for the 'UserName' key and process a single user. Valid options for this field are "true" and "false". If any other values are supplied the value of the key will be inferred as false.
  • UserName - This key specifies a username for the utility to backup, if the ProcessDomain key is set to true this value key is ignored
  • OAuthConsumerKey - OAuth Consumer Key.
  • OAuthConsumerSecret - OAuth Consumer Secret.

Backup Options

  • BackupDirectory - Enter a folder to enable local file system backup.
  • AWSAccessKeyId - Amazon S3 Access Key Id. Use with AWSSecretAccessKey and AWSBucket to enable Amazon S3 backup.
  • AWSSecretAccessKey - Amazon S3 Secret Access Key
  • AWSBucket - Amazon S3 Bucket to backup documents to. This bucket must exist before start of backup.

    • Optional Fields

    • OutputDirectory- This is the directory that output will be exported to. In the case of the backup utility this folder will contain a folder for each user's documents. This field is not required and will default to 'C:\export' which will be created if it doesn't exist in the folder where the utility is being ran.
    • RunSilent - This argument does not require a value. This argument indicates that the utility will disable all output to the console. All output will be be stored in log files.
    • SleepInterval - By default the document backup utility will sleep between a specified amount of requests. The default value is 100 requests, but can be changed by specifying a different numerical value here.
    • SleepPeriod - By default the document backup utility will sleep between a specified amount of requests. The default value for this parameter is 10 seconds, but can be changed by specifying a different numerical value here. If the value 0 is supplied it will effectively disable sleeping between requests.
    • EnableFolders - This enables the document utility to respect folder structure for user documents. This means that documents will be placed in a folder structure on the file system that mimics the folders on Google Apps. Since Google Apps allows documents to be in multiple folders this means that documents may also be downloaded to multiple folders.
    • FolderFilter - This optional allows for documents to only be downloaded if they are contained in a folder.

    Document Formats

    • DocumentFormat - The format document items will be exported as. Valid parameters are any of the following: DOC, HTML, PDF, PNG, RTF, TXT, ZIP.
    • PresentationFormat - The format presentations will be exported as. Valid parameters are any of the following: PPT, PDF, PNG, SWF, TXT.
    • SpreadsheetFormat - The format spreadsheets will be exported as. Valid parameters are any of the following: XLS, CSV, PDF, ODS, TSV, HTML.

    Example Configuration File

    *Note* Configuration settings can be encrypted on the command line by passing the --encrypt parameter, however the config will be rendered unable to be altered. Configuration settings are automatically saved encrypted when using the Windows Application Utility.


    Example Use Cases

    In all cases the following keys are assumed to be contained under the appSettings section in the configuration file 'DocumentBackup.exe.config'

    Use Case
    		 Configuration Keys
    Basic Domain Backup <add key="GoogleAppsDomain" value="domain.com"/>
    <add key="GoogleAppsAdminLogin" value="administrator"/>
    <add key="GoogleAppsAdminPassword" value="password"/>
    <add key="ProcessAllUsers" value="true"/>
    <add key="OAuthConsumerKey" value="domain.com"/>
    <add key="OAuthConsumerSecret" value="DdhDnX8921d"/>
    Basic User Backup <add key="GoogleAppsDomain" value="domain.com"/>
    <add key="GoogleAppsAdminLogin" value="administrator"/>
    <add key="GoogleAppsAdminPassword" value="password"/>
    <add key="ProcessAllUsers" value="false"/>
    <add key="UserName" value="exampleuser"/>
    <add key="OAuthConsumerKey" value="domain.com"/>
    <add key="OAuthConsumerSecret" value="DdhDnX8921d"/>
    Override Export Directory <add key="GoogleAppsDomain" value="domain.com"/>
    <add key="GoogleAppsAdminLogin" value="administrator"/>
    <add key="GoogleAppsAdminPassword" value="password"/>
    <add key="ProcessAllUsers" value="true"/>
    <add key="OAuthConsumerKey" value="domain.com"/>
    <add key="OAuthConsumerSecret" value="DdhDnX8921d"/>
    <add key="OutputDirectory" value="C:\export\googleapps\domain.com\documents" />
    Override Document Formats (Document and Spreadsheet) <add key="GoogleAppsDomain" value="domain.com"/>
    <add key="GoogleAppsAdminLogin" value="administrator"/>
    <add key="GoogleAppsAdminPassword" value="password"/>
    <add key="ProcessAllUsers" value="true"/>
    <add key="OAuthConsumerKey" value="domain.com"/>
    <add key="OAuthConsumerSecret" value="DdhDnX8921d"/>
    <add key="DocumentFormat" value="RTF" />
    <add key="SpreadsheetFormat" value="XLS" />
    Override Document Formats (Document and Presentation) <add key="GoogleAppsDomain" value="domain.com"/>
    <add key="GoogleAppsAdminLogin" value="administrator"/>
    <add key="GoogleAppsAdminPassword" value="password"/>
    <add key="ProcessAllUsers" value="true"/>
    <add key="OAuthConsumerKey" value="domain.com"/>
    <add key="OAuthConsumerSecret" value="DdhDnX8921d"/>
    <add key="DocumentFormat" value="RTF" />
    <add key="PresentationFormat" value="SWF" />
    Run Silently <add key="GoogleAppsDomain" value="domain.com"/>
    <add key="GoogleAppsAdminLogin" value="administrator"/>
    <add key="GoogleAppsAdminPassword" value="password"/>
    <add key="ProcessAllUsers" value="true"/>
    <add key="OAuthConsumerKey" value="domain.com"/>
    <add key="OAuthConsumerSecret" value="DdhDnX8921d"/>
    <add key="RunSilent" value="true" />
    Folder Filter <add key="GoogleAppsDomain" value="domain.com"/>
    <add key="GoogleAppsAdminLogin" value="administrator"/>
    <add key="GoogleAppsAdminPassword" value="password"/>
    <add key="ProcessAllUsers" value="true"/>
    <add key="OAuthConsumerKey" value="domain.com"/>
    <add key="OAuthConsumerSecret" value="DdhDnX8921d"/>
    <add key="FolderFilter" value="example" />

    Configuration Mode: Prompt

    Manual configuration mode is enabled by supplying the argument '--manual' at the command line. In this mode the user will be prompted for configuration parameters in sequence.


    Logging Setup

    Logging for the document backup utility is handled using log4net.

    Configuration options for log4net are stored in the application configuration file. Selecting different input modes for configuration keys in the document backup utility does not affect the configuration keys for log4net.

    Below is the default configuration settings for log4net. The main configuration option that will be changed from the default setting is the log file location value (bolded). This can be a relative path or a fully qualified location on a drive or network share.

    Changing the root level value will modify increase and decrease the amount of logging statements outputted, but may also omit useful logging information. It is not recommended to raise the value to "WARN" or above, as most logging statements are logged at the "INFO" level. Lowering the level to "DEBUG" will output logging statements indicating start and end points of methods.

    For more information about configuring log4net see:
    http://logging.apache.org/log4net/release/manual/configuration.html


    Sample log4net Configuration

     <log4net>
        <appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
          <file value="DocBack.log.txt" />
          <appendToFile value="true" />
          <rollingStyle value="Size" />
          <maxSizeRollBackups value="50" />
          <maximumFileSize value="20000KB" />
          <staticLogFileName value="true" />
          <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%date [%thread] %-5level %logger - %message%newline" />
          </layout>
        </appender>
        <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
          <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%date %-5level %message%newline" />
          </layout>
        </appender>

        <!-- Setup the root category, add the appenders and set the default level -->
        <root>
          <level value="DEBUG" />
          <appender-ref ref="RollingLogFileAppender" />
          <appender-ref ref="ConsoleAppender" />
        </root>
      </log4net>



    Output

    Document backups are stored in the export directory specified at runtime. Output for the backup utility consists of a directory for each user. Inside each folder there will be contained the user's documents, and if enabled, any folders that a user has created. Folders are stored in a tiered layout similar to the relationships in Google Docs.

    Google's document and folder structure allows for a document or folder to have multiple parent folders. When enabled, the document backup utilities will mimic Google's entity relationships, so any folder or document with multiple parent folders will be stored in both folders on the file system.


    Examples

    Example user folders:




    Example user documents:



    Windows Application Utility

    The windows application utility is a graphic interface for the document backup utility. Configuration options are entered manually, but a configuration option is available as well that allows configuration options to be saved for future usage.

    Configuration

    Configuration is broken down into five separate sections on the main form.

    Required Fields

    • Domain - This is the source Google Apps domain to use for the export.
    • Admin User - This is the admin user that will be used to query Google Apps for user information.
    • Admin Pass - The admin user's password.
    • OAuthConsumerKey - OAuth Consumer Key.
    • OAuthConsumerSecret - OAuth Consumer Secret.

    Select Users

    • Select All - Click select all to select all domain users.
    • Clear Selection - Clears the selected users from the check box list.
    • Upload File - Upload a comma separated or line separated file containing the users that you would like backed up.


    Backup Options

    • File System Backup - Click to enable file system backup.
    • Output Path - Select a path that backed up documents will be stored to
    • Amazon S3 Backup - Click to enable Amazon S3 backup.
    • Access Key Id - This is your unique Access Key Id for Amazon S3
    • Secret Access key - This is the secret access key for your Amazon S3 account. You may find out the value of this key by logging into your Amazon S3 dashboard.
    • Bucket - This is the Amazon S3 bucket that documents will be saved to. This bucket must exist before start of backup.


    Additional Options

    • Output Path - This is the directory that output will be exported to. In the case of the backup utility this folder will contain a folder for each user's documents. This field is not required and will default to 'C:\export' which will be created if it doesn't exist in the folder where the utility is being ran.
    • Sleep After - By default the document backup utility will sleep between a specified amount of requests. The default value is 100 requests, but can be changed by specifying a different numerical value here.
    • Sleep For - By default the document backup utility will sleep between a specified amount of requests. The default value for this parameter is 10 seconds, but can be changed by specifying a different numerical value here. If the value 0 is supplied it will effectively disable sleeping between requests.

    Folders

    • Enable Folders - This enables the document utility to respect folder structure for user documents. This means that documents will be placed in a folder structure on the file system that mimics the folders on Google Apps. Since Google Apps allows documents to be in multiple folders this means that documents may also be downloaded to multiple folders.
    • Folder Filter - This optional allows for documents to only be downloaded if they are contained in a folder.

    Document Formats

    • Documents - The format document items will be exported as. Valid parameters are any of the following: DOC, HTML, PDF, PNG, RTF, TXT, ZIP.
    • Spreadsheets - The format spreadsheets will be exported as. Valid parameters are any of the following: XLS, CSV, PDF, ODS, TSV, HTML.
    • Presentations - The format presentations will be exported as. Valid parameters are any of the following: PPT, PDF, PNG, SWF, TXT.

    Executing Backups

    Once all required fields are entered the backup is ready to be ran. To run the utility you can click the 'Run' button that is placed along with the required fields, or you can click over to the 'Run' tab and start the backup from there.

    If all parameters are valid, and the backup is started, all logging will be printed to the main text box on the 'Run' tab. At any point you can the output from the utility can be exported to a .txt file.

    Reporting

    The document backup utility supports reports for both users and domains. After completing processing, a user report is created that lists the following information:

    • Date of backup
    • Username
    • User first name and last name
    • Domain
    • Number of successfully exported documents
    • Number of documents unable to be exported
    • Elapsed time
    • Document format
    • Spreadsheet format
    • Presentation format
    • List of successfully exported documents
    • List of documents unable to be exported and reason per document

    User reports are stored under each user's folder, and are named report.txt.

    Example User Report

    Domain reports are available when processing the entire domain. They are not created when processing single users. Domain reports aggregate content from the user reports and provide a summary of the processing that was done for the domain.

    Domain reports contain the following information:

    • Date of backup
    • Domain
    • Number of users
    • Number of successfully exported documents
    • Average number of documents exported per user
    • Number of documents unable to be exported
    • Average number of documents unable to be exported per user
    • Elapsed time
    • Document format
    • Spreadsheet format
    • Presentation format
    • List of users exported, plus success and error counts for each user.

    Example Domain Report