PowerShell : Compress Folder (IN PROGRESS)

Introduction

This script was written to enable the backup of an application’s logs. The reason for this requirement is that the logs generated are overwritten between system or service restarts and we have a duty to maintain them for 5 years.

Input

#ItemRequired?Description / Location
1PARAMETERyes-folder <full path to target folder for compression>
2PARAMETERyes-zipfolder <full path to folder which will receive the zip file>

Output

# Item Description / Location
1 LOG All successful and failed operations will be logged in the log file defined by the script variable $LOG
2File Creation If successful a new *.zip file will be created in the -zipfolder <dir> defined by the caller

Logging

Currently the logging is managed from within the script.

Log file location: C:\Scripts\Powershell\Logs\compress-archive.ps1.log

$LOGTIME is s timestamp in the format “yyyy-MM-dd_hh-mm-ss”

Log call within the script:
“$LOGTIME <<TEXT TO LOG>>” | Out-File $LOG -Append -Force

The script is then contained with a try{} catch{} parameter with logging output for all exception messages and items.

I’ve also tried to log throughout each action within the script so both SUCCESS and ERRORs are logged which provides a debugging method (at least I should know where it falls over!).

TODO; incorporate an email alert for all failures

Usage Examples

Contained in the header of the script – see below

The Script

As this script is running at the lowest privileges (ENG\svEngScript via Task Scheduler) I had to create a new GPO to permit that pre-existing account “Log on as batch job” via ticket #26725. I also had to them provide read permissions to the Cadence log file source on svlic1801 (C:\Cadence\Logfiles) along with read+write access to the target directory for the zip file at “C:\srvinstall\Cadence\CadenceLogFileBackup”. After these changes the user account was able to execute the powershell script successfully.

# Name : Compress Archive
# Author : Dave 
# Date : 
# Ticket# : 
#
#
# Scope : The source folder to be compressed and an output folder to receive the compressed file (both passed via parameters)
#
# Input(s) : Parameters[-folder <full path to folder>, -zipfolder <full path to zip output FOLDER>]
#
# Output(s) : A compressed .zip file of the [-folder] specified at the [-zipfolder] location specified
 
 
####################################################
# This script provides a controlled method of zipping a folder
####################################################
 
####################################################
# USAGE EXAMPLES
#
# Compress a folder:
# C:\> compressarchive.ps1 -folder "c:\test" -zipfolder "C:\zipped"
# EXPECTED OUTPUT:
# 1of2: A log entry at the defined $LOGFILEDIR\$LOGFILENAME confirming success of the compression in the format:
# [DATETIME STAMP] SUCCESS <path to the source folder> compressed to <path to the target output dir>\2019-03-08_154256_<source folder name>.zip
# 2of2: A new .zip file at <path to the target output dir>\2019-03-08_154256_<source folder name>.zip should exist
####################################################
 
# Define the required parameter of an input file (will need to be in CSV format with 3 columns: Hostname,IPAddress,VendorName - tested later in the script)
##################################################################################
param(
 [Parameter(Mandatory=$true)]
 [string]$folder,
  
 [Parameter(Mandatory=$true)]
 [string]$zipfolder
)
 
# Define the Variables
##################################################################################
$datetime = get-date -Uformat "%Y-%m-%d_%H%M%S"
##################################################################################
$LOGFILEDIR = "SOME DIRECTORY OF YOUR CHOICE" # the full path of the log file for the script to output to
$LOGFILENAME = "compressarchive.ps1.log" # the full name of the log file for the script to output to
$LOG = ($LOGFILEDIR + "\" + $LOGFILENAME)
$LOGTIME = Get-Date -Format "yyyy-MM-dd_hh-mm-ss"
 
# Prepare the logging
##################################################################################
# Check for the $LOGFILEDIR & $LOGFILENAME and create them if they don't exist
# NOTE: No logging until this happens except to STDOUT (Screen)
# Test for the folder and create if it doesn't exist:
Try
 {
 if(! (Test-Path -Path $LOGFILEDIR )){
 New-Item -ItemType directory -Path $LOGFILEDIR >$null
 }
 if(! (Test-Path $LOGFILEDIR\$LOGFILENAME )){
 New-Item -ItemType file -Path $LOGFILEDIR\$LOGFILENAME >$null
 }
 }
Catch
 {
 $ErrorMessage = $_.Exception.Message
 $FailedItem = $_.Exception.ItemName
 }
"$LOGTIME BEGIN SCRIPT" | Out-File $LOG -Append -Force
 
# Main
##################################################################################
Try {
    # examine the variable $folder and remove any trailing \'s
    if ($folder.substring($folder.length -1) -eq "\"){
        "$LOGTIME INFO $($folder) needs the trailing \ removed" | Out-File $LOG -Append -Force
        $folder = $folder.substring(0,$folder.length -1 )
        "$LOGTIME INFO $folder has been renamed to $($folder)" | Out-File $LOG -Append -Force
    } else {
        "$LOGTIME INFO $($folder) matches the required format" | Out-File $LOG -Append -Force
    }
     
    # check the directory exists
    if (test-path $folder -pathtype Any) {
        "$LOGTIME SUCCESS $($folder) exists on the filesystem" | Out-File $LOG -Append -Force
    } else {
        "$LOGTIME ERROR $($folder) does not exist on the filesystem, exiting" | Out-File $LOG -Append -Force
        exit
    }
     
    # examine the variable $zipfolder and remove any trailing \'s
    if ($zipfolder.substring($zipfolder.length -1) -eq "\"){
        "$LOGTIME INFO $($zipfolder) needs the trailing \ removed" | Out-File $LOG -Append -Force
        $zipfolder = $zipfolder.substring(0,$zipfolder.length -1 )
        "$LOGTIME INFO $zipfolder has been renamed to $($zipfolder)" | Out-File $LOG -Append -Force
    } else {
        "$LOGTIME INFO $($zipfolder) matches the required format" | Out-File $LOG -Append -Force
    }
     
    # check the directory exists
    if (test-path $zipfolder -pathtype Any) {
        "$LOGTIME SUCCESS $($zipfolder) exists on the filesystem" | Out-File $LOG -Append -Force
    } else {
        "$LOGTIME ERROR $($zipfolder) does not exist on the filesystem, exiting" | Out-File $LOG -Append -Force
        exit
    }
     
    # create a unique filename $zip_file_name for the zip using the $datetime variable as part of the file
    # (eg: C:\output\2019-03-07_113010_logfiles.zip
    # where $zipfolder = c:\output and $folder = c:\someapplication\logfiles
    $zip_file_name = split-path $folder -leaf
    $zipfilepath = "$zipfolder\$datetime`_$zip_file_name.zip"
    "$LOGTIME SUCCESS `$zipfilepath assigned to '$($zipfilepath)' " | Out-File $LOG -Append -Force
     
    # compress the $folder outputting it to $zipfilepath and leaving the original folder in place
    compress-archive -path $folder -CompressionLevel Optimal -DestinationPath $zipfilepath
    "$LOGTIME SUCCESS $folder compressed to $($zipfilepath)" | Out-File $LOG -Append -Force
}
 
Catch {
 $ErrorMessage = $_.Exception.Message
 $FailedItem = $_.Exception.ItemName
 "$LOGTIME ERROR $($ErrorMessage) $($FailedItem)" | Out-File $LOG -Append -Force #LOGACTION
 }

VSS Admin Writers & Services

VSS WriterService NameService Display Name
ADAM $instanceName WriterADAM_$instanceName$instanceName
ASR WriterVSSVolume Shadow Copy
BITS WriterBITSBackground Intelligent Transfer Service
Certificate AuthorityCertSvcActive Directory Certificate Services
COM+ REGDB WriterVSSVolume Shadow Copy
DFS Replication service writerDFSRDFS Replication
DHCP Jet WriterDHCPServerDHCP Server
FRS WriterNtFrsFile Replication
FSRM writersrmsvcFile Server Resource Manager
IIS Config WriterAppHostSvcApplication Host Helper Service
IIS Metabase WriterIISADMINIIS Admin Service
Microsoft Exchange Replica WriterMSExchangeReplMicrosoft Exchange Replication Service
Microsoft Exchange WriterMSExchangeISMicrosoft Exchange Information Store
Microsoft Hyper-V VSS WritervmmsHyper-V Virtual Machine Management
MSMQ Writer (MSMQ)MSMQMessage Queuing
MSSearch Service WriterWSearchWindows Search
NPS VSS WriterEventSystemCOM+ Event System
NTDSNTDSActive Directory Domain Services
OSearch VSS WriterOSearchOffice SharePoint Server Search
OSearch14 VSS WriterOSearch14SharePoint Server Search 14
OSearch15 VSS WriterOSearch15SharePoint Server Search 15
Registry WriterVSSVolume Shadow Copy
Shadow Copy Optimization WriterVSSVolume Shadow Copy
SharePoint Services WriterSPWriterWindows SharePoint Services VSS Writer
SMS WriterSMS_SITE_VSS_WRITERSMS_SITE_VSS_WRITER
SPSearch VSS WriterSPSearchWindows SharePoint Services Search
SPSearch4 VSS WriterSPSearch4SharePoint Foundation Search V4
SqlServerWriterSQLWriterSQL Server VSS Writer
System WriterCryptSvcCryptographic Services
TermServLicensingTermServLicensingRemote Desktop Licensing
WDS VSS WriterWDSServerWindows Deployment Services Server
WIDWriterWIDWriterWindows Internal Database VSS Writer
WINS Jet WriterWINSWindows Internet Name Service (WINS)
Windows Server Storage VSS WriterWseStorageSvcWindows Server Essentials Storage Service
WMI WriterWinmgmtWindows Management Instrumentation

IPv4 Subnet Calculating

Calculating the Netmask Length (also called a prefix):

https://networkengineering.stackexchange.com/questions/7106/how-do-you-calculate-the-prefix-network-subnet-and-host-numbers

Convert the dotted-decimal representation of the netmask to binary. Then, count the number of contiguous 1 bits, starting at the most significant bit in the first octet (i.e. the left-hand-side of the binary number).

255.255.248.0   in binary: 11111111 11111111 11111000 00000000
                           -----------------------------------
                           I counted twenty-one 1s             -------> /21

The prefix of 128.42.5.4 with a 255.255.248.0 netmask is /21.

Calculating the Network Address:

The network address is the logical AND of the respective bits in the binary representation of the IP address and network mask. Align the bits in both addresses, and perform a logical AND on each pair of the respective bits. Then convert the individual octets of the result back to decimal.

Logical AND truth table:

Logical AND
128.42.5.4      in binary: 10000000 00101010 00000101 00000100
255.255.248. 0   in binary: 11111111 11111111 11111000 00000000
                           ----------------------------------- [Logical AND]
                           10000000 00101010 00000000 00000000 ------> 128.42.0.0

As you can see, the network address of 128.42.5.4/21 is 128.42.0.0

Calculating the Broadcast Address:

The broadcast address converts all host bits to 1s…

Remember that our IP address in decimal is:

128.42.5.4      in binary: 10000000 00101010 00000101 00000100

The network mask is:

255.255.248.0   in binary: 11111111 11111111 11111000 00000000

This means our host bits are the last 11 bits of the IP address, because we find the host mask by inverting the network mask:

Host bit mask            : 00000000 00000000 00000hhh hhhhhhhh

To calculate the broadcast address, we force all host bits to be 1s:

128.42.5.4      in binary: 10000000 00101010 00000101 00000100
Host bit mask            : 00000000 00000000 00000hhh hhhhhhhh
                           ----------------------------------- [Force host bits]
                           10000000 00101010 00000111 11111111 ----> 128.42.7.255

Calculating subnets:

You haven’t given enough information to calculate subnets for this network; as a general rule you build subnets by reallocating some of the host bits as network bits for each subnet. Many times there isn’t one right way to subnet a block… depending on your constraints, there could be several valid ways to subnet a block of addresses.

Let’s assume we will break 128.42.0.0/21 into 4 subnets that must hold at least 100 hosts each…

subnetting

In this example, we know that you need at least a /25 prefix to contain 100 hosts; I chose a /24 because it falls on an octet boundary. Notice that the network address for each subnet borrows host bits from the parent network block.

Finding the required subnet masklength or netmask:

How did I know that I need at least a /25 masklength for 100 hosts? Calculate the prefix by backing into the number of host bits required to contain 100 hosts. One needs 7 host bits to contain 100 hosts. Officially this is calculated with:

Host bits = Log2(Number-of-hosts) = Log2(100) = 6.643

Since IPv4 addresses are 32 bits wide, and we are using the host bits (i.e. least significant bits), simply subtract 7 from 32 to calculate the minimum subnet prefix for each subnet… 32 – 7 = 25.

The lazy way to break 128.42.0.0/21 into four equal subnets:

Since we only want four subnets from the whole 128.42.0.0/21 block, we could use /23 subnets. I chose /23 because we need 4 subnets… i.e. an extra two bits added to the netmask.

This is an equally-valid answer to the constraint, using /23 subnets of 128.42.0.0/21…

subnetting, 2nd option

Calculating the host number:

This is what we’ve already done above… just reuse the host mask from the work we did when we calculated the broadcast address of 128.42.5.4/21… This time I’ll use 1s instead of h, because we need to perform a logical AND on the network address again.

128.42.5.4      in binary: 10000000 00101010 00000101 00000100
Host bit mask            : 00000000 00000000 00000111 11111111
                           ----------------------------------- [Logical AND]
                           00000000 00000000 00000101 00000100 -----> 0.0.5.4

Calculating the maximum possible number of hosts in a subnet:

To find the maximum number of hosts, look at the number of binary bits in the host number above. The easiest way to do this is to subtract the netmask length from 32 (number of bits in an IPv4 address). This gives you the number of host bits in the address. At that point…

Maximum Number of hosts = 2**(32 – netmask_length) – 2

The reason we subtract 2 above is because the all-ones and all-zeros host numbers are reserved. The all-zeros host number is the network number; the all-ones host number is the broadcast address.

Using the example subnet of 128.42.0.0/21 above, the number of hosts is…

Maximum Number of hosts = 2**(32 – 21) – 2 = 2048 – 2 = 2046

Finding the maximum netmask (minimum hostmask) which contains two IP addresses:

Suppose someone gives us two IP addresses and expects us to find the longest netmask which contains both of them; for example, what if we had:

  • 128.42.5.17
  • 128.42.5.67

The easiest thing to do is to convert both to binary and look for the longest string of network-bits from the left-hand side of the address.

128.42.5.17     in binary: 10000000 00101010 00000101 00010001
128.42.5.67     in binary: 10000000 00101010 00000101 01000011
                           ^                           ^     ^
                           |                           |     |
                           +--------- Network ---------+Host-+
                             (All bits are the same)    Bits

In this case the maximum netmask (minimum hostmask) would be /25

NOTE: If you try starting from the right-hand side, don’t get tricked just because you find one matching column of bits; there could be unmatched bits beyond those matching bits. Honestly, the safest thing to do is to start from the left-hand side.

Powershell : Useful Commands

Get members of a group:

get-adgroupmember -identity <GROUPNAME>

Get a list of user’s “PasswordLastSet” field has a date greater than 31/01/2000 along with their usernames and email addresses:

get-aduser -filter * -Properties PasswordLastSet | where {$_.passwordLastSet -ge [DateTime] "01/31/2000 00:01 AM"} | Select-Object Name, PasswordLastSet, SamAccountName, EmailAddress

Compare two CSV files for differences:

$refCSV = import-csv .\Source.csv 
$compCSV = import-csv .\Reference.csv 
compare-object -referenceobject $refCSV -DifferenceObject $compCSV | foreach { $_.InputObject}

Iterate over a text file of usernames (one per line) and query AD for some values, printing the useraccount’s containing OU in a easily readable form and output to results.csv:

$usersaffected = "c:\tmp\listofusernames.txt"
$output = foreach ($line in get-content $usersaffected) {get-aduser $line -Properties * | Select @{l='OU';e={$_.DistinguishedName.split(',')[1].split('=')[1]}},"whenCreated", "emailaddress", "passwordLastSet", "distinguishedName"}
$output | export-csv -path c:\tmp\results.csv

Create El Capitan 10.11 USB Installer Disk

  1. Download the installer from the App Store but do not install it, close the installer when it launches
  2. Format an 8GB+ USB Drive in the GUID type and Journaled (not case sensitive) partition
  3. run the command (where Untitled is the name of the partition you created in step 1):
    sudo /Applications/Install\ OS\ X\ El\ Capitan.app/Contents/Resources/createinstallmedia --volume /Volumes/Untitled --applicationpath /Applications/Install\ OS\ X\ El\ Capitan.app --nointeraction
  4. Sit back and relax, smoke a pipe while you wait maybe

django setup with virtualenv & PyCharm

Setting up my django development (basic).

The following steps assuming that Python is installed.

  1. Setup a repository on Bitbucket:
    Ensure my development machine has access via SSH keys
  2. Install pip on development machine:
    https://pip.pypa.io/en/stable/installing.html:
    To install pip, securely download get-pip.py
    Then run the following (which may require administrator access):

    $[sudo] python get-pip.py
  3. Install virtualenv on the development machine:
    https://virtualenv.pypa.io/en/latest/installation.html

    $ [sudo] pip install virtualenv
  4. Create a working directory for the project:
    $mkdir /path/to/project/root
  5. Create the virtual environment to work from:
    $virtualenv /path/to/project/root/env
  6. Activate the virtual environment:
    $source /path/to/project/root/env/bin/activate
  7. Install mysql-python:
    (env)$[sudo]apt-get install python-dev libmysqlclient-dev
    
    (env)$pip install MySQL-python
  8. Install django 1.7:
    (env)$pip install django==1.7

    exit the virtualenv for now by:

    (env)$ deactivate
  9. Install pycharm to manage the project from a GUI:
    First install java:

    $ sudo add-apt-repository ppa:webupd8team/java
    $ sudo apt-get update
    $ sudo apt-get install oracle-java8-installer
    $ sudo apt-get install oracle-java8-set-default

    Next install pycharm:

    Download from https://www.jetbrains.com/pycharm/download/
    or from a directory outside of your project directory:

    $ wget http://download.jetbrains.com/python/pycharm-community-4.5.3.tar.gz

    change to that directory and :

    $ tar -xvf pycharm-community-4.5.3.tar.gz

    now to launch pycharm:

    $ ./pycharm-community-4.5.3/bin/pycharm.sh

 

Django 1.7, mysql-python and the pycharm editor should now be ready to use!

 

Next: Setting up the itstock django project http://www.davegernon.co.uk/techblog/initial-django-setup-for-itstock-project/