Blog - Mathieu Chateau

Trying to deploy SQL Server 2012, I had the following error:

 

Clicking on Failed:

 

Resolution

A GPO changed members of this privilege, and wasn’t incluing mine. I added the install account to this privilege to fix it.

This Nagios plugin check Hyper-V 2012 server health through PowerShell + NRPE (nsclient++):

• Execute and retrieve BPA scan result (Best Practice Analyzer)
• Check that all VM are running
• Check that no VM are in any state other than “Operating normally”

Warning and Critical state are triggered by the sum of:

• BPA alerts
• # VM in unhealthy state

VM (models…) can be excluded through an argument

A new BPA scan is triggered at every execution, but new result is retrieved next time (async) (take too much time).

Provided performance data:

• # VM stopped,
• # VM unhealthy,
• # BPA errors
• # BPA warning

Tested setup

Linux:

• Centos 6.4 x64
• Nagios 3.4.4
• check_nrpe 2.13
• Centreon 2.4.2

Windows:

• Windows Server 2012 (needed to have PowerShell Hyper-V module)
• nsclient++ 0.4.1 x64

Script argument

• excludeVM (comma separated)
• maxWarn (Warning if aboce) (1 by default)
• maxError (Criticial if above) (5 by default)

maxWarn and maxCrit must be integer

Usages samples

Directly in PowerShell:

PS C:Program FilesNSClient++scripts>. .lotp_check_hyper-v.ps1
WARNING: 'VM Not Running:'2 'VM issues:'0 BPA Error:1 BPA Warning:12|VM_Not_Running=2 VM_issues=0 BPA_Error=1 BPA_Warning=12
PS C:Program FilesNSClient++scripts>

Through NRPE:

[root~]# /usr/lib64/nagios/plugins/check_nrpe -H myMonitoredServer -n -c check_hyper-v
WARNING: 'VM Not Running:'2 'VM issues:'0 BPA Error:1 BPA Warning:12|VM_Not_Running=2 VM_issues=0 BPA_Error=1 BPA_Warning=12
[root~]#

Install:

On Windows:

• Enable powershell script execution without signed : Set-ExecutionPolicy RemoteSigned
• copy script in folder C:Program FilesNSClient++scripts
• Add to nsclient.ini:
  • [/settings/external scripts/wrapped scripts]
  • check_hyper-v=lotp_check_hyper-v.ps1 -excludeVM $ARG1$ -maxWarn $ARG2$ -maxError $ARG3$

Setup:

On Centreon, by adding this command:

$USER1$/check_nrpe -H $HOSTADDRESS$ -n -t 60 -c check_hyper-v -a $ARG1$ $ARG2$ $ARG3$

Download

lotp_check_hyper-v.ps1

Source code:

# ====================================================================
# Ping a list of targets through NRPE
# Author: Mathieu Chateau - LOTP
# mail: mathieu.chateau@lotp.fr
# version 0.1
# ====================================================================

#
# Require Set-ExecutionPolicy RemoteSigned.. or sign this script with your PKI 
#

# ============================================================
#
#  Do not change anything behind that line!
#
param 
(
	[string]$targets,
	[int]$maxWarn = 1,
	[int]$maxError = 5
)

$output=""
$exitcode=2
$countOK=0
$countKO=0
$targetsArray=@()
$targetsArray=$targets -split(' ')
Remove-Job -Name * -Confirm:$false -Force
foreach($t in $targetsArray)
{
	Start-Job -Name $t -ArgumentList $t -ScriptBlock {param($t);if(Test-Connection -ComputerName $t  -Count 2 -Quiet -ErrorAction SilentlyContinue){return $true}else{return $false}} |Out-Null
}
while(Get-Job -State Running)
{
	Start-Sleep -Milliseconds 500
}
foreach ($job in Get-Job)
{
	$temp=Receive-Job -Name $job.Name
	if($temp)
	{
		$countOK++
	}
	else
	{
		$countKO++
		$output+=$job.Name+" - "
	}
}
if ($countKO -gt $maxError)
{
	$state="CRITICAL"
	$exitcode=2
}
elseif ($countKO -gt $maxWarn)
{
	$state="WARNING"
	$exitcode=1
}
else
{
	$state="OK"
	$exitcode=0
}

$output=$state+":"+$countOK+" online"+" - "+$countKO+" offline - "+$output
$output+='|'
$output+="online="+$countOK+";"+$maxWarn+";"+$maxError+";"+" "
$output+="offline="+$countKO+";"+$maxWarn+";"+$maxError+";"
Write-Host $output
exit $exitcode

While writing a Nagios plugin on Active Directory health, dcdiag /e /c got the good idea to give this alert:

Starting test: VerifyEnterpriseReferences

The following problems were found while verifying various important DN

references. Note, that these problems can be reported because of

latency in replication. So follow up to resolve the following

problems, only if the same problem is reported on all DCs for a given

domain or if the problem persists after replication has had

reasonable time to replicate changes. 

 [1] Problem: Missing Expected Value

Base Object: CN=myDC,OU=Domain Controllers,DC=mydomain,DC=net

Base Object Description: "DC Account Object"

Value Object Attribute Name: msDFSR-ComputerReferenceBL

Value Object Description: "SYSVOL FRS Member Object"

Recommended Action: See Knowledge Base Article: Q312862

 

Article Q312862 is not really helpful, but the underlying problem is true.

Since Windows 2008, AD replication is meant through DFS-R and not FRS anymore. But it implies to have only DC > Windows Server 2003.

So if your DC are updated enough, you can use dfsrmig to migrate, which main commands are:

Check global state:

PS C:usersmchateauDesktop> dfsrmig /GetGlobalState

Current DFSR global state: 'Start'

Succeeded.

Values can be:

0 'Start'
1 'Prepared'
2 'Redirected'
3 'Eliminated'

==>if you are already in DFS-R, state is Eliminated.

Check migration state:

PS C:usersmchateauDesktop> dfsrmig /GetMigrationState

All domain controllers have migrated successfully to the Global state ('Start').
Migration has reached a consistent state on all domain controllers.
Succeeded.

 

Go from start to Prepared:

PS C:usersmchateauDesktop> dfsrmig /SetGlobalState 1

Current DFSR global state: 'Start'

New DFSR global state: 'Prepared'

Migration will proceed to 'Prepared' state. DFSR service will copy the contents of SYSVOL to SYSVOL_DFSR folder.
If any domain controller is unable to start migration, try manual polling.
Or run with option /CreateGlobalObjects.
Migration can start anytime between 15 minutes to 1 hour.
Succeeded.

 

Check state (in progress):

PS C:usersmchateauDesktop> dfsrmig /GetMigrationState

The following domain controllers have not reached Global state ('Prepared'):

Domain Controller (Local Migration State) - DC Type
===================================================

myDC01 ('Start') - Writable DC
myDC02 ('Start') - Writable DC
myDC03 ('Start') - Primary DC
myDC04 ('Start') - Writable DC

Migration has not yet reached a consistent state on all domain controllers.
State information might be stale due to Active Directory Domain Services latency.
PS C:usersmchateauDesktop>

 

You just have to go through the last two steps, turn by turn:

dfsrmig /SetGlobalState 2
dfsrmig /SetGlobalState 3

 

In the end:

PS C:usersmchateauDesktop> dfsrmig /GetMigrationState
All domain controllers have migrated successfully to the Global state ('Eliminated').
Migration has reached a consistent state on all domain controllers.
Succeeded.

Multithreaded multiple ping targets from a remote server through nrpe / nsclient.

For example, if you have a remote site with one server, you can then use it to ping different targets (local switch, google…).

Return number of online/offline targets.

the multithreaded allow a quick execution even with high number of targets, even if they are offline.

typical execution time: 1.75 seconds per target, in average

Provide performance data to get the ratio offline/online.

Can specify number of target that must be offline before triggering warning and critical state.

Tested setup

Linux:

• Centos 6.4 x64
• Nagios 3.4.4
• check_nrpe 2.13
• Centreon 2.4.2

Windows:

• Windows Server 2008 R2 / 2012
• nsclient++ 0.4.1 x64 et x86
• Servers Core & GUI

Script arguments

• targets (separated by commas, MANDATORY)
• maxWarn (Warning if offline number above) (1 par défaut)
• maxError (Critical if offline number above) (5 par défaut)

maxWarn et maxCrit must be integer

Sample usages

Directly in PowerShell:

PS C:Program FilesNSClient++scripts>. .lotp_check_multiping.ps1 www.google.com,www.lotp.fr 1 2 
OK:2 online - 0 offline - |online=2;1;2; offline=0;1;2;
PS C:Program FilesNSClient++scripts>

Through NRPE:

[root~]# /usr/lib64/nagios/plugins/check_nrpe -H myMonitoredServer -n -c check_multiping -a www.google.com,www.lotp.fr 1 2
OK:2 online - 0 offline - |'online'=2;1;2 'offline'=0;1;2
[root~]#

Install:

On Windows:

• Enable powershell script execution without signed : Set-ExecutionPolicy RemoteSigned
• copy script in folder C:Program FilesNSClient++scripts
• Add to nsclient.ini:
  • [/settings/external scripts/wrapped scripts]
  • check_multiping=lotp_check_multiping.ps1 -targets $ARG1$ -maxWarn $ARG2$ -maxError $ARG3$

Setup:

On Centreon, by adding this command:

$USER1$/check_nrpe -H $HOSTADDRESS$ -n -t 60 -c  check_multiping -a $ARG1$ $ARG2$ $ARG3$

Download

lotp_check_multiping.ps1

(remove .txt at the end)

Source code :

# ====================================================================
# Ping a list of targets through NRPE
# Author: Mathieu Chateau - LOTP
# mail: mathieu.chateau@lotp.fr
# version 0.1
# ====================================================================

#
# Require Set-ExecutionPolicy RemoteSigned.. or sign this script with your PKI 
#

# ============================================================
#
#  Do not change anything behind that line!
#
param 
(
	[string]$targets,
	[int]$maxWarn = 1,
	[int]$maxError = 5
)

$output=""
$exitcode=2
$countOK=0
$countKO=0
$targetsArray=@()
$targetsArray=$targets -split(' ')
Remove-Job -Name * -Confirm:$false -Force
foreach($t in $targetsArray)
{
	Start-Job -Name $t -ArgumentList $t -ScriptBlock {param($t);if(Test-Connection -ComputerName $t  -Count 2 -Quiet -ErrorAction SilentlyContinue){return $true}else{return $false}} |Out-Null
}
while(Get-Job -State Running)
{
	Start-Sleep -Milliseconds 500
}
foreach ($job in Get-Job)
{
	$temp=Receive-Job -Name $job.Name
	if($temp)
	{
		$countOK++
	}
	else
	{
		$countKO++
		$output+=$job.Name+" - "
	}
}
if ($countKO -gt $maxError)
{
	$state="CRITICAL"
	$exitcode=2
}
elseif ($countKO -gt $maxWarn)
{
	$state="WARNING"
	$exitcode=1
}
else
{
	$state="OK"
	$exitcode=0
}

$output=$state+":"+$countOK+" online"+" - "+$countKO+" offline - "+$output
$output+='|'
$output+="online="+$countOK+";"+$maxWarn+";"+$maxError+";"+" "
$output+="offline="+$countKO+";"+$maxWarn+";"+$maxError+";"
Write-Host $output
exit $exitcode

For people using RSS or that didn’t notice, i just published a page about tools i use.

Enjoy

Check local time against a provided source or AD(autodetect) or pool.ntp.org through NRPE / nsclient++.

Windows Server that are not DC can’t be tested through NTP by default.
This script is executed locally and compare time with w32tm tool.

Workflow:

• If reference server is provided, it’s used directly.
• If no reference server is provided:
• if domain joined:
  • Try to use logonserver variable, to get closest live DC.
  • Else auto find a DC (random) which is not itself and compare time.
  • Else, use pool.ntp.org
• If workgroup:use pool.ntp.org

Provide performance data (offset jig) to get graphs

tested setup

Linux:

• Centos 6.4 x64
• Nagios 3.4.4
• check_nrpe 2.13
• Centreon 2.4.2

Windows:

• Windows Server 2003 / 2008 R2 / 2012
• nsclient++ 0.4.1 x64 et x86
• Servers Core & GUI

Script arguments:

• refTimeServer (AD autofind or pool.ntp.org by default)
• maxWarn (Warning if above, in second) (1 by default)
• maxError (Critical if above, in second) (5 by default)

 

maxWarn and maxCrit but me integer.
The time is compared in absolute, so it works either the shift is positive or negative

Sample usages

Directly in PowerShell:

PS C:Program FilesNSClient++scripts> . .lotp_check_time.ps1 
OK:-00.0307684s - checked against pool.ntp.org|offset=-00.0307684s;1;5;
PS C:Program FilesNSClient++scripts>

Through NRPE:

[root~]# /usr/lib64/nagios/plugins/check_nrpe -H myserver -n -c check_time -a "myPDCServer" 1 5 
OK:+00.0671837s - checked against myPDCServer|'offset'=0.06718s;1;5
[root~]#

Install:

On Windows:

• Enable powershell script execution without signed : Set-ExecutionPolicy RemoteSigned
• copy script in folder C:Program FilesNSClient++scripts
• Add to nsclient.ini:
  • [/settings/external scripts/wrapped scripts]
  • check_time=lotp_check_time.ps1 -refTimeServer $ARG1$ -maxWarn $ARG2$ -maxError $ARG3$

Setup:

On Centreon, by adding this command:

$USER1$/check_nrpe -H $HOSTADDRESS$ -n -c check_time -a "$ARG1$" $ARG2$ $ARG3$

Download

(remove .txt at the end)

lotp_check_time.ps1

 

# ====================================================================
# Check time against DC or specified server through NRPE / w32tm
# Author: Mathieu Chateau - LOTP
# mail: mathieu.chateau@lotp.fr
# version 0.1
# ====================================================================

#
# Require Set-ExecutionPolicy RemoteSigned.. or sign this script with your PKI 
#

# ============================================================
#
#  Do not change anything behind that line!
#
param 
(
	[string]$refTimeServer,
	[int]$maxWarn = 1,
	[int]$maxError = 5
)

$output=""
$exitcode=2
$random=
if(($refTimeServer -eq $null) -or ($refTimeServer -eq "") -or ($refTimeServer -eq " "))
{
	$refTimeServer=$env:LOGONSERVER -replace ('\',"")
	if(($refTimeServer -match "^$|^ $") -or ($env:LOGONSERVER -match $refTimeServer))
	{
		if((gwmi win32_computersystem).partofdomain -eq $true)
		{
			#Must use select and not .Name directly. If some DC are down, command will be empty with .Name
			$fromAD=(([System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain() | % { $_.DomainControllers }))| select Name 
			foreach ($entry in $fromAD)
			{
				if(($env:COMPUTERNAME -match $entry) -or ($entry -match $env:COMPUTERNAME))
				{
					#this server is a DC, can't check time against itself
					$fromAD.Remove($entry)
				}
			}
			if($fromAD.Count -gt 1)
			{
				#get a random DC from AD, as no server provided and no logon server could be found
				$refTimeServer=(Get-Random -InputObject $fromAD).Name
			}
			else
			{
				#only one DC, defaulting to internet
				$refTimeServer="pool.ntp.org"
			}
		}
		else
		{
			#Workgroup but no server to check time against provided. Defaulting to internet to do something
			$refTimeServer="pool.ntp.org"
		}
	}
}

if(($refTimeServer -eq $null) -or ($refTimeServer -eq "") -or ($refTimeServer -eq " "))
{
	#Something bad happened. Should never happen
	Write-Host "CRITICAL: can't auto detect logon server to check against. Need to specify manually using refTimeServer argument"
	exit 2
}

$temp=w32tm /stripchart /computer:$refTimeServer /period:1 /dataonly /samples:1
$temp=($temp | select -Last 1) -replace (".*, ","")
$temp=$temp -replace ("s$","")

if ($temp -match "^(+|-)[0-9]+.[0-9]+$")
{
	$output=$temp+"s - checked against "+$refTimeServer
	if ([math]::abs($temp) -gt $maxError)
	{
		$state="CRITICAL"
		$exitcode=2
	}
	elseif ([math]::abs($temp) -gt $maxWarn)
	{
		$state="WARNING"
		$exitcode=1
	}
	else
	{
		$state="OK"
		$exitcode=0
	}
}
else
{
	$output="Error: - used $refTimeServer as time server - output:$temp"
	$exitcode=2
}

$output=$state+":"+$output+$random+'|'+"offset="+$temp+"s"+";"+$maxWarn+";"+$maxError+";"
Write-Host $output
exit $exitcode