Category: Commercial

Windows: remove one item from filesystem ACL

PowerShell snippets to remove a group/user from a filesystem ACL in Windows (and compare how to do it in Linux/FreeBSD/Solaris).

Problem

There may be a folder (with broken inheritance) with files / directories where you want to make sure that there is no specific group / item included.

Solution

Here are some PowerShell snippets to solve this.

Populate $folders with all directories in the current directory:

$folders =  get-childitem . -directory

To test with one folder:

$folders = "X:\path\to\folder"

PowerShell to list folders with BUILTIN\Users in the ACL (to see which item will be affected):

foreach ($dir in $folders) { $value = get-acl $dir | Select-object -ExpandProperty Access | where { $_.IdentityReference -eq "BUILTIN\Users"} | Select -Expand IdentityReference; if ($value) {echo $dir} }

Print ACL of before and “to be” after removal (but not removing anything):

foreach ($item in $folders) { $value = get-acl $item | Select-object -ExpandProperty Access | where { $_.IdentityReference -eq "BUILTIN\Users"} | Select -Expand IdentityReference; if ($value) {echo $item; $ACL = (get-item $item).getAccessControl('Access'); $ACL.SetAccessRuleProtection($true, $true); echo $ACL |Select-object -ExpandProperty Access; $ACL = (get-item $item).getAccessControl('Access'); $ACL.Access | where {$_.IdentityReference -eq "BUILTIN\Users"} |%{$acl.RemoveAccessRule($_)}; echo $ACL |Select-object -ExpandProperty Access } }

Set the ACL (disable inheritance (convert current settings to explicit ACL) and remove BUILTIN\Users):

foreach ($item in $folders) { $value = get-acl $item | Select-object -ExpandProperty Access | where { $_.IdentityReference -eq "BUILTIN\Users"} | Select -Expand IdentityReference; if ($value) {echo $item; $ACL = (get-item $item).getAccessControl('Access'); $ACL.SetAccessRuleProtection($true, $true); Set-Acl -Path $item -AclObject $ACL; $ACL = (get-item $item).getAccessControl('Access'); $ACL.Access | where {$_.IdentityReference -eq "BUILTIN\Users"} |%{$acl.RemoveAccessRule($_)}; Set-Acl -Path $item -AclObject $ACL } }

How would this be solved in Solaris?

setfacl -d <entry> *

How would this be solved in FreeBSD/Linux?

setfacl -x <entry> *

Share/Save

VMware 6.5 and 6.7 update failure – reason and fix

Problem

If you want to update an ESXi host (or you have upgraded the VMware vSphere appliance and try to enable HA on some hosts), sometimes it may be the case that it fails (most likely when you download the update either from VMware via esxcli command and a https source, or local download from the vCenter — instead of having the update on a datastore and pointing esxcli to the datastore). If you investigate this (in the case of failure to enable HA you may not have a look at esxupdate.log, but if you just updated you vCenter appliance and the hosts are still on the old version, just go there and have a look), you may stumble on some VMware knowledge base article which tells that this may be a disk space issue on the ESXi host. If you then have a look at the host, you see plenty of free space.

Reason

You can not see this issue after it has failed, it is only observable during the update process. The reason for this is, that the update process is creating a ramdisk where it tries to store the update temporary (at least if the hypervisor is not installed on something which is detected as a real disk — if you have it installed on a SD card which is detected by the ESXi kernel on a USB connection, then it is not a real disk in this case; the same is true if it is installed on a SSD behind an USB connection, I haven’t found a way to convince/force the installer or the hypervisor after the installation to assume it is a SSD or at least a real disk were it does not need to care about wear-leveling like on a SD card, feel free to add a comment to point me to a howto). This ramdisk is then too small and the update fails.

Solution

Put the update on a datastore and run the update via the esxcli command. If you use the remote source feature of esxcli, you will find the download URL in esxupdate.log. In case the update comes from the vCenter, you can at least find the VIP name in the esxupdate.log and you can use the “find” command to search it in the vCenter shell and then scp it to a (shared) datastore on a/the host.

Share/Save

Solaris: remove unusable SAN disks

Problem

Your Solaris system may have “lost” some SAN disks (for whatever reason). If you can not get them back and arrive at the stage where you want to cleanup (if the system can not do it automatically), you want to have a solution which does not need much thinking about rarely executed tasks.

Solution

for i in $(luxadm -e port | cut -d : -f 1); do
  luxadm -e forcelip $i
  sleep 10
done

for i in $(cfgadm -al -o show_FCP_dev | awk '/unusable/ {print $1}' | cut -d , -f 1); do
  cfgadm -c unconfigure -o unusable_SCSI_LUN $i
done

devfsadm -Cv

If you still have some devices here, you may have to reboot to release some locks or such.

Share/Save

VMware: setting the storage device queue depth (HDS fibre channel disks)

Problem

In 2017 we replaced an IBM storage sysdem with an Hitachi Vantara storage system (actually, we replaced the complete SAN infrastructure). We handled it by attaching both storage systems to VMware (v5.5) and migrating the datastores. A recommendation from Hitachi Vantara was to set the queue depth for fibre channel disks to 64.

Solution

Here is a little script which does that. Due to issues as described in a previous post which caused HA/FA (High Availability / Fault Tolerance) reactions in VMware to trigger, we played it safe and added a little sleep after each change. The script also checks of the queue depth is already set to the desired value and does nothing in this case. It’s small enough to just copy&paste it directly into a shell on the host.

SLEEPTIME=210 # 3.5 minutes  !!!! only if all RDMs on the host are reserved!!!
TARGET_DEPTH=64
for LDEV in $(esxcli storage core device list | grep "HITACHI Fibre Channel Disk" | awk '{gsub(".*\\(",""); gsub("\\).*",""); print}'); do
  echo $LDEV
  DEPTH="$(esxcli storage core device list -d $LDEV | awk '/outstanding/ {print $8}')"
  if [ "$DEPTH" -ne $TARGET_DEPTH ]; then
    echo "   setting queue depth $TARGET_DEPTH"
    esxcli storage core device set -d $LDEV -O $TARGET_DEPTH
    echo "   sleeping $SLEEPTIME"
    sleep $SLEEPTIME
  else
    echo "    queue depth OK"
  fi
done
Share/Save

VMware: blank performance graph issue

Problem / Story

In 2017 we replaced a storage system with another storage system (actually, we replaced the complete SAN infrastructure). We handled it by attaching both storage systems to VMware (v5.5) and migrating the datastores. In this process we stumbled upon issues which made some hosts unresponsive in VCenter (while the VMs were running without issues). Before the hosts went unresponsive, the performance graphs of them started to blank out. So at the moment the issue appeared until it was resolved, any graph continued to advance, but had no values listed in the corresponding timeframe (left = colorful lines, middle = white space, and after the issue was resolved the colorful lines appeared again). Some times the issue of the blank performance graph resolved itself, sometimes the hosts became unresponsive and VCenter greyed them out and triggered a HA/FT (High Availability / Fault Tolerance) reaction.

Root cause

On the corresponding hosts we had RDMs (Raw Device Mappings) which are used by Microsoft Cluster Service (there is a knowledge-base article). The issues showed up when we did some SAN operations in VMware (like (automatic) scanning) of new disks after having presented new disks to VMware. VMware tried to do something clever with the disks (also during the boot of a host, so if you use RDMs and booting the host takes a long time, you are in the situation I describe here). If only a small amount of changes happened at the same time, the issues fixed itself. A large amount of changes caused a HA/FT reaction.

Workaround when the issue shows up

When you see that the performance graphs start to show blank space and your VMs are still working, go to the cluster settings and disable vSphere HA (High Availability): cluster -> “Edit Settings” -> “Cluster Features” -> remove checkmark in front of “Turn On vSphere HA”. Wait until the graph shows some values again (for all involved hosts) and then enable vSphere HA again.

Solution

To not have this issue show up at all, you need to change some settings for the devices on which you have the RDMs. Here is a little script (small enough to jsut copy&paste it into a shell on the host) which needs the IDs of the devices which are used for the RDMs (attention, letters need to be lowercase) in the “RDMS” variable. As we did that on the running systems, and each change of the settings caused some action in in he background which made the perfromance graph issue to show up, there is a “little” sleep between the making the changes. The amount of sleep depends upon your situation, the more RDMs are configured, the bigger it needs to be. For us we had 15 of such devices and a sleep of 20 minutes between each change was enough to not trigger a HA/FT reaction. The amount of time needed in the end is much lower than in the beginning, but as this was more or less an one-off task, this simple version was good enough (it checks if the setting is already active and does nothing in this case).

For our use case it was also beneficial to the the path selection policy to fixed, so this is also included in this script. Your use case may be different.

SLEEPTIME=1200              # 20 minutes per LDEV!
# REPLACE THE FOLLOWING IDs   !!! lower case !!!
RDMS="1234567890abcdef12345c42000002a2 1234567890abcdef12345c42000003a3 \
1234567890abcdef12345c42000003a4 1234567890abcdef12345c42000002a5 \
1234567890abcdef12345c42000002a6 1234567890abcdef12345c42000002a7 \
1234567890abcdef12345c42000003a8 1234567890abcdef12345c42000002a9 \
1234567890abcdef12345c42000002aa 1234567890abcdef12345c42000003ab \
1234567890abcdef12345c42000002ac 1234567890abcdef12345c42000003ad \
1234567890abcdef12345c42000002ae 1234567890abcdef12345c42000002af \
1234567890abcdef12345c42000002b0"

for i in $RDMS; do
  LDEV=naa.$i
  echo $LDEV
  RESERVED="$(esxcli storage core device list -d $LDEV | awk '/Perennially/ {print $4}')"
  if [ "$RESERVED" = "false" ]; then
    echo "   setting prerennially reserved to true"
    esxcli storage core device setconfig -d $LDEV --perennially-reserved=true
    echo "   sleeping $SLEEPTIME"
    sleep $SLEEPTIME
    echo "   setting fixed path"
    esxcli storage nmp device set --device $LDEV --psp VMW_PSP_FIXED                    
  else
    echo "    perennially reserved OK"
  fi
done

Share/Save