Cisco Nexus 1000v licenses not freed after removing hosts

posted Jan 20, 2016, 10:26 AM by Jayme Snyder   [ updated Jan 20, 2016, 10:27 AM ]

Hosts are new, Cisco 1000v VSM has run out of licensed modules.


Using the "show log" command on the VSM, we could see the log message in plain English

"cannot be inserted because all vem slots are in use"


According to this doc:


64 licensed modules across 512 CPUs can be configured on the Essential edition.


Since some hosts have been deprecated, licenses should be available, but were not since they were in use by the deprecated hosts.


to rectify the issue we deleted the definitions for hosts which were not connected to the Cisco Nexus 1000v VSM using the following process:


Identify VEM modules which are not connected (be sure all expected hosts are connected to the VSM first)


VSM-1# show module vem license-info | include "n/a"

36    2              -                     -                 n/a          

37    2              -                     -                 n/a          

39    2              -                     -                 n/a          

41    2              -                     -                 n/a          

43    2              -                     -                 n/a          

45    2              -                     -                 n/a          

47    2              -                     -                 n/a          

50    2              -                     -                 n/a          

51    1              -                     -                 n/a          

59    2              -                     -                 n/a          

60    2              -                     -                 n/a          

61    2              -                     -                 n/a          

62    2              -                     -                 n/a          

63    2              -                     -                 n/a          

64    2              -                     -                 n/a  


Find the port channel for the module you are going to remove... for example, to remove VEM 36, look for the port channel with ports on Eth36. In this case Po34. Note the(SD) means that the port channel is down... so we shouldn't be impacting traffic by removing it.


VSM-1# show port-channel summa | include Eth36

34    Po34(SD)    Eth      NONE      Eth36/3(r)   Eth36/4(r)  


Switch to config mode


VSM-1# conf t

Enter configuration commands, one per line.  End with CNTL/Z.


Delete the port channels first:

VSM-1(config)# no int po34


Delete the vem definition:

VSM-1(config)# no vem 36

VSM-1(config)# exit



Write the running config to startup



This frees up the available vem slots.


You should now be able to see on an ESXi host that `vemcmd show card` shows the VEM as licensed.

Unified Remote: Media buttons don't work

posted Feb 15, 2015, 10:16 PM by Jayme Snyder

I was having some issues with a Unified Remote server not working - the media buttons would no longer workI couldn't figure out why. Un-installing and re-installing would not work. Solution: uninstall Unified Remote server, then delete "c:\ProgramData\Unified Remote" directory, re-install.

Booting OS-X into Single User Mode on VMware ESXi

posted Jun 21, 2014, 11:08 AM by Jayme Snyder

So, you have an OS-X VM running on ESXi on a MAC mini and you want to boot it into single user mode?
Your PC keyboard and standard CMD+S or Option+S not working for you?

Open the EFI Shell (delete/space at boot, Boot Manager)
Select your partition and run the boot.efi with the -s, -v option or any other boot options you wanted...
cd System\Library\CoreServices\
boot.efi -s

Nexenta / Solaris / Comstar / illumos one port fiber channel target and one initiator

posted Jan 5, 2014, 9:00 PM by Jayme Snyder   [ updated Jan 5, 2014, 9:27 PM ]

So you have a 2 or 4 port HBA or multiple Q-Logic HBAs in the same host and you want to make some ports targets and some ports initiators.. Fear not, it is relatively simple.

In my example, I am using Q-Logic cards which use either the qlc driver (default) for initiator ports, or the qlt driver for the target ports.

To make an FC initiator port into a target port, we simply change the driver. In order to change which driver for a specific port, instead of changing the driver for all of a specific type of hardware, we first must know where the card is on host. We will call this the identify-name...
We can use the modular debugger to list the current devices claimed by the driver and map it back to the node ID.

root@vnexenta:/volumes# mdb -k

Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs mpt sd sockfs ip hook neti sctp arp usba qlc stmf_sbd stmf lofs idm cpc random crypto smbsrv nfs logindmux ptm nsctl sdbc ufs nsmb sv rdc ii ]

> ::devbindings -q qlc

ffffff018abd7040 pciex1077,2432, instance #0 (driver name: qlc)

ffffff018abd4cf0 pciex1077,2432, instance #1 (driver name: qlc)

> $q

Okay so we see that QLC has claimed two ports.  Now if these are currently FC initiator ports, you can use luxadm to quickly hint to us the identify-name / node path. (If they are not FC initiator ports, read to the bottom)

root@vnexenta:/volumes# luxadm -e port

/devices/pci@0,0/pci15ad,7a0@15/pci1077,138@0/fp@0,0:devctl NOT CONNECTED

/devices/pci@0,0/pci15ad,7a0@15/pci1077,138@0,1/fp@0,0:devctl NOT CONNECTED

Now we can update one of those ports to use the qlt driver as follows using the update_drv command and the devices identify-name, and reboot.

root@vnexenta:/volumes# update_drv -i '/pci@0,0/pci15ad,7a0@15/pci1077,138@0,1' -a qlt

root@vnexenta:/volumes# init 6

So what does update_drv actually do and how does the change persist across reboots?!? It attempts to do its job to the running kernel, then it updates /etc/driver_aliases. In this case it simply appends to the bottom:

qlt "/pci@0,0/pci15ad,7a0@15/pci1077,138@0,1"

Once the host boots back up, we can check who claimed the drivers...

root@vnexenta:/volumes# mdb -k

Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs mpt sd sockfs ip hook neti sctp arp usba qlc stmf_sbd stmf lofs idm cpc random crypto smbsrv nfs logindmux ptm nsctl sdbc ufs nsmb sv rdc ii ]

> ::devbindings -q qlt

ffffff018abd4cf0 /pci@0,0/pci15ad,7a0@15/pci1077,138@0,1, instance #0 (driver name: qlt)

> ::devbindings -q qlc

ffffff018abd7040 pciex1077,2432, instance #0 (driver name: qlc)


And there you have it!

Okay so what if you don't have luxadm or luxadm doesn't show your hardware?? Well fear not, all the info you need to compose the identify name is available in mdb... get the address of your device from loaded driver, figure out how it is connected, then walk the tree for node names that make up the entirety of identify name. Note that the root node in the identify name is assumed and does not need to be looked up. Since we want to replace qlc, the identify-name is a path of the node name from after the root to qlc separated by / .
root@vnexenta:/volumes# mdb -k

Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs mpt
sd sockfs ip hook neti sctp arp usba qlc stmf_sbd stmf lofs idm cpc random crypto smbsrv nfs
logindmux ptm nsctl sdbc ufs nsmb sv rdc ii ]

> ::devbindings -q qlc

ffffff018abd7040 pciex1077,2432, instance #0 (driver name: qlc)

> ffffff018abd7040::prtconf


ffffff018ab78cb0 i86pc (driver name: rootnex)

ffffff018ab787a0 pciex_root_complex, instance #0 (driver name: npe)

ffffff018ab6b7b0 pciexclass,060400, instance #0 (driver name: pcieb)

ffffff018abd7040 pciex1077,2432, instance #0 (driver name: qlc)

ffffff018ab78a28 fp, instance #0 (driver name: fp)

> ffffff018abd7040$<devinfo_brief




ffffff018abd7040 224 3 pci1077,138@0 DS_READY

0 0 pciex1077,2432 <S_EVADD,S_NEED_RESET>


> ffffff018ab6b7b0$<devinfo_brief




ffffff018ab6b7b0 184 5 pci15ad,7a0@15 DS_READY

0 0 pciexclass,060400 <S_EVADD,S_NEED_RESET>


> ffffff018ab787a0$<devinfo_brief




ffffff018ab787a0 183 42 pci@0,0 DS_READY

0 0 pciex_root_complex <S_EVADD,S_NEED_RESET>



So in this example, the identify name for update_drv is /pci@0,0/pci15ad,7a0@15/pci1077,138@0
This information can also be handy when looking to replace other drivers in Solaris derived environments.


Do not use Jumbo Frames, Do not set MTU to 9000, Do not collect $200

posted Jan 5, 2014, 7:36 PM by Jayme Snyder   [ updated Jan 5, 2014, 7:39 PM ]

I'm seeing so often storage performance issues where someone has configured jumbo frames somewhere because they were following some best practices guide, but it seems they only did it on the one device who's doc they were reading... or better yet they think that Jumbo frames are so awesome that they should configure them on almost everything. The operative word there is almost. Maybe I am just in the wrong line of work right now but it's depressing. Folks... please remember if you are going to tweak something, it is very important to realize the implications of your tweak. Jumbo frames are great if you have everything on the layer 2 domain configured correctly. If you are doing routing, don't forget to clamp the TCP MSS appropriately before leaving the networks that support Jumbo. In most cases you are only going to get a minimal performance increase with jumbo frames IF you have it configured correctly.

RedHat RHCE Exams!

posted Dec 25, 2013, 8:51 PM by Jayme Snyder   [ updated Dec 25, 2013, 9:12 PM ]

We did the RedHat RHCE RH255 rapid track course at work. It was during our holiday party week and not too many people had time to study.

The exam itself was a practical exam which had a lot of trick questions and curve balls in it. Except for me and this other guy, everyone failed both exams. I think this was because many of the people in the group were very rusty with their Linux administration experience or barely familiar with Linux. The rapid track course didn't give them the training on some of the basic stuff like LVM and some of the stuff they did teach went over their heads. The other guy who managed to pass an exam and get his RHCSA studied.

I had doubts on whether or not I had a high enough mark to pass the second exam, but it turns out I did. Everyone at the end of the week felt they learned a bunch of useful things.

I was a little disappointed that the training course seemed very RedHat focused (using keytool instead of openssl) but didn't seem to go in depth with selinux, but if they did, I bet there would be even fewer RHCEs.

In the new year, I plan to some VMware certs and maybe the Cisco CCNP.

Serial port console terminal FreeBSD & Solaris

posted Dec 24, 2013, 5:54 AM by Jayme Snyder

I know a lot of people who forget about the built-in tip command when trying to fire up a console session:

# tip -115200 /dev/cua0

Where -115200 is the speed/character rate and /dev/cua0 is the console port character device.
In tip, to exit/break out of the session, pres <enter>~. or if you are in a nested ssh session, <enter>~~.

No need to install minicom or the like.

Connecting HP-UX to FC Comstar Target / Nexenta

posted Dec 23, 2013, 5:09 PM by Jayme Snyder   [ updated Dec 25, 2013, 9:14 PM ]

So I was playing around with my IA64 HP-UX servers trying to get it to connect to my Nexenta CE based fibre channel SAN and was having troubles.
fcmsutil shows:

# fcmsutil /dev/fcd0
Vendor ID is = 0x1077
Device ID is = 0x2312
PCI Sub-system Vendor ID is = 0x103C
PCI Sub-system ID is = 0x12BA
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.3.175
ISP Chip version = 3
Link Speed = 2Gb
Local N_Port_id is = 0x000001
Previous N_Port_id is = 0x000001
Local Loop_id is = 125
N_Port Node World Wide Name = 0x50060b0000331891
N_Port Port World Wide Name = 0x50060b0000331890
Switch Port World Wide Name = N/A
Switch Node World Wide Name = N/A
N_Port Symbolic Port Name = hpux1_fcd0
N_Port Symbolic Node Name = hpux1_HP-UX_B.11.31
Driver state = ONLINE
Hardware Path is = 0/4/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
NPIV Supported = NO
Driver Version = @(#) fcd B.11.31.1311 Nov 5 2013

# fcmsutil /dev/fcd0 get remote all
Target N_Port_id is = 0x0000ef
Target Loop_id is = 0
Target Port World Wide Name = 0x2103001b326e962a
Target port is in invalid state

and on Nexenta..

root@nexenta:/volumes# fcadm remote-port -p 2103001B326E962A -l
Remote Port WWN: 50060b0000331890
Active FC4 Types:
SCSI Target: unknown
Port Symbolic Name:
Node WWN: 50060b0000331891
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0

I tried many things wasting many hours - tried in both FCAL and Fabric connections (hard and soft zoning) but this is as far as I have gotten. HP-UX will not detect the LUN and just shows target as "Target port is in invalid state". Since this is a hobby I always seem to google last. I mean, this is just supposed to work and if it doesn't, I didn't think I'd find much in the way of information on it anyway. I did end up finding this gem of a mailing list post though.

[prev in list] [next in list] [prev in thread] [next in thread]
List: opensolaris-storage-discuss Subject: Re: [storage-discuss] COMSTAR and AIX, HP-UX/PA support From: Albert Chin <opensolaris-storage-discuss () mlists ! thewrittenword ! com> Date: 2009-09-22 6:17:02 Message-ID: 20090922100911.GB13517 () songoku ! il ! thewrittenword ! com [Download message RAW]
On Fri, Sep 18, 2009 at 04:01:02PM -0400, Peter Cudhea wrote:
> Were you able to make progress with this? Based on a quick scan of the
> code, you may need to look at the following links:
> 1) SCMD_WRITE_VERIFY.  I am not an expert in this area of the code,
> but it looks like you could add support for SCMD_WRITE_VERIFY to the
> following routine.  You may also need to modify the code that calls
> this routine.
> comstar/lu/stmf_sbd/sbd_scsi.c#472
Haven't started yet. Will probably start in two weeks.
> Would you be willing to open an RFE in category solaris/storage_sw/sbd
> that details the requirement for the SCMD_WRITE_VERIFY command and
> perhaps how to test it?   Or would you like to have a sponsor so that
> you can put back the change yourself?   We cannot test all the
> initiators out there, and simple changes like this that allow us to be
> compatible with an important class of initiators are worth doing.
I don't have a problem with this but our current solution is simply a
hack, treating SCMD_WRITE_G1 like SCMD_WRITE_VERIFY (which basically
makes SCMD_WRITE_VERIFY just a WRITE with no VERIFY). I'll see if we
have time to produce a "correct" version that fully supports VERIFY.
And, if we do, we would like to push the code back upstream. When we
have something, I'll ping.
> 2) Including the port number in the SendTargets response.  Looks like it is
> already handled:
> comstar/port/iscsit/iscsit_text.c#iscsit_add_portal
Great, thanks.
albert chin (
storage-discuss mailing list
Annnnnnd then nothing of interesting value...

So I'm considering just shelving the whole HP-UX cluster until I either find a use for it or get a proper SP. I mean I am trying real hard to find a use for them but for now, in my basement, Itanium is dead.

---EDIT--- 2013-12-23 10:54PM 
I configured the iSCSI initiator and added the iqn into the initiator group in Nexenta and now it seems the LUN is showing up in ioscan... I don't understand how only adding the IQN into the initiator group on the Nexenta fixed it - the WNN was in there and it didn't work... oh well!

# /usr/sbin/ioscan
H/W Path       Class                                 Description
0/4/1                 slot                           PCI Slot
0/4/1/0                  fc                          HP A6826-60001 2Gb Dual Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
0/4/1/0.8                   fcp                      FCP Protocol Adapter
0/4/1/                            ext_bus         FCP Device Interface
0/4/1/                             target
0/4/1/                              disk      NEXENTA COMSTAR
255/0              iscsi                             iSCSI Virtual Node
255/0/0.0                ext_bus                     iSCSI-SCSI Protocol Interface
255/0/0.0.0                 target
255/0/                  disk                  NEXENTA COMSTAR

Now to play - Long live the Itanium!


Home Automation

posted Dec 6, 2012, 4:46 PM by Jayme Snyder

After moving back from the U.A.E., we have purchased a small little bungalo in Kitchener, Ontario. In an effort to make it mine and save energy, I have decided to invest in some DIY smart home technology.

First off, I have purchased a Radiostat thermostat. This thermostat is WiFi connected and has cloud based control (boo) and a local API (yeay!). I have begun using the local API through a Soekris net-4801 running Debian. I have written some PHP scripts that control the automation.

The first is the poller. It polls the thermostat (every minute) and the local weather station data at the University of Waterloo (every fifteen minutes)  and places the data in a postgresql database only if the data has changed from last poll. 

The second script is the event script. It will check for certain events (calendar events/presence indicators/weather data events/etc) and take appropriate actions (adjust thermostat/email alerts/control lighting/etc.).

Finally, there is a data export script that will reformat the data into standard minute intervals (averaging values from the irregular intervals stored in the database which I will then display using HighCharts. The results (below) in MS Excel already look promising.

I have investigated some other inputs: 
DSC alarm panel - I am purchaing an IT-100 for integration with my DSC-1864 and existng sensors. This will help me determine presence and also control/give status for my garage doors.
Dallas Semiconductor 1-wire bus solution from HW-Group they call Poseidon, but it came to almost $2K CAD for only a hadful of temp/humidity sensors. I want to monitor my fridge as well as automate the bathroom exhaust fans (to prevent mould/mildew)
For outputs:
I am going to buy some insteon ceiling fan controllers and other insteon gear. I will make my outdoor lights run off a combination of moton sensors (data from DSC alarm panel), sun-times (dim after dark) and solar radiation data from the University of Waterloo weather station.

Land Rover Discovery II Door Latch

posted Jul 2, 2011, 9:46 PM by Jayme Snyder   [ updated Jul 2, 2011, 11:31 PM ]

I am the proud owner of a 2001 Land Rover discovery. It's a great off-road toy, but one thing that has always annoyed me since I got it (second hand) is that the rear drivers side door always took a lot of force to open... and it was slowly getting worse. I decided to fix it. To fix it I needed a tool kit with torx screw bits, a flexible screwdriver attachment, allen key and small hands.
This is how to go about it: 

First, remove the door trim. There are 3 screws (one on door latch release trim and two on the handles) and a bunch of clips.
the clips by slowly prying the door panel away. Try to pull equally from both sides of the clips so the door panel comes straight away from the door. If you don't you will probably break the clips or where the panel mounts to the clips.

Remove the door latch release trim. The door latch release will fit through the hole as you rotate it through.
Lock the door.

Unclip the power window and speaker wires, and remove the trim panel, careful not to bend the door lock rod.

Unclip the connector for the power locks. You should now be able to see the door latch assembly. Watch what happens when you pull the cable and the external door latch.

The action should look something like this (picture upside down, black torx scre removed), the solid lines should move as a single piece.

In my case, when you pulled the external door latch, it pulled the rod, but there was a lot of play before the latch that moved with the cable release moved.
This play was caused by an alen bolt that is used to adjust the pull distance of the rod for the external door release.

To get at the allen bolt, you could either remove the entire window assembly, allowing you to remove the latch assembly or you can work with the latch assembly in the door. I chose the latter option.
I removed the lock rod and external release rod (don't drop it in the door). Then undid the screws for external door lever/handle. To remove the handle, slide it towards the front of the truck(it may need a nudge), then gently pull it out from the back of the truck as if you were opening the door.

I then released the 3 screws that hold in the latch, carefully oriented it inside the door as so that I could remove the black torx screw with a flexible attachment screwdriver extension.

Once the black torx screw was removed, I exposed the loose allen bolt,

adjust the mechanism as appropriate, tightened, test fit, and re-assembled.

It appears that this might be a little bit of a design aggravated issue, as forces from opening and closing the door could back off an insufficiently tightened adjustment bolt. Although the issue appears to be fixed for me, I think maybe I could have put some lock tight on the bolt or used a lock washer. From factory it probably should have had a lock washer. Installing one while the assembly is in the door will likely result in pieces falling into oblivion. Anyway, hopefully I will never do this again.

1-10 of 18