Exinda 4010 re-purposing with Ubuntu (or Debian)

I wanted a quick and dirty route reflector for my home lab (more on that to come) and ended up repurposing an old Exinda 4010 WAN optimiser I had lying about. This unit is really just a re-branded Lanner FW-8756 (refer user manual), a generic x64-64 platform in a 1U rackmount case. It features six Intel gigabit Ethernet ports which was perfect for my intended use case.

It’s certainly not fancy; in the unit I have we find:

  • Intel Pentium E5300 CPU (dual core, x86-64, VT-x virtualisation support)
  • 4GB RAM (expandable to 8GB by adding another DIMM although more might be possible).
  • 500GB SATA HDD (replaceable – the bracket can also has holes to accommodate 2x 2.5″ HDDs but standoffs are required for the SATA cables to clear the motherboard).
  • 100Mbps Realtek management port with PXE support
  • 6x Intel 82574L Gigabit Ethernet ports
  • 2x front panel USB 2.0 ports (plus an internal header for more ports)

In this case my intention was to run a more generic distro on it and install Free Range Routing. In my case I’ve opted for Ubuntu 20.04 LTS as it’s popular, well supported and super easy to build an image for. Pretty much any Linux distro will run on it though. Preparing a basic image to copy directly to headless devices like this is pretty straightforward, and much easier than the more common traditional method connecting a monitor to the motherboard headers and booting off an external drive.

This post will focus on getting this specific unit running as a generic Linux box, as well as explaining the process used for building an image that should work with minimal effort on a large range of similar devices.

I’ve also since tried this on a Watchguard M400 which is another rebranded Lanner product with a similiar design. The only major difference to the steps below being that there is no LAN bypass feature to worry about and the network interface name for the first Gigabit port is enp3s0.

Disable LAN Bypass

When I originally tried running a vanilla distro on the FW-8756 I couldn’t get the gigabit Ethernet ports to detect a link, despite everything looking normal in the OS.

It turns out the FW-8756 is one of several similar models with a “LAN bypass” feature. This consists of a series of bistable relays that can electrically connect each pair of network interfaces on the front panel, bypassing the NICs on the motherboard entirely. The bypass feature is software controllable via an I2C interface.

The FW-8756 platform was designed for use by WAN optimisers (as the Exinda 4010 is), so this bypass feature means that when it fails or is shut down the network link can continue to function as normal, albeit without the optimisation features provided by the unit.

This is all very clever except that it the default state is to go into bypass mode by default and isolate the ports from the internal NICs until switched over by a software included in the original firmware. When we install a generic OS there is no means out of the box to switch this over, and the LAN Bypass settings in the BIOS appeared to have no effect.

Rather than fudge a software fix, it appeared from the board layout that U93 next to the mini PCI slot was likely to be responsible for controlling this feature. There is an unmarked and undocumented jumper on the motherboard near this slot that appears to disable this bypass function permanently, making the network interfaces behave normally. There is an amber LED (LED12) near this jumper that appears to illuminate steadily when the LAN bypass function is operational, and flash when it is in bypass mode.

To make the network interfaces function with a generic Linux distro like Ubuntu you should permanently disable the LAN bypass feature entirely. Move the jumper so that it bridges pins 2-3 (closest to the front panel). The unit must be completely powered off (i.e. physically remove the power cord for at least five seconds) for this to apply.

If LED12 does not illuminate then the bypass feature is properly disabled and the LAN ports will function normally.

Serial Console and BIOS Reset

In the case of the Exinda 4010 I wasn’t able to get a serial console immediately so resorted to resetting the BIOS via the jumper next to the battery. After this I was able to get access to the console at 9600 8N1 using a common Cisco-style serial cable.

The instructions below assume the use of screen as a terminal emulator, although you can use your preferred application (e.g. minicom, HyperTerminal, PuTTY, etc). Ensure you use ANSI terminal mode so that the output displays correctly. Command line terminal emulators typically rely on the TERM environment variable for this, e.g:

TERM=ansi screen /dev/ttyUSB0 9600

When the device is first booted after a BIOS reset the machine will pause for input during POST, requiring F1 to be pressed to enter the BIOS setup. Several options exist:

  • Some terminal emulators will interpret F1 properly – just press the key and away you go.
  • Connecting a USB keyboard to the machine itself and pressing F1 also works as an alternative.
  • If your terminal emulator doesn’t interpret the F1 key, another option is to quit the terminal emulator and run the commands below from a shell. The first command sets the serial port to 9600 baud and the second sends an escape character followed by the letters OP which combine to simulate pressing F1.
stty -F /dev/ttyUSB0 9600
echo -e '\eOP' > /dev/ttyUSB0

To enter the BIOS setup on subsequent boots press the tab key during POST instead of F1.

In most cases (including if you flash the image I’ve described below) I’d recommend setting the following settings in the BIOS to improve serial access:

Advanced -> Remote Access:

  • Serial Port Mode: 115200 8, n, 1
  • Terminal Type: VT100

From this point on you should connect to the serial console at 115200 baud.

Quick Start

If you just want to use the thing and don’t care about making an image yourself, here are the basic steps:

  1. Download the image.
  2. Decompress and write the image directly to a disk:
    zcat ubuntu-focal_exinda4010_amd64.img.gz | sudo dd of=/dev/sdb bs=1M oflag=direct status=progress
  3. Optionally grow the partition size and filesystem to fill the disk:
    sudo parted /dev/sdb resizepart 1 100%
    sudo e2fsck -f /dev/sdb1
    sudo resize2fs /dev/sdb1
  4. Insert the disk into the device and boot it up.
  5. Access the device via serial or SSH.

Creating an Image From Scratch

If you would prefer to create an image from scratch rather than download a pre-made one you can do so using the following method.

As I’m opting for Ubuntu and using debootstrap to create the image, these steps will need to be run on a system running a Debian derivative of some type (including Ubuntu itself).

Install debootstrap on your system:

sudo apt install debootstrap

Now we need to decide on the storage media. You could either make a disk image file and perform all the operations in that, or do all the work directly on a disk.

Disk

If you opt for a disk then connect it to the system you are building the image on (e.g. via a USB adapter) and immediately check dmesg for the device node:

dmesg| tail
[110875.167672] usb 4-1: SerialNumber: 1254201617020
[110875.175233] scsi host7: uas
[110875.176092] scsi 7:0:0:0: Direct-Access     ASMT     2115             0    PQ: 0 ANSI: 6
[110875.176833] sd 7:0:0:0: Attached scsi generic sg2 type 0
[110875.179118] sd 7:0:0:0: [sdb] 31277232 512-byte logical blocks: (16.0 GB/14.9 GiB)
[110875.179248] sd 7:0:0:0: [sdb] Write Protect is off
[110875.179264] sd 7:0:0:0: [sdb] Mode Sense: 43 00 00 00
[110875.179480] sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[110875.179880] sd 7:0:0:0: [sdb] Optimal transfer size 33553920 bytes
[110875.203855] sd 7:0:0:0: [sdb] Attached SCSI disk

In the above example the disk is /dev/sdb but yours may vary slightly. If it is different to sdb then you will need to modify all of the commands in the examples below to suit.

Image File

As an alternative to directly connecting the disk, create a 4GB image file we can using the following command:

 dd if=/dev/zero of=exinda-4010-ubuntu-focal-amd64.img bs=1M count=1 seek=4095

This generates a 4GB “sparse” (also known as thin provisioned) file; although ls will report it as 4GB in size, du will show the disk usage as the actual space used up by the file. This process saves writing out a large number of zeros to fill the file, although at the risk of filling the parent filesystem as the file is written to later if there is insufficient space available.

Now set up the image as a loop device so that we can treat it just like a normal block device (i.e. disk):

sudo losetup --show -f exinda-4010-ubuntu-focal-amd64.img

Note the output of the above command; this is the path to the loopback device (e.g. /dev/loop0).

From here use this path as the path to the disk for the remaining commands in this post. The rest of the process is exactly the same.

Partition and Format

The first step is to create a partition table, partition and format the disk, regardless of whether you opted for an image file or directly writing to disk. Don’t blindly copy and paste these commands – you will need to bear in mind the path to the disk or image file and use what is appropriate for your system.

sudo parted /dev/sdb mklabel msdos

Create a primary partition that fills the disk:

sudo parted /dev/sdb mkpart pri ext2 0% 100%

If you used a disk, the path to the new partition device is likely the same as the disk path with a 1 appended (e.g. /dev/sdb1). If you are using an image attached with a loopback device, the partition path will be the loop device appended with p1 (e.g. /dev/loop0p1). Now create an ext4 filesystem in the new partition:

sudo mkfs.ext4 /dev/sdb1

Mount the new partition to a location on your system. In many cases the /mnt directory is a good choice, unless you’ve already used it for something else in which case you can probably figure out how to make a suitable mount point of your own. ūüôā

sudo mount /dev/sdb1 /mnt

Build the Image

Now use debootstrap to build a base operating system image. The two major arguments you need are:

  • Distro codename (e.g. focal for Ubuntu 20.04 LTS “Focal Fossa”).
  • Path to a mirror to use for downloading packages.

Consult your preferred distro documentation to find out what codenames and mirror locations to use. The command below will work for Ubuntu 20.04 from the New Zealand mirror. If you are not in New Zealand then you may wish to substitute nz. with your own country code, or remove it altogether for the primary Ubuntu mirror:

sudo debootstrap focal /mnt http://nz.archive.ubuntu.com/ubuntu

Once that’s done, mount the /dev tree inside the image so that the disk devices on your system are accessible while in the chroot:

sudo mount -o bind /dev /mnt/dev

Now chroot into the image to get a root shell. Note that we should set the locale to C to avoid warnings being thrown about locale mis-matches when running some commands. We can fix up locales later.

LANG=C sudo chroot /mnt

Temporarily mount /proc and /sys so the chroot environment looks like a normal root file system and other things work as expected:

mount -t proc proc /proc
mount -t sysfs sysfs /sys

Configure and Update the Image

Add an entry for the root filesystem to /etc/fstab (noting the disk path used by the blkid command should match the one you used in previous steps):

cat << EOF > /etc/fstab
UUID=$(blkid /dev/sdb1 | cut -d\" -f2) / ext4 errors=remount-ro 0 1
EOF

If you are using Ubuntu, ensure the updates and security repositories are enabled and then patch the image with the latest packages:

cat << EOF > /etc/apt/sources.list
deb http://nz.archive.ubuntu.com/ubuntu focal main restricted universe multiverse
deb http://nz.archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu focal-security main restricted universe multiverse
EOF

apt update
apt -y dist-upgrade

Install the kernel image, the GRUB bootloader, SSH and some other useful tools for the system:

apt -y install linux-image-generic grub2-common openssh-server bridge-utils ethtool

Clear out the package cache to reduce the image size a little:

apt clean

Add the following config to /etc/default/grub to enable the serial console, we well as disabling the existing quiet boot mode so that kernel messages are logged to the console:

GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8"
 
# Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

Disable the GRUB OS prober – this prevents the GRUB update command from adding entries for operating systems on other disks in the host system:

rm /etc/grub.d/30_os-prober

Update GRUB config to apply the above changes and then write the GRUB bootloader to the disk. Note the disk path again – change this to suit your system if need be.

update-grub
grub-install /dev/sdb

Set the hostname (feel free to change this to anything you like):

echo exinda > /etc/hostname

Generate a basic network config for the management port (enp7s1). We can either use DHCP or set a static IP address – up to you! If you’d rather use the first gigabit network port then change enp7s1 to ens32 in the examples below. Other configurations are also possible.

DHCP Config

Run this command to generate network configuration to obtain an IP address via DHCP:

cat << EOF > /etc/netplan/01-default.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    enp7s1:
      dhcp4: true

Static IP Config

Run this command to generate network configuration to configure a static IP address:

cat << EOF > /etc/netplan/01-default.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    enp7s1:
      addresses:
        - 192.168.1.10/24
      gateway4: 192.168.1.1
      nameservers:
        addresses:
          - 192.168.1.1
          - 192.168.1.2
EOF

The final step is to make a group called admin (this enables sudo access out of the box in Ubuntu) and user account to log in with that is a member of this group. The third command sets the password for this user.

groupadd admin
useradd -s /bin/bash -m -d /home/user -G admin user
passwd user

Finish Up

That’s it! Exit the chroot, then unmount the filesystems before disconnecting the disk:

exit
sudo umount /mnt/{dev,proc,sys} /mnt

If you used an image file instead of writing directly to a disk there are a couple of extra steps. If you intend to compress the image for distribution then you can choose to zeroise any unused blocks from old files that were removed during patching to further improve compression – this almost halved the size of the compressed image in my case. This step is purely optional and a downside is that it will fill up a sparse image file with zeros causing it to take up the entire space on disk:

sudo zerofree /dev/loop0p1

Finally, if you are working on an image file then you should remove the loopback device attached to the image:

sudo losetup -d /dev/loop0

From here the drive can be installed directly into the Exinda appliance. If you opted for an image file then write this to a suitable disk and install that, e.g:

sudo dd if=exinda-4010-ubuntu-focal-amd64.img of=/dev/sdb bs=1M oflag=direct status=progress

Filesystem Expansion

The image I created is only 4GB, so if you want to use all of the space on a bigger disk it’s pretty easy to expand this after writing it to a disk.

First resize the partition:

sudo parted /dev/sdb resizepart 1 100%

Then force a filesystem check and resize it:

sudo e2fsck -f /dev/sdb1
sudo resize2fs /dev/sdb1

Testing

Connect a serial cable and start session at 115600 baud using your favourite terminal emulator, e.g:

screen /dev/ttyUSB0 115200

Make sure the system boots up and that you can login using the credentials supplied when making the image.

Alternatively you should be able to access the unit via SSH on the management interface.

From here it’s just another Linux box so do with it what you please!

Footnote: Exinda 4010 Hardware Info

A couple of quick hardware stats from the Exinda 4010 for those interested:

lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              2
On-line CPU(s) list: 0,1
Thread(s) per core:  1
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               23
Model name:          Pentium(R) Dual-Core  CPU      E5300  @ 2.60GHz
Stepping:            10
CPU MHz:             1227.586
BogoMIPS:            5186.89
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            2048K
NUMA node0 CPU(s):   0,1
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm xsave lahf_lm pti tpr_shadow vnmi flexpriority dtherm

lspci
00:00.0 Host bridge: Intel Corporation 4 Series Chipset DRAM Controller (rev 03)
00:02.0 VGA compatible controller: Intel Corporation 4 Series Chipset Integrated Graphics Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation NM10/ICH7 Family PCI Express Port 1 (rev 01)
00:1c.1 PCI bridge: Intel Corporation NM10/ICH7 Family PCI Express Port 2 (rev 01)
00:1c.2 PCI bridge: Intel Corporation NM10/ICH7 Family PCI Express Port 3 (rev 01)
00:1c.3 PCI bridge: Intel Corporation NM10/ICH7 Family PCI Express Port 4 (rev 01)
00:1c.4 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 5 (rev 01)
00:1c.5 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 6 (rev 01)
00:1d.0 USB controller: Intel Corporation NM10/ICH7 Family USB UHCI Controller #1 (rev 01)
00:1d.1 USB controller: Intel Corporation NM10/ICH7 Family USB UHCI Controller #2 (rev 01)
00:1d.2 USB controller: Intel Corporation NM10/ICH7 Family USB UHCI Controller #3 (rev 01)
00:1d.3 USB controller: Intel Corporation NM10/ICH7 Family USB UHCI Controller #4 (rev 01)
00:1d.7 USB controller: Intel Corporation NM10/ICH7 Family USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1)
00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface Bridge (rev 01)
00:1f.2 IDE interface: Intel Corporation NM10/ICH7 Family SATA Controller [IDE mode] (rev 01)
00:1f.3 SMBus: Intel Corporation NM10/ICH7 Family SMBus Controller (rev 01)
01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
07:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 10)

Creating a bootable Windows USB installer in Linux

Background

I recently had a case to install Windows Server 2016 on a Dell PowerEdge R630 server. Microsoft make the ISO images available for download on their website, however the Windows Server 2016 ISO clocks in at over 6GB. Actually getting the server to boot and install from this ISO image was surprisingly difficult:

  • It’s too large for a regular DVD-R disc, and I didn’t have any dual-layer discs on hand.
  • The ISO image contains a file (install.wim) that is greater than 4GB, so won’t fit on a FAT32 filesystem as created by tools such as Unetbootin.
  • There is a command in Windows (dism) to split a WIM file but hadn’t come across any Linux tools to date. Update: wimlib can do this and means you can probably do this on a legacy FAT filesystem after all!
  • It’s too large to fit into Dell vFlash which has a image size limit of 4GB (presumably as it uses FAT32 internally).
  • Tools that were known to work such as Rufus and the Microsoft USB imager only run in Windows, which I didn’t have handy.
  • The ISO mounting capability via the server iDRAC console actually worked, but was painfully slow (hours).

So I set about to find a way to make a bootable Windows USB stick when you’ve only got immediate access to a Linux box. This took a few attempts to get right but turns out to be relatively simple. I imagine this will also be perfectly doable on a Mac as well, with a few modified commands.

Process

Here’s the actual process to create a bootable Windows installer on a USB stick from a Windows installer ISO image for a UEFI system. This should work for any Windows version released over the last decade or so. Secure Boot must be disabled for this to work.

WARNING: This will erase any existing data on the USB stick!

You will need:

Plug the USB stick in and run dmesg to see what identifier it gets:

[28959.294103] usb 4-1: new SuperSpeed Gen 1 USB device number 15 using xhci_hcd
[28959.319266] usb 4-1: New USB device found, idVendor=0781, idProduct=5581, bcdDevice= 1.00
[28959.319278] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[28959.319282] usb 4-1: Product: Ultra
[28959.319286] usb 4-1: Manufacturer: SanDisk
[28959.319290] usb 4-1: SerialNumber: 1234567890
[28959.321513] usb-storage 4-1:1.0: USB Mass Storage device detected
[28959.322014] scsi host7: usb-storage 4-1:1.0
[28960.347536] scsi 7:0:0:0: Direct-Access     SanDisk  Ultra            1.00 PQ: 0 ANSI: 6
[28960.348477] scsi 7:0:0:0: Attached scsi generic sg2 type 0
[28960.349336] sd 7:0:0:0: [sdb] 60063744 512-byte logical blocks: (30.8 GB/28.6 GiB)
[28960.350654] sd 7:0:0:0: [sdb] Write Protect is off
[28960.350664] sd 7:0:0:0: [sdb] Mode Sense: 43 00 00 00
[28960.351430] sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[28960.399678]  sdb: sdb1 sdb2
[28960.402932] sd 7:0:0:0: [sdb] Attached SCSI removable disk

In this case the disk shows up as sdb, so the device we’ll use for this exercise is /dev/sdb. Yours may vary depending on how many disks you have connected to the system so you will need to modify the remaining commands accordingly.

The first step is to create a new GPT partition table:

sudo parted /dev/sdb mklabel gpt

Create an exFAT partition for the Windows ISO contents. We’ll set the partition type as NTFS; that is correct for exFAT. For the sake of simplicity I suggest using 0% and 99% as the boundaries as the exFAT driver filesystem image is only 512k. It’s a quick hack that saves calculating partition sizes and also helps ensure that parted correctly block aligns the partitions.

We’ll also need to enable the msftdata flag on the partition and then format it with an exFAT filesystem:

sudo parted /dev/sdb mkpart winsetup ntfs 0% 99%
sudo parted /dev/sdb set 1 msftdata on
sudo mkfs.exfat -n winsetup /dev/sdb1

Mount the Windows ISO and the exFAT USB stick partition. I suggest making some temporary working directory for these:

mkdir -p /tmp/{iso,usb}
sudo mount ~/Downloads/Windows_Server_2016_Datacenter_EVAL_en-us_14393_refresh.iso /tmp/iso -o loop,ro
sudo mount /dev/sdb1 /tmp/usb
cp -r /tmp/iso/* /tmp/usb/

Now create a partition for the exFAT/NTFS driver, and set the msftdata flag on this partition as well:

sudo parted /dev/sdb mkpart uefi-ntfs 99% 100%
sudo parted /dev/sdb set 2 msftdata on

We don’t need to format this one, we’ll just copy the Rufus driver image directly to it:

sudo dd if=~/Downloads/uefi-ntfs.img of=/dev/sdb2 bs=512k count=1

The last step is to clean up. Unmount the partitions and remove the temporary mount point directories:

sudo umount /tmp/{iso,usb} && sudo rmdir /tmp/{iso,usb}

From here you can remove the USB stick and use it to deploy Windows!

Fixing reused certificate issuer and serial in Firefox the proper way

Sometimes when deploying stuff in the lab with self-signed certificates I find myself quickly rebuilding VMs after making breaking changes. Normally that’s not a problem, except when Firefox is being used to access the rebuilt VM occasionally we might see this error:

Secure Connection Failed

An error occurred during a connection to 192.168.24.201. You are attempting to import a cert with the same issuer/serial as an existing cert, but that is not the same cert.

Error code: SEC_ERROR_REUSED_ISSUER_AND_SERIAL

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.

Please contact the website owners to inform them of this problem.

I often see this occur with Foreman where the default configuration relies on the Puppet CA for certificate issuing.

In theory you should be able to fix this by heading to Settings -> Privacy & Security -> View Certificates, selecting the Servers tab and then deleting the offending entry. In practice, however, this doesn’t often work.

Some forum posts suggest removing the entire certificate database. This works, but it’s a bit heavy handed as it clears out any other exceptions you may have saved in the past.

To fix this properly we need to use certutil as included with the Mozilla NSS tools. This command shouldn’t be confused with certutil.exe included with Windows, which is completely unrelated!

In install certutil on Ubuntu we need the libnss3-tools package:

sudo apt install libnss3-tools

In macOS, I recommend using Homebrew or MacPorts to install the nss package e.g:

brew install nss

Quit Firefox if it’s running. In theory it’s possible to do this in-place with SQL NSS databases but I’ve found it a bit hit and miss in practice.

First we need to figure out the Firefox profile directory. The configuration for this is usually in ~/.mozilla/firefox/profiles.ini(Linux) or ~/Library/Application Support/Firefox/profiles.ini (Mac).

Identify the Path to your profile directory, in this case

[Profile0]
Name=default
IsRelative=1
Path=profiles/w6znk57b.default
Default=1

[General]
StartWithLastProfile=1
Version=2

So in this case my profile directory would be ~/.mozilla/firefox/profiles/w6znk57b.default in Linux or ~/Library/Application Support/Firefox/Profiles/w6znk57b.default on Mac. Yours may vary slightly – just note that the Path entry is relative to the location of profiles.ini so you need to include any subdirectories in your commands. Note that the path is case sensitive.

From here we can use certutil to list the certificates (-L) in this store to find the offending entry. Modern versions of Firefox use the SQL NSS database, so we need to provide the location of the database in the form of sql:path.

A tip here – it might be useful to use grep to filter for the common name of the certificate used by the page throwing the error.

certutil -d sql:$HOME/.mozilla/firefox/profiles/w6znk57b.default -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

DigiCert SHA2 Secure Server CA                               ,,   
DigiCert TLS RSA SHA256 2020 CA1                             ,,   
R3                                                           ,,   
Amazon                                                       ,,   
GTS CA 1O1                                                   ,,   
foreman1.lab.example                                         ,,   

In this case the foreman1.lab.example entry is the offending one that we want to remove. To get rid of it we use the certutil delete command (-D) and specify the certificate nickname listed above after the -n argument. I recommend putting quotes around the nickname in case it includes spaces.

certutil -d sql:$HOME/.mozilla/firefox/profiles/w6znk57b.default -Dn "foreman1.lab.example"

If certutil doesn’t return any messages then it’s likely to have worked. To verify we can simply run the list command again to check:

certutil -d sql:$HOME/.mozilla/firefox/profiles/w6znk57b.default -L
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

DigiCert SHA2 Secure Server CA                               ,,   
DigiCert TLS RSA SHA256 2020 CA1                             ,,   
R3                                                           ,,   
Amazon                                                       ,,   
GTS CA 1O1                                                   ,,  

At this point you should be able to reload the offending site and it should now work (although you will likely be prompted to accept the certificate again!).

If the above steps don’t work, try closing Firefox and repeating the delete command without the sql: prefix.

OpenWRT AirPlay server on HP Thin Terminal

HP t5525 thin terminalI decided I needed an easier way of playing music on my old stereo in the workshop so it was time to see what could be done. I’m already quite a fan of Apple’s AirPlay protocol as it is very easy to use, however I wasn’t so keen on its proprietary nature. Fortunately this side of things has been solved and popular FOSS AirPlay server has been developed for Linux called ShairPlay.

The second part of the puzzle was to find some suitable hardware. I wanted something small and low power, preferably without moving part or dongles hanging of ports. Many folks use the Rasberry Pi and while I have  some lying about, the fact it uses PWM for the analogue audio output is a serious let down for music playback.

Diving deeper into the junk box I dug up an old HP t5525 thin terminal. This is really just a standard x86 PC crammed into a little box and designed for low power operation. It’s very well built and has no fans or other moving bits, and best of all it features far better audio performance than the Rasberry Pi. Here’s the basic specs:

  • VIA Eden 800MHz CPU
  • 128MB DDR RAM
  • 256MB IDE flash
  • VIA Rhine II Fast Ethernet
  • VIA AC’97 audio codec
  • 4x USB 2.0 ports
  • 1x serial, 1x parallel port

These thin terminals and later variants with AMD Sempron and Intel Atom CPUs can often be found dirt cheap on the second hand market (often for less than an RPi) and feature very low power consumption.

As far as software goes, OpenWRT supports x86 platforms and is designed for embedded platforms so it fits nicely with the limited amount of memory and storage on the thin terminal.

Image Download

If you just want to turn your VIA-based HP thin terminal into an AirPlay server with little fuss, just grab the image below.

Download OpenWRT 12.09 ‘Attitude Adjustment’ image with AirPlay support for VIA-based HP thin terminals.

This includes SSH, NTP, ShairPlay, ALSA, Avahi and kernel modules for the VIA Padlock crypto engine, VIA Rhine network and VIA AC’97 codec. Basic IPv6 support is included as ShairPlay seems to get upset without it. There’s also USB keyboard support which is handy if you don’t have a null modem cable lying about.

It’s very lean and I haven’t bothered to include much more than the bare necessities to administer the box and get AirPlay working. There is no web interface, wireless support or even drivers for any other brands of network interface. The root partition is 120MB so the while thing also fits onto the 128MB flash module fitted to the t5520 (Windows CE version of the t5525).

If you want additional functionality I suggest reading below to build an image with other utilities and packages.

OpenWRT Build Process

I’ve listed all the steps I took to build OpenWRT from scratch just in case it’s useful to anybody else.

This is easiest on a Linux distro. I suggest Linux Mint if you are new to that game and the instructions below are aimed at Ubuntu or Mint. If you use OS X then that should be fine; the OpenWRT wiki has a list of dependencies and you can install these easily via MacPorts or similar.

Install Dependencies

First install all the software we will require for obtaining and compiling the OpenWRT source:

sudo apt-get install build-essential gawk gcc-multilib git-core libncurses5-dev libxml-parser-perl subversion zlib1g-dev

Download the OpenWRT Source Tree

I used Attitude Adjustment so that’s what we will download here. Feel free to use a different version.

svn co svn://svn.openwrt.org/openwrt/branches/attitude_adjustment

Once this is complete you should find the entire source tree in the attitude_adjustment directory. The remaining commands assume you are working from this directory.

cd attitude_adjustment

Add ShairPlay Source

Mike Juni has forked an OpenWRT-specific version of ShairPlay. To include it, simply add the following line to feeds.conf.default:

src-git mikejuni git://github.com/mikejuni/OpenWRT-ShairPort;master

Now update the feeds:

scripts/feeds update -a
scripts/feeds install -a

Enable VIA AC’97 Kernel Modules

By default, OpenWRT doesn’t include modules for the VT8200 series AC’97 audio codec used in these old VIA HP thin terminals. That’s okay though as the modules we need are included in the mainline kernel, so it’s simply a matter of creating a suitable package definition to have OpenWRT include this.

Add the following lines to /kernel/modules/sound.mk. Note that the vt82xx modules depend on mpu401-uart.

define KernelPackage/sound-mpu401-uart
  TITLE:=MPU 401 UART support
  DEPENDS:=+kmod-sound-core
  KCONFIG:=CONFIG_SND_MPU401_UART
  FILES:=$(LINUX_DIR)/sound/drivers/mpu401/snd-mpu401-uart.ko
  AUTOLOAD:=$(call AutoLoad,36,snd-mpu401-uart)
  $(call AddDepends/sound)
endef

define KernelPackage/sound-mpu401-uart/description
  Routines for control of MPU-401 in UART mode
endef

$(eval $(call KernelPackage,sound-mpu401-uart))

define KernelPackage/sound-via82xx
  TITLE:=VIA VT82xx series AC97 Controller
  DEPENDS:=+kmod-ac97 +kmod-mpu401-uart
  KCONFIG:=CONFIG_SND_VIA82XX
  FILES:=$(LINUX_DIR)/sound/pci/snd-via82xx.ko
  AUTOLOAD:=$(call AutoLoad,37,snd-via82xx)
  $(call AddDepends/sound)
endef

define KernelPackage/sound-via82xx/description
 Support for the integrated AC97 sound device on motherboards
 with VIA chipsets.
endef

$(eval $(call KernelPackage,sound-via82xx))

Those trying to get sound support on older VIA C3 ITX boards may also find this useful.

Package Configuration

Now it’s time to configure the packages to be built in the OpenWRT image.

make menuconfig

This will start a menu driven interface similar to the one presented when selecting Linux kernel options.

OpenWRT configration menu

Here’s the options I chose; feel free to change these to suit your system. I set all to * to include them in the base image rather than as a package.

  • Target System: x86
  • Subtarget: Generic
  • Target Profile: Generic
  • Target Images
    • Root filesystem partition size: 120MB
  • IPv6
    • <*> 6scripts
  • Kernel Modules
    • Cryptographic API Modules
      • kmod-crypto-aes
      • kmod-crypto-hash
      • kmod-crypto-hw-padlock
    • Filesystems
      • kmod-fs-ext4
    • Libraries
      • kmod-lib-zlib
    • Network Devices
      • kmod-via-rhine
    • Sound Support
      • kmod-sound-mpu401-uart
      • kmod-sound-seq
      • kmod-sound-soc-ac97
      • kmod-sound-via82xx
    • USB Support
      • kmod-usb-hid
      • kmod-usb-storage
      • kmod-usb2
  • Network
    • IP Addresses and Names
      • avahi-autoipd
      • avahi-dnsconfd
    • SSH
      • openssh-client
      • openssh-server
    • Time Synchronization
      • ntpd
  • Libraries
    • Filesystem
      • libext2fs
    • SSL
      • libopenssl
    • alsalib
    • libavahi-client
    • libavahi-dbus-support
  • Utilities
    • alsa-utils
  • Sound
    • shairport

Exit the configuration menu and save changes when prompted.

Now go make some coffee while we compile; depending on the options chosen and the speed of your machine this can take over an hour.

make

If you are an optimisation freak check the make options on the OpenWRT wiki.

One the build process is finished the OpenWRT images can be found in bin/x86.

Installing onto the Thin Terminal

The easiest way is to copy the resulting image to a USB flash drive and boot off that. This process completely overwrites any existing content on the drive!

Plug the USB drive in and then locate the device node it uses:

dmesg | tail

Decompress the OpenWRT image and copy the OpenWRT image to the USB drive shown above (in this case /dev/sdc). It’s easiest to do this in one command and shown below:

gunzip -c bin/x86/openwrt-foo.img.gz | sudo dd of=/dev/sdc bs=1M

Once that’s complete, rescan the partitions with partprobe or unplug and reinsert the drive. If we mount the partitions now we can make some configuration changes before booting so you don’t need a keyboard or display on the thin terminal.

If it doesn’t mount automatically you will need to do this manually (once again replace /dev/sdc with your USB drive device):

sudo mkdir -p /mnt/openwrt/boot /mnt/openwrt/root
sudo mount /dev/sdc1 /mnt/openwrt/boot
sudo mount /dev/sdc2 /mnt/openwrt/root

Configuration

Root Password

You will need to change the root password before you can login via SSH. If you have mounted the USB stick simply do this with chroot (replace the path to the OpenWRT root partition if necessary):

chroot /mnt/openwrt/
passwd root

Enter a new password then type exit to leave the chroot.

Network

Network settings are configured by editing /etc/config/network. Here’s an example:

# Copyright (C) 2006 OpenWrt.org

config interface loopback
    option ifname   lo
    option proto    static
    option ipaddr   127.0.0.1
    option netmask  255.0.0.0

config interface lan
    option ifnam    eth0
    option type     bridge
    option proto    static
    option ipaddr   192.168.1.5
    option netmask  255.255.255.0
    option gateway  192.168.1.1
    option dns      192.168.1.1

AirPlay

You will need to change the name of the mixer control to use for the AirPlay volume control or ShairPlay will crash.

Edit etc/config/airplay and set the volume line to PCM. You can also change the display name that ShairPoint anncounces itself as, for example ‘Workshop’ or ‘Games Room’.

config airplay
        option bname 'Workshop'
        option initbuf 320
        option daemonize true
#       option port 5002
#       option password '123456'
        option pcm 'default'
        option ctl 'default'
        option volume 'PCM'

Boot Config

Because we will be initially booting off a USB stick we will need to change the target device in the GRUB configuration. This is held in the first partition on the USB stick (e.g. /mnt/openwrt/boot).

As an alternative, if you have a keyboard and display attached to your target machine you can press e in the GRUB boot menu and edit the line manually during each boot. This will save you needing to edit it again once it is copied to the internal flash.

Edit boot/grub/menu.1st and change /dev/sda2 to /dev/sdb2 (you can use command below to do this for you).

sed -i -e 's/sda/sdb/' boot/grub/menu.1st

Don’t forget to unmount the USB stick before unplugging it!

Installing on the Target System

Insert the USB stick and boot up the target machine. It should be accessible via SSH after a few moments. If not, plug in a screen and check for any errors.

The last remaining steps are to configure ALSA to actually output sound and then install the image onto the built-in IDE flash.

ALSA Configuration

Run alsamixer to set the output volume. Set Headphones to 100% and press M to unmute it. Do the same for PCM but I recommend setting the level to about 60%.

alsamixer

ALSA will forget this when the system is shut down so we need to make it persistent. First save the configuration as it is now:

alsactl -f /etc/alsa0.state store 0

Then to restore this on boot, save the following into /etc/init.d/alsa-cfg. Setting the order to 98 ensures that ALSA is set up correctly before ShairPlay starts.

#!/bin/sh /etc/rc.common
START=98
start() {                                 
    alsactl -f /etc/alsa0.state restore 0
}

Make it executable:

chmod +x /etc/init.d/alsa-cfg

Then enable it on boot:

/etc/init.d/alsa-cfg enable

Install on Internal Flash

From here simply block copy the USB stick onto the internal IDE flash:

dd if=/dev/sdb of=/dev/sda bs=1M

Once that’s complete, unplug the USB stick and reboot. Hopefully everything works as expected!

Bootnote

It’s getting late and I should really be doing homework so please accept my apologies if I’ve missed something!

I also found that image also works perfectly on the HP t5515 which is very similar but based on an 800MHz Transmeta Crusoe CPU. It draws even less power than the t5525 and appears to run happily off a 12v 1.5A power supply.

References

OpenWRT build documentation http://wiki.openwrt.org/doc/howto/build

ShairPort discussion https://forum.openwrt.org/viewtopic.php?id=29491

Vodafone Huawei K3772 3G modem support for TP-Link routers

TP-Link TL-MR3020 and Vodafone K3772 3G modemI recently purchased a Vodafone 3G USB modem and TP-Link TL-MR3020 3G router. After unboxing the modem I found that it was a Huawei K3772 which is unfortunately not compatible with TP-Link routers. After much searching around I contacted TP-Link support just for a laugh. Much to my surprise they responded within a few hours! Although most of the advice was just general troubleshooting, they provided a link to toolkit for Windows for producing a so-called ‘bin file’ for loading onto the router.

Following their somewhat quirky procedure worked and I now have a bin file that can be used to get these modems working. I submitted this back to the very helpful support crew at TP-Link and they appear to have handed it on to their engineers. Just how long it will take to appear on their website though is anyone’s guess.

If you would like to make your K3772 modem work with a TP-Link router using the simplest means possible, just grab the following file, unpack it and upload it to the router under the 3G modem settings page:

K3772 bin file

In the 3G/4G page (under Network), click Modem Settings:

TP-Link 3G modem settings page

If there are already any files loaded, delete them and restart the router before continuing. Click Add New… to upload the bin file you downloaded above.

TP-Link 3G modem bin file

Once the file is uploaded, restart the router and insert the 3G modem. You can check that it is working in the Status page. The LED on the modem should also illuminate in steady light blue colour.

TP-Link 3G status

If you would like a copy of the USB sniffer tool and the various scripts provided, it is available from TP-Link’s FTP server at ftp://ftp.tp-link.com/Temp/3G/TP-LINK_3G_USBSniffer_Guide_105.zip. Login is tplink and the password is wr641g@. I’m not sure how long that will last so I’ve also placed a mirror on Dropbox.

For those not in the know, the reason for all the mucking about is because many of these modems present themselves as virtual CD-ROM drives when connected so that they can automatically install drivers. Once the driver is installed it sends a command to the modem to switch it into modem mode.

Usb_modeswitch is a popular Linux tool created for this exact purpose, and unpacking the MR3020 firmware image with firmware_mod_kit reveals that it uses this very tool to set up modems that are connected. The bin file above simply seems to be some obfuscated instructions for usb_modeswtich.

Update: I’ve moved the modem file to a new storage provider as Dropbox kept blocking my public links due to excessive traffic.

Another update: If you are a little more technically inclined then I’d recommend installing OpenWRT instead (if your model is supported) and setting up the 3G modem as per the 3G instructions on the OpenWRT wiki. I can confirm that this works well for the K3772. Also, if you are thinking of buying a TL-MR3020 specifically for installing OpenWRT then don’t. Get the D-Link DIR-505 instead – it is based on the same platform (Atheros AR9330) but features double the flash and memory capacity (8MB and 64MB respectively) and the power supply is built-in.

Home telephone wiring in NZ

I had to perform some work on my home phone line including running new cable to a newly installed patch panel. I was suprised to notice that the existing cable had simply been twisted onto the incoming cable pair so this was quickly remedied.

Anyway, the incoming cable contained a single pair comprising of yellow and black wires. There had been several fingers in the pie and the cabling through the house was quite a mess, and figuring out the colour codes took a little bit of work, Here’s a guide for those living in an older NZ home who need to perform work on the phone line:

Cable Type Tip Ring
Incoming line Black Yellow
Existing house wiring Red White
Cat5 cable White(w/blue) Blue
Flexible phone leads Green Red

If you are still unsure, grab an LED and 10k resistor and connect the across the line. If it glows, the cathode is connected to tip and the anode to ring.

Connector Type Tip Ring
BS6312 (BT style) 5 2
RJ11 (6P6C) 4 3
RJ45 (8P8C) 5 4

HTC Wildfire S factory ROM update

A relative recently purchased a shiny new HTC Wildfire S from eking.co.nz. The phone itself is fantastic and I’ve had pretty good service from that site, however this particular Wildfire S turned up installed with a Chinese ROM, version 1.35.707. This version has loads of applications specific to China that cannot be removed and waste considerable space. I’m sure they are great for Chinese users but they are completely useless in New Zealand.

So the first step was to locate a suitable factory ROM image, so I decided on the European one. This took a bit of searching, but some ROMs are available from the following URLs. Note that the Wildfire S codename is Marvel (the original Wildfire codename is Buzz).

http://www.shipped-roms.com/index.php?category=android&model=Marvel
http://forum.xda-developers.com/showthread.php?t=1058089

I went for the Shipped ROMs version which is an official RUU installer from HTC. This contains a European ROM image. I’ve also made a copy available for download below:

HTC_Marvel_Europe_10.35.3029H_7.46.35.08_release_185724_signed.zip (use for SD and GoldCard updates)
10.35.3029H_7.46.35.08_release_185724_signed.exe
HTC Windows Drivers (install these when using Windows-based update)

ROM zip image MD5 sum:
0eb793740771a9d96ae4f5085c431516

Here’s the steps I tried to replace the factory firmware on the Wildfire S. While all three showed promise I ended up having to use the GoldCard method as the phone CID did not match that of the European firmware image, despite the fact that the images are signed by HTC.

If you purchased your phone from Eking and want to perform this I’d recommend going straight to attempt 3.

Note that I take no responsibility for any damage to your phone resulting from any of the instructions on this site!

The end result of the update was a fully functional phone with with an additional 60-70MB of internal storage freed up.

Attempt 1 – Official HTC Update Application (RUU)

My first attempt was to simply run the .exe file. Unfortunately this requires Windows but normally the process is fairly straightforward:

  1. Download and install HTC Sync.
  2. Ensure HTC Sync is not running or uninstall it and leave the HTC drivers installed.
  3. Power on the phone and enable USB debugging mode.
  4. Run the RUU application and follow the steps.

It did show that the update I was applying was actually a downgrade (v1.35.707 to v1.33.401) but it allowed me to proceed anyway. Unfortunately this failed after copying the image to the phone as it said the phone was the wrong type. After disconnecting the phone I powered it on and it still worked as before. Phew!

Attempt 2 – Flashing from the Boot Loader

The Wildfire S is like most HTC phones in that it can be forced to enter the boot loader by holding Volume Down when powering it on. From here it will look for a file named PG76IMG.zip on the root of SD card and attempt to load it if it exists, otherwise it will present the usual Android bootloader menu (text console with a picture of a three Androids at the bottom of the screen).

The neat thing about the Wildfire S supplied here is that it came already supplied in S-OFF mode.

The required zip file can be extracted from the HTC RUU application above. Don’t worry if the Windows built-in zip decompression doesn’t know how to open it; this does not mean that it is corrupt.

  1. Run the RUU application but leave it open at the welcome screen.
  2. Search your temp directory for a file called rom.zip and copy it to the SD card of the phone.
  3. Rename rom.zip to PG76IMG.zip.
  4. Close the RUU application.
  5. Unmount/safely eject the SD card (important!).
  6. Disable fast boot on the phone if it is enabled.
  7. Power off the phone and disconnect it from the USB port.
  8. Hold down the Volume Down button and power on the phone.
  9. When the bootloader screen appears it should locate the the ROM image file.
  10. Begin the update process when prompted.
  11. The phone will go through various update cycles and reboot itself several times.
  12. When the normal HTC splash screen appears it may take some time for the phone to finish booting; give it at least 10 minutes.

If you receive a message saying the update failed because of an incorrect CID (like I did) you will need to perform the GoldCard method below.

Attempt 3 – GoldCard Method

This method overrides the CID check on the phone.

  1. Format an SD card as FAT32 (if not already done).
  2. Boot the phone up with the SD card in it.
  3. Install GoldCard Helper from the Android Market and run it.
  4. Copy the CID.
  5. Visit http://psas.revskills.de/?q=goldcard and enter your email and CID. If you do this on the phone you can paste the CID from the GoldCard Helper app.
  6. When the email arrives it will have an attachment called goldcard.img. Save this somewhere on your computer.
  7. Connect the phone to the computer and mount the SD card on the computer.
  8. Copy the GoldCard image to the SD card.
    • If you use Linux, simply use dd to copy the GoldCard image onto the SD card. You will need to be root or use sudo for this. Also make sure you use the path to the SD card itself, not a partition on it. Mine was /dev/sdb but yours may differ.
      dd if=goldcard.img of=/dev/sdb bs=384
    • If you are stuck with Windows, download and install HxD Hex Editor.
    • Run HxD as Administrator (right-click and select Run as Administrator).
    • Go to Extras > Open Disk.
    • Select the physical SD device, not the logical one. Uncheck the read-only option and open it.
    • Open the goldcard.img file you received in another tab.
    • Select the entire contents of the GoldCard image and save it over the start of the SD device, address range 00000000 to 00000170. Save the changes and close HxD.
  9. Copy the ROM image to the root of the SD card and ensure it is named PG76IMG.zip.
  10. Unmount the SD card and disconnect the phone.
  11. Ensure the phone has fastboot disabled (change in Settings).
  12. Power off the phone.
  13. Hold down the Volume Down button and power it on.
  14. When the boot loader screen appears, release Volume Down.
  15. Confirm the update process when prompted.
  16. The phone will go through various update cycles and reboot itself several times.
  17. When the normal HTC splash screen appears it may take some time for the phone to finish booting; give it at least 10 minutes.
  18. Once the phone is back to normal operation, zero out the first 512 bytes of the SD card to destroy the GoldCard information.
    • In Linux simply run (as root), assuming your SD card is /dev/sdb:
      dd if=/dev/zero of=/dev/sdb bs=384 count=1
    • In Windows just use HxD to overwrite 00000000 through 00000170 with zeros and save.
  19. Enjoy!
  20. Make a donation to RevSkills for their awesome GoldCard generator.

Eee 900 Bluetooth mod

I finally decided it was time to kit out my ancient but trusty Asus Eee 900 with internal Bluetooth. I had an old micro Bluetooth adapter lying about which came apart very easily to reveal an exceptionally small PCB.

I stuck this to the motherboard of the Eee using a small piece of double sided foam tape, just behind the external VGA connector, and then wired it to the unused USB lines connected to the WiFi adapter PCIe slot.

Power was derived from the same 5v rail that supplies the external USB ports. I simply scraped some solder mask off the tracks and soldered the wires directly.

Fortunately once I put it all together again it worked!

Another reason I try to avoid Windows

I decided to replace the hard disk in my Dell Optiplex 755 with a larger model (500GB). Migrating Ubuntu was trivial; Windows Vista was not. Clonezilla did its best but I ended up running into all sorts of Windows BCD corruption issues. Most of the various repair tools on the Windows DVD failed with incredibly non-descriptive error messages.

After giving up I quickly discovered that Windows would refuse to even install, citing Windows is unable to find a system volume that meets its criteria for installing. This was even after zeroising the first few GB of the disk with dd. WTF Microsoft?

Even a BIOS update didn’t do the trick. (hint to Dell, et al.: DOS is dead. Please give us a modern, OS independent BIOS update mechanism)

It turns out that two things seemed to have an affect on this: my HP Photosmart printer with its built-in card reader and floppy drive setting in the BIOS. Disconnecting the printer and setting the floppy drive mode from USB (default if there is no internal floppy drive) to None cured the problem.

By this stage I had wasted enough hours so didn’t even bother trying to see if the clone would work this time around.

Of course all along Ubuntu behaves just perfectly.