misc

Graphical workstation kvm lab

First install ubuntu deskop with zfs-on-root* (Optional)

Get the desktop iso

curl https://releases.ubuntu.com/22.04/ubuntu-22.04.1-desktop-amd64.iso -o ubuntu-22.04.1-desktop-amd64.iso

flash USB using balena or dd (dd example below)
This could destroy your system if you flash to the wrong drive! Identify your flash drive first. The hash at the start of the command here is for safety purposes if your are copy/pasting. You must fist edit the command and remove the hash in order to run it!
```
# sudo dd if=ubuntu-22.04.1-desktop-amd64.iso of=/dev/sdx bs=1024k status=progress
```

Install Packages and setup shell

Install packages using `apt`

sudo apt install -y qemu-kvm virt-manager libvirt-daemon-system virtinst libvirt-clients \
  bridge-utils zsys cockpit cockpit-machines net-tools openssh-server \
  zsh software-properties-common curl git libnss3-tools jq xsel xrdp

NOTE: Alternatively, you may first want to simply install openssh-server in the graphical environment, then do the rest over ssh, so that you can copy/paste.

(Optionally) Allow passwordless sudo for your user

echo "${USER} ALL=(ALL) NOPASSWD: ALL" |sudo tee -a /etc/sudoers

(Optionally) Setup `zsh` with oh-my-zsh and syntax highlighting and autocompletion

sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

git clone https://github.com/zsh-users/zsh-autosuggestions.git $ZSH_CUSTOM/plugins/zsh-autosuggestions \
  && git clone https://github.com/larkery/zsh-histdb $HOME/.oh-my-zsh/custom/plugins/zsh-histdb \
  && git clone https://github.com/zsh-users/zsh-syntax-highlighting.git $ZSH_CUSTOM/plugins/zsh-syntax-highlighting \
  && sed -i '/plugins=(git)/c\plugins=(git zsh-autosuggestions zsh-syntax-highlighting)' ~/.zshrc \
  && source ~/.zshrc

(Optionally) Disable wayland

Edit /etc/gdm3/custom.conf
```
sudo nano /etc/gdm3/custom.conf
```
uncomment WaylandEnabled=false

Setup grub

Edit `/etc/default/grub`

sudo nano /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"

Update grub
```
sudo update-grub
```
Add required Modules

Edit `etc/modules`

sudo nano /etc/modules

Add the following modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Enable SATA Controller to be passed through to TrueNAS VM

This requires a discrete Sata controller, and a seperate drive/controller for the host. The board being used in this example has a sata controller and a seperate nvme drive. The ubuntu host is installed on the nvme drive, on which the default libvirt pool will also be used to store the image for the TrueNAS VM’s boot drive.
First locate your SATA Controller
lspci | grep SATA
Example output:
07:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
Add your the SATA Controller to /etc/modprobe.d/vfio.conf
echo "options vfio-pci ids=07:00.2" | sudo tee -a /etc/modprobe.d/vfio.conf
NOTE:

These steps will also work for an LSI HBA installed in a pcie slot

Reboot System and verify changes

Reboot
```
sudo reboot now
```
Check for iommu
```
sudo dmesg | grep -e DMAR -e IOMMU
```

Configure the system to never poweroff or autosuspend

If you are running your host headless you can access the graphical environment using reminna over xrdp if you install xrd and disabled wayland

Login to the desktop environment

Navigate to Settings>Power

Set Screen Blank to Never

Set Automatic Suspend to Off

Create a bridge for VM Networking

If you want to have VMs on the regular LAN and want to allow Host/VM communication you will need to create a bridge interface for your VMs. One of the simplets way to do this is using by using cockpit

Open cockpit in web browser (https://${server_ip}:9090)

First enable administrative access (I advise logging out and back in after enabling it for your user)

Navigate to Networking>[Add Bridge]
Bridge Settings

Name br0

Ports

eno1 <–use whichever port you want to use to connect VM’s to the internet

Repeat for as many seperate ports as you want to give to your VM(s)

In the case of this example, the host will have a single TrueNAS-SCALE VM which will host all other VMs using nested virtualization. In this case it is a good idea to give the VM at least two network ports – one for management and another for everything else

Navigate to Networking>[Add Bridge]
Bridge Settings

Name br1

Ports

eno2 <–use whichever port you want to use to connect VM’s to the internet

Create a virtual network with the bridge

The instructions below are to replace the default virtual network with a bridged network. If you wish to create a seperate bridged network and leave the default network as is skip steps 3-4 below, and use another name for the network in place of default.

create an xml file
```
  nano bridged-network.xml
```
Paste in the following:

```xml

default

3. #### Stop the default network
  ```bash
  virsh net-destroy default

Undefine the default network
```
  virsh net-undefine default
```

Re-define the network

  sudo virsh net-define bridged-network.xml

Start the network
```
  sudo virsh net-start default
```
set the network to autostart
```
  sudo virsh net-autostart default
```
NOTE:

This can also be done in virt manager gui by going to Edit>Preferences and ticking “Enable XML Editing”.

xml-editing net-xml

Deploy TrueNAS-SCALE VM

Download the iso

curl https://download.truenas.com/TrueNAS-SCALE-Angelfish/22.02.4/TrueNAS-SCALE-22.02.4.iso -o TrueNAS-SCALE-22.02.4.iso

Add the iso to the default storage pool

sudo cp TrueNAS-SCALE-22.02.4.iso /var/lib/libvirt/images

Use latest debian flavor
20G disk is more than enough
Give it at least one bridged network device
Set cpu model to host-passthrough
At least 4 vcpus
At least 8G RAM
Attach a SATA Controller or LSI HBA
Complete install via graphical spice or vnc

create bridge in truenas scale (Use vnc or spice console instead of ssh!*)

 cli -c "network interface create name=br0 type=BRIDGE bridge_members=eno1"

commit changes
```
 cli -c "network interface commit"
```
persist changes
```
cli -c "network interface checkin"
```
reboot the TrueNAS-SCALE VM!

**due to the horrendous state of networking in scale, these changes must be done over direct console via spice or vnc and all network connectivity (to truenas scale) will be lost until reboot!* https://ixsystems.atlassian.net/browse/NAS-118915

This site is open source. Improve this page.

misc

Graphical workstation kvm lab

First install ubuntu deskop with zfs-on-root* (Optional)

Get the desktop iso

flash USB using balena or dd (dd example below)

Install Packages and setup shell

Install packages using apt

(Optionally) Allow passwordless sudo for your user

(Optionally) Setup zsh with oh-my-zsh and syntax highlighting and autocompletion

(Optionally) Disable wayland

Edit /etc/gdm3/custom.conf

Setup grub

Edit /etc/default/grub

Update grub

Add required Modules

Edit etc/modules

Enable SATA Controller to be passed through to TrueNAS VM

First locate your SATA Controller

Add your the SATA Controller to /etc/modprobe.d/vfio.conf

NOTE:

Reboot System and verify changes

Reboot

Check for iommu

Configure the system to never poweroff or autosuspend

Login to the desktop environment

Create a bridge for VM Networking

Open cockpit in web browser (https://${server_ip}:9090)

Bridge Settings

Name br0

Ports

Repeat for as many seperate ports as you want to give to your VM(s)

Bridge Settings

Name br1

Ports

Create a virtual network with the bridge

create an xml file

Paste in the following:

Undefine the default network

Re-define the network

Start the network

set the network to autostart

NOTE:

Deploy TrueNAS-SCALE VM

Download the iso

flash USB using balena or `dd` (`dd` example below)

Install packages using `apt`

(Optionally) Setup `zsh` with oh-my-zsh and syntax highlighting and autocompletion

Edit `/etc/gdm3/custom.conf`

Edit `/etc/default/grub`

Edit `etc/modules`

Add your the SATA Controller to `/etc/modprobe.d/vfio.conf`

Name `br0`

Name `br1`