ProxMox Best Practice Guide <WIP>

Summary

This Best Practice Guide will instruct users on how to do the initial configuration for the ProxMox Virtual Environment (PVE) to ensure a successful deployment with the StorONE Storage System

Introduction

These instructions and procedures were created while working with StorONE 3.8.0.56 and PVE 8.3

PVE System Configurations

Block Volume Considerations

Unlike some other hypervisors ProxMox is allowing the use of 4k Sector Volumes with block storage.

iSCSI Initiator ID

When you create the LUN on the StorONE you will need to provide the hosts iSCSI initiator name. This can be found here:

cat /etc/iscsi/initiatorname.iscsi 
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1993-08.org.debian:01:6411345199be

iSCSI Configuration

It is recommended that you make modifications to the iSCSI configuration in order to have the best experience when connecting to the StorONE Storage Appliance. Using SSH or the Console in the ProxMox UI access the system CLI and edit the file located at /etc/iscsi/iscsid.conf and ensure the following options are set accordingly:

node.startup = automatic
node.session.timeo.replacement_timeout = 10
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 10
node.session.nr_sessions = 4
node.session.queue_depth = 64
node.session.cmds_max = 256

Once modified the service should be restarted:

service iscsid restart

Multipathing IO - MPIO

ProxMox does not have Multipathing installed by default. It must be added by logging into the ProxMox CLI and running the following commands:

apt-get update
aot-get install multipath-tools

create /etc/multipath.conf

defaults {
    find_multipaths yes
    user_friendly_names yes
    path_selector "round-robin 0"
    path_grouping_policy multibus
    no_path_retry 30
    max_sectors_kb      1024
    queue_without_daemon no
    max_fds max
    flush_on_last_del yes
    log_checker_err once
}
devices {
  device {
    vendor "STORONE*"
    product "S1*"
    detect_prio yes
    prio "alua"
    path_selector "queue-length 0"
    path_grouping_policy group_by_prio
    failback immediate
    path_checker tur
  }
}
blacklist {
  wwid ".*"
}
blacklist_exceptions {
    wwid                  "36882e5a*"
}

Adding iSCSI Storage from the StorONE

In the Add iSCSI dialog:

- ID - a recognizable name for the inventory
- Portal - the discovery ip of the iSCSI target from the StorONE software
- Target - this should auto-populate after supplying the Portal IP
  - this can also be discovered in the StorONE CLI or GUI
    - CLI - nodes list --gateay
    - GUI - Inventory -> Nodes + Gateway Checkbox -> Submit
- Nodes: Optional but will default to ALL within the datacenter
- Enable: should be checked by default
- Use LUNs Directly: Checked by default and is comparable to using a LUN(s) like an RDM (RAW Device Mapping) in other hypervisors

The above process adds the discovery for the StorONE iSCSI target. The name used here will be displayed for ALL LUNs discovered. The use or applications of individual LUNs will be applied in other contexts.

The LUN should appear in the Storage list:

From the ProxMox CLI/Console verify that all the expected paths are seen:

root@pve:/etc/iscsi# multipath -ll
mpatha (36882e5a0000000010000000000000507) dm-5 STORONE,S1
size=233G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='queue-length 0' prio=50 status=active
  |- 19:0:0:1 sdh 8:112 active ready running
  |- 18:0:0:1 sdg 8:96  active ready running
  |- 17:0:0:1 sdf 8:80  active ready running
  `- 20:0:0:1 sdi 8:128 active ready running

If you add a LUN while the system is up and already configured you can rescan with the following command: iscsiadm -m node -I default -T <target iqn> -p <target ip> -R (you may have to run it twice to discovery the LUN)

At this stage we can use the LUN(s) as-is as an RDM. Using the volume this way would provide access to a single VM. To use the Volume/LUN for multiple VMs proceed to applying an LVM

StorONE HA (dual node configuration)

In the case of using a dual node StorONE configuration you will have to add the discovery information from the CLI/Shell. This will require that you have the second node's IP address available. The command will look like this:

iscsiadm -m discovery -t sendtargets -p <target IP>

In the following example in the UI we've already connected a LUN to target 10.10.50.130. Now we want to add the second node's IP of 10.10.50.131:

root@pve:~# iscsiadm -m discovery -t sendtargets -p 10.10.50.131
10.10.50.131:3260,1 iqn.2012-01.com.storone:5882e5a-0002-0004

root@pve:~# iscsiadm -m node -l
Logging in to [iface: default, target: iqn.2012-01.com.storone:5882e5a-0002-0004, portal: 10.10.50.131,3260] (multiple)
Logging in to [iface: default, target: iqn.2012-01.com.storone:5882e5a-0002-0004, portal: 10.10.50.131,3260] (multiple)
Logging in to [iface: default, target: iqn.2012-01.com.storone:5882e5a-0002-0004, portal: 10.10.50.131,3260] (multiple)
Logging in to [iface: default, target: iqn.2012-01.com.storone:5882e5a-0002-0004, portal: 10.10.50.131,3260] (multiple)
iscsiadm: default: 4 sessions requested, but 4 already present.
Login to [iface: default, target: iqn.2012-01.com.storone:5882e5a-0002-0004, portal: 10.10.50.131,3260] successful.
Login to [iface: default, target: iqn.2012-01.com.storone:5882e5a-0002-0004, portal: 10.10.50.131,3260] successful.
Login to [iface: default, target: iqn.2012-01.com.storone:5882e5a-0002-0004, portal: 10.10.50.131,3260] successful.
Login to [iface: default, target: iqn.2012-01.com.storone:5882e5a-0002-0004, portal: 10.10.50.131,3260] successful.
iscsiadm: Could not log into all portals

root@pve:~# multipath -ll
mpatha (36882e5a0000000020000000000000d30) dm-5 STORONE,S1
size=233G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='queue-length 0' prio=50 status=active
| |- 17:0:0:1 sdg 8:96  active ready running
| |- 18:0:0:1 sdh 8:112 active ready running
| |- 19:0:0:1 sdi 8:128 active ready running
| `- 20:0:0:1 sdj 8:144 active ready running
`-+- policy='queue-length 0' prio=10 status=enabled
  |- 14:0:0:1 sdd 8:48  active ready running
  |- 13:0:0:1 sdc 8:32  active ready running
  |- 16:0:0:1 sdf 8:80  active ready running
  `- 15:0:0:1 sde 8:64  active ready running

Adding LUNs to a running system

If you add a LUN while the system is up and already configured there are two ways to get the system to see the new device(s):

Reboot the PVE Server

You can reboot the server and the system should find the new LUN and all it's PATHs if you've followed the configuration advice in this document

Rescan the iSCSI targets from the command line

rescan with the following command: iscsiadm -m node -I default -T <target iqn> -p <target ip> -R (you may have to run it twice to discover the LUN)

If you are doing this connected to an HA StorONE system you will need to run the command against both target IPs/IQNs

LVM applied to discovered LUN:

In the ProxMox UI proceed to Datacenter -> Storage -> Add -> LVM

In the Add LVM dialog:

- - ID - an identifiable name for the inventory
  - Base Storage - Select the StorONE Storage Device created in the previously step
  - Base Volume - Select the LUN ID you want to apply the LVM
  - Volume Group - a required field that can be named anything
  - Content - By default Disk Image, Container. In most cases this default should be sufficient
  - Nodes - Set to ALL by default and can be directed to an individual server
  - Enabled - checked by default
  - Shared - Unchecked by default but can be selected if the LVM is being deployed on a shared LUN.
  - Wipe Removed Volumes - Unchecked by default. Also referred to as Safe Remove. This ensures that in the event of the removal of a volume the data will be "zeroed" out

The new LVM should appear in the Storage Inventory:

Applying the devices to VMs

SCSI Discard

A note about SCSI Discard (aka TRIM). The StorONE devices will supposed the use of SCSI discard to reclaim unused space on a volume. The ProxMox default is to NOT have discard enabled when you create a VM. It is the recommendation of StorONE that you enabled this feature for both RAW and disk image storage devices used in VMs.

When creating a Virtual Machine when you get to the Disks tab ensure that the Discard checkbox is marked as shown below:

Both the RAW device and the LVM will now display in the Storage option when creating a VM:

In the VM creation if you chose to use an RDM you will select the StorONE iSCSI device and then select the LUN to use

Do NOT use the same LUN for both RDM activities and LVM activities.

Adding NFS Storage From StorONE

In the Add NFS dialog:

- ID - a name to allow you to locate the StorONE NFS device in the Storage Inventory
- Server - The IP of the floating address of the StorONE NAS Server
- Export - Should auto-populate the names of shares available at the Server IP you supplied
  - Note: Only choose the name of the share that was created. Do NOT use the path that looks like /volume/Vol-XXX/exports
- Content - Defaults to Disk Image. You can add other options as fits the needs of this storage device
- Nodes - defaults to ALL. Can be specified for a specific server in the datacenter
- Enable - checked by default

The new NFS storage device appears in the Storage Inventory:

Congratulations. You should now have full access to the StorONE devices to work with ProxMox