Proxmox Storage

Storage definitions live in /etc/pve/storage.cfg and are managed with pvesm. Default install: local (directory /var/lib/vz, ISOs, templates, backups) and local-lvm (LVM-thin, VM and container disks). ZFS installs get local-zfs instead.

Documentation: Storage · pvesm(1)

Verification: pvesm add dir, add nfs, add pbs, scan nfs, set, remove and the default storage layout were checked against the docs. The zfspool, lvmthin, cifs and zfs/lvs examples, the ARC limit and the Ceph notes were not verified against the docs. The LVM, BTRFS, iSCSI, CephFS, RBD, ZFS over iSCSI and ESXi sections were read in the storage chapter (PVE 9.2 docs). Gaps are marked (unverified). No command was executed.

See also: Proxmox VE Overview and Install · Proxmox Backup and Restore · Proxmox qm and pct · Proxmox Disk Image Formats · NFS · TrueNAS Community Edition (ZFS and NFS server side)

pvesm #

Content Types #

The Content drop-down in the Add Storage dialog maps to the content option (comma-separated).

Not every storage type offers every content type. The drop-down only lists what the selected type supports, for example:

none is for storage you only use as a base for another, such as an iSCSI LUN under LVM. Change the list later with pvesm set <STORAGE> --content images,rootdir. Content types for the other storage types are in the storage chapter (not re-checked here).

Add Storage #

For a directory on a separate disk: partition, mkfs.ext4, mount via /etc/fstab (use the UUID), then add as dir. Add is_mountpoint 1 on the storage to stop Proxmox writing into the empty mount point if the disk is missing.

Which Type #

Snapshot support depends on disk format: raw disks on plain directory storage cannot snapshot, qcow2 can. See Proxmox Disk Image Formats for raw, qcow2 and VMDK.

LVM (Thick) #

Block storage on a volume group, with each VM disk a full-size logical volume. Content types are images and rootdir.

The docs mark LVM as not shared by itself. To share it, put it on an LUN every node can see (iSCSI), but no full walkthrough was found (unverified).

Snapshots as Volume Chain #

The Snapshots as Volume-Chain checkbox (storage option snapshot-as-volume-chain) gives thick LVM storage real VM snapshots. Each snapshot is a thick logical volume holding only the differences from the volume behind it, and taking a snapshot starts a new volume backed by the last. It uses qcow2 only to layer the chain, and the PVE storage layer does the snapshot work, not qcow2’s internal snapshots.

The docs say this avoids the I/O penalty and the dangerous out-of-space behaviour of native LVM snapshots, and that it works on block storage such as iSCSI and FC SANs, with cluster-wide locking when the storage is shared.

Caveats:

• Each snapshot uses its full size unless the backing storage is thin provisioned and passes through discard.
• LVM still has no linked clones.
• The roadmap says migrating VMs that have snapshots on local volume-chain storage is blocked.
• The docs do not say how existing raw disks, backup, clone or move-disk behave, or how much overhead qcow2 adds. The idea that only new disks become qcow2 is inferred (unverified).
• The Preallocation setting shown in the same dialog is documented (off, metadata, falloc, full) only for raw and qcow2 images on file-based storage, so how it applies here is (unverified).

BTRFS #

File-based storage on a BTRFS filesystem. The docs mark it as a technology preview and show only a storage.cfg example, not a pvesm add command.

btrfs: <STORAGE>
        path /mnt/<BTRFS_MOUNT>
        content rootdir,images
        is_mountpoint 1

Snapshots are supported, and storage is not shared. Treat it as unsuitable for anything you cannot rebuild.

iSCSI #

Raw LUNs from an iSCSI target. The LUNs are not managed per guest, so the usual pattern is content none on the iSCSI storage with LVM on top. Content types are images and none. Table 10 of the docs says no snapshots, though the summary table says “volume chains”, so the per-backend table is trusted here.

iscsiadm is only mentioned in the docs, without commands, so low-level session commands are not covered here (unverified).

CephFS and RBD #

Both talk to a Ceph cluster, either one built with pveceph (Ceph (overview)) or an external one.

The keyring file is copied to /etc/pve/priv/ceph/<STORAGE>.keyring for RBD, and the CephFS secret lives at /etc/pve/priv/ceph/<STORAGE>.secret. Set krbd 1 to use the kernel RBD client instead of librbd. CephFS snapshots exist, but the docs do not guarantee they are stable.

ZFS over iSCSI #

Proxmox creates a zvol per disk on a remote ZFS server and attaches it over iSCSI. It is the zfs storage type, and not to be confused with zfspool for local ZFS. The remote host needs an iSCSI target provider and SSH access from every PVE node. Content is images. Snapshots work, clones do not.

Set up passwordless SSH from each node first:

mkdir /etc/pve/priv/zfs
ssh-keygen -f /etc/pve/priv/zfs/192.0.2.1_id_rsa
ssh-copy-id -i /etc/pve/priv/zfs/192.0.2.1_id_rsa.pub root@192.0.2.1

The key file name starts with the portal address. The docs give storage.cfg examples for each provider but no pvesm add zfs command (unverified). blocksize and sparse are also documented for the local zfspool type.

ESXi Import #

The esxi type connects Proxmox to an ESXi host or vCenter so you can import its VMs. It is for migration, not for running guests. The import wizard is GUI-only, tech preview, added during the 8.2 cycle, and was tested with ESXi 6.5 to 8.0.

The docs have no pvesm add esxi example, so add it from Datacenter > Storage > Add > ESXi (unverified from the CLI). Whether qm importdisk applies is not addressed there. Migration context is in Hypervisor Comparison.

LVM-thin #

A full thin pool freezes guests. Monitor Data% and do not over-commit heavily.

ZFS #

Limit ARC (default can use up to about half of RAM) in /etc/modprobe.d/zfs.conf:

options zfs zfs_arc_max=8589934592

Then run update-initramfs -u -k all and reboot. More ZFS commands in TrueNAS Community Edition.

Ceph (overview) #

Hyper-converged shared storage built into PVE: each node runs MON, MGR and OSD daemons, VMs use RBD pools. Needs at least 3 nodes, a dedicated fast network (10 GbE or more is the usual advice) and one OSD per disk.

Not verified against the docs. Read the Ceph chapter before building a cluster.

Lightbits SDS (NVMe/TCP) #

Lightbits (LightOS) is software-defined block storage served over NVMe/TCP. It is not built into PVE: the community-supported pve-lightbits plugin adds a lightbits storage type, and each VM disk becomes one Lightbits volume (vm-<VMID>-<VMGENID>-disk-<N>). It is shared storage, so live migration moves no disk data.

Plugin: LightBitsLabs/pve-lightbits · Lightbits docs

Requirements (per the plugin README): PVE 9.x, LightOS 3.x, Perl 5.36 or newer, kernel 5.0 or newer with nvme_tcp, and the nvme-cli and discovery-client packages. Network ports to the Lightbits cluster: 443 (REST API), 4420 (NVMe/TCP I/O) and 8009 (discovery). Only NVMe/TCP is supported.

Install on every node: clone the repository and run its installer (scripts/install.sh). It installs the module, pulls in nvme-cli and discovery-client if missing and restarts the PVE daemons. Re-run it to upgrade. The uninstaller refuses to run while lightbits storage entries exist, so remove them first with pvesm remove.

Add the storage from the CLI with pvesm add lightbits <STORAGE> plus the options above. The GUI “Add Storage” dialog does not list this type. Change settings later with pvesm set <STORAGE> <OPTION> <VALUE>.

Behaviour to know about:

• Snapshots and native rollback work but are crash-consistent. A snapshot cannot be attached, and backup or clone from a snapshot is not supported. Deleting a disk deletes its snapshots.
• Only full clones work, not linked clones. Resize is grow-only.
• Live migration (qm migrate <VMID> <NODE> --online) and HA restart on another node work.
• The plugin writes /etc/discovery-client/discovery.d/lightbits-<STORAGE>.conf and symlinks volumes under /dev/lightbits/<STORAGE>/. It only manages volumes it labelled itself.

Not verified. Facts come from a read of the plugin README and Lightbits docs, not from running it. The installer path was taken from a search snippet, and the literal pvesm add example was not captured, so check the README for the exact syntax. The Lightbits product page mentions PVE 8.x, the README says 9.x. Check the README for current versions.

Check Free Space and Clean Up #