Linux Storage

Disk management

Layers

1. GUI apps to manage multiple layers: GNOME Disks, GParted, VeraCrypt, Stratis (XFS-only), etc.
   
   
2. Linux standard organization of files and directories in a system: /, /bin, /etc, /etc/passwd, /usr, and so on.
   Chris Hoffman's "The Linux Directory Structure, Explained"
   
   
   
3. Linux OS API to a filesystem: inodes which represent files (some of which are directories), operations such as read/write/createfile/deletefile/makedir/rmdir.
   Wikipedia's "File system"
   
   Format on disk (of ext* filesystem):
   - Boot block ?
   - Superblock (maybe run "sudo dumpe2fs -h /dev/sda1 | less" to see info).
   - Inode (index node) table. (Each inode points to data blocks)
   - Dentries (translate between names and inode numbers, and maintain relationships between directories and files).
   - Data storage blocks (actual contents of files and directories).
   Diagram from Simone Demblon and Sebastian Spitzner's "Linux Internals - Filesystems"
   M. Tim Jones' "Anatomy of the Linux file system"
   SoByte's "Linux File System"
   Kernel.org's "ext4 Data Structures and Algorithms - High Level Design - Special inodes"
   
   
   
4. Plaintext or individually encrypted files: e.g. normal files and directories; app-encrypted files such as password manager databases or encrypted SQL databases.
   
   
5. Filesystem mounted instances: mount-point name, and type, and device name.
   e.g. "/" is the mount location of an ext4 filesystem stored on /dev/sda5.
   Run "df -khT" to see the mounted filesystems.
   
   

---

---

[I'm told layers 6 through 10 really can be mixed in any order, you can stack any block device on top of any other. Maybe consider following to be a typical order.]

6. Upper (stacked) filesystem formats: eCryptfs; EncFS; gocryptfs; Windows' Encrypting File System (EFS); AVFS.
   
   Wikipedia's "ECryptfs"
   SK's "How To Encrypt Directories With eCryptfs In Linux"
   I'm told the eCryptfs code is considered old and unmaintained, so Ubuntu has dropped that option.
   Wikipedia's "Encrypting File System" (EFS)
   
   
   
7. Base filesystem formats: format of data stored inside a partition. E.g. ext4, fat32, NTFS, Btrfs, ZFS.
   Jim Salter's "Understanding Linux filesystems: ext4 and beyond"
   ArchWiki's "File systems"
   
   ext4 can have a file/directory encryption module added to it: fscrypt (article1, article2).
   
   Scan filesystem and map bad blocks to "don't use" inode (ext* filesystem only): "e2fsck -c".
   
   Xiao Guoan's "How to Fix 'can't read superblock' Error on Linux (ext4 & Btrfs)"
   
   
   
8. Manager: e.g. Linux's LVM (Logical Volume Manager), or software RAID, or ZFS, or Btrfs.
   
   Jesse Smith's "Combining the storage space of multiple disks"
   Karthick's "Linux Logical Volume Manager (LVM) Guide"
   David Both's "A Linux user's guide to Logical Volume Management"
   
   "Device mapper" is a framework that things such as LVM and dm-verity and dm-crypt talk to.
   "sudo dmsetup info"
   Wikipedia's "Device mapper"
   
   LVM is oriented toward providing flexibility, while RAID is oriented toward providing reliability.
   
   I think LVM can be used in two opposite ways:
   
   

   • [On a large system:] Present a single "virtual partition" to the layer above, but the data is stored across multiple physical partitions and devices.
     
     
   • [On a small/old single-disk MBR system limited to 4 physical partitions:] Present multiple "virtual partitions" to the layer above, which can use them for swap, /, and /home, but the data is stored in a single physical "extended partition" on disk.
     
     
   
   Ubuntu Wiki's "LVM"
   Wikipedia's "Logical volume management"
   Wikipedia's "Logical Volume Manager (Linux)"
   Sarath Pillai's "The Advanced Guide to LVM"
   Kenneth Aaron's "Install Linux with LVM"
   terminalblues' "LVM Lab Setup With VirtualBox"
   With LVM, see partition type "lvm" in lsblk.
   LVM concepts, from lowest: PV (Physical Volume), then VG (Volume Group), then LV (Logical Volume).
   Corresponding LVM commands: "sudo pvs --all", "sudo vgs --all", "sudo lvs --all".
   "sudo lvm fullreport"
   Example corresponding LVM names: /dev/sda6, vgubuntu-mate, root.
   
   Heard on a podcast: LVM can do snapshots, but they kill performance.
   
   LVM can use a small fast drive as a cache for a large slow drive; see "man lvmcache".
   
   "Linux does not allow manipulation of the root file system if it's in use (even with LVM). To resize an LVM Root partition, you must boot into a live disk." (May refer to partition boundaries, and certain filesystem types. For example you might be able to shrink a Btrfs root filesystem while it's in use.)
   
   Software RAID: "mdadm" command. LVM also has RAID capabilities.
   Marius Ducea's "Mdadm Cheat Sheet"
   Thomas-Krenn's "Mdadm recovery and resync"
   Andy's "Debian-installer, mdadm configuration and the Bad Blocks Controversy"
   Chris Siebenmann's "Getting the names of your Linux software RAID devices to stick"
   "sudo mdadm --detail-platform"
   "sudo ls /proc/mdstat"
   Warning: if a drive drops out of the array with errors, and you add it back in and it immediately says "OK", do not trust this. Force a full scrub/read on the RAID.
   
   
   
9. Device-mapper and full-volume/block-level encryption: e.g. dm-crypt (a LUKS-compliant implementation); VeraCrypt "full-disk" (really, full-partition) encryption; BitLocker, Loop-AES.
   
   Wikipedia's "dm-crypt"
   Wikipedia's "Linux Unified Key Setup" (LUKS)
   ArchWiki's "dm-crypt / Encrypting an entire system"
   Wikipedia's "VeraCrypt"
   Loop-AES README
   
   

   BitLocker

   
   Wikipedia's "BitLocker"
   
   Dislocker (might be in distro's repo)
   "man libbde-utils" and "bdemount"
   Hasleo's BitLocker Anywhere For Linux
   Versions 2.3.0+ of cryptsetup support Bitlocker [but not Encrypt-On-Write conversion mode].
   
   Get recovery key:
   From Microsoft account (Devices / View details / Bitlocker data protection / Manage recovery keys).
   Or (on Windows, run as Administrator) "manage-bde -protectors -get c:" (recovery key is shown as "password").
   Also could get keyfile (.bek file) ? Don't know how.
   Microsoft's "manage-bde"
   See encryption status: run as admin "manage-bde -status".
   

   
   
   With dm-crypt / LUKS, see partition type "crypt" in lsblk.
   With VeraCrypt, see partition type "dm" in lsblk.
   
   Device mapping also can be used to implement other things, such as:
   block integrity-checking: dm-integrity
   
   Network Block Device (NBD):
   nbdkit
   
   Some software (LUKS) has a recognizable header on the encrypted volume; others (VeraCrypt, Loop-AES) do not. Not having a header may make it harder to attack, and may give plausible deniability. [But with LUKS see Shufflecake.]
   
   
   
10. Physical partitions: e.g. /dev/sda5, /dev/sdb1, /dev/nvme0n1p3.
    And a partition table (Master Boot Record (MBR, sometimes called DOS partition table) or GUID Partition Table (GPT)) to list the partitions.
    
    "hwinfo --short --block"
    "lsblk"
    Volume and partition IDs are different:
    "lsblk --output NAME,PATH,MOUNTPOINTS,UUID,PARTLABEL,PARTUUID,TYPE"
    
    From someone on StackExchange:
    "Volume implies formatting and partition does not. A partition is just any continuous set of storage sectors listed in some table (e.g. MBR or GPT). A volume is a set of sectors belonging to the same filesystem, i.e. an implemented filesystem.".
    
    A key thing I would put at this level: for container files, command "losetup" makes a regular file appear as a block device (misleadingly named as a "loop" device). Try "losetup --list".
    
    To see device block size (but it may lie to you): "sudo blockdev --getbsz /dev/sdaN"
    "cat /sys/block/sda/queue/logical_block_size"
    "cat /sys/block/sda/queue/physical_block_size"
    "udevadm info -a -n /dev/nvme0n1p2"
    
    To see performance stats every N seconds:
    "iostat -xz 10" (from sysstat package)
    "watch -n 1 iostat -xy --human 1 1"
    
    Sergio Prado's "RPMB, a secret place inside the eMMC"
    
    
    
11. Physical devices: e.g. /dev/sda, /dev/nvme0n1.
    
    
    "sudo tlp-stat --disk"
    
    
    

---

---

11. Disk hardware striping/mirroring (if any). E.g. hardware RAID.
    Wikipedia's "RAID"
    Wikipedia's "RAID-Z" (with ZFS)
    
    Note: each type of controller probably has its own custom format for data on disk. So if your hardware dies, you have to replace with identical hardware, you can't just attach the disks to any random system.
    
    
12. Intermediate/bridge physical interfaces to media: e.g. USB, SCSI, RAID controller.
    Try doing "cat /proc/scsi/scsi" and "lsusb" and "udevadm info -a -n /dev/nvme0n1p2".
    
    
13. Physical interfaces to media: e.g. IDE, SCSI, SATA.
    Wikipedia's "Hard disk drive interface"
    
    
14. Disk controller and firmware.
    
    Some drives support eDrive or Opal encryption (AKA "self-encrypting drive"; SED).
    Generally the system BIOS has to support using it.
    ArchWiki's "Self-encrypting drives"
    
    Most/all SSDs do block-remapping at the firmware level, to do wear-leveling.
    
    
15. Raw media: e.g. spinning hard disk, SSD, flash drive (MMC or UFS), SD card, CD-ROM, DVD-ROM, floppy disk.
    Appears as e.g. /dev/sda, /dev/sdb, /dev/nvme0n1 (SSD), /dev/vda (in a VM), /dev/mmcblk0 (MMC device).
    "udevadm info /dev/nvme0n1"
    "lsblk --json -O /dev/nvme0n1p1"
    "udisksctl info -b /dev/nvme0n1"
    "udisksctl info -b /dev/nvme0n1p1"
    Use GParted View/DeviceInformation to see info.
    Use "badblocks" (not on SSD) to test for bad blocks on a raw empty partition.
    
    

Thomas Krenn's "Diagram of Linux Storage Stack"
Thomas Krenn's "Diagram of Linux I/O Stack"

Example 1, my Linux Mint 19.2 system:

$ df -hT
Filesystem             Type      Size  Used Avail Use% Mounted on
/dev/sda5              ext4       33G   25G  7.0G  78% /
/dev/sda6              ext4      259G  182G   65G  74% /home
/dev/sda1              ext4      945M  175M  705M  20% /boot
/home/user1/.Private   ecryptfs  259G  182G   65G  74% /home/user1

1. Standard Linux organization, except my personal files under /home.
2. Standard Linux filesystem API.
3. My password manager database file "KeePassDatabase.kdbx" is app-encrypted.
4. It is under "/home/user1", which is the mount location of an eCryptfs filesystem stored on "/home/user1/.Private".
5. "/home/user1/.Private" is using upper filesystem format eCryptfs.
6. The base filesystem format of "/home" is ext4.
7. Manager: none; not using LVM or RAID.
8. Device-mapper and full-volume/block-level encryption: none ?
9. Physical partitions: /home is on /dev/sda6.
   Partition table on /dev/sda is a Master Boot Record (MBR) table.
10. Disk hardware striping/mirroring: none.
11. Intermediate/bridge physical interfaces to media: SCSI.
12. Physical interface to media: SATAII / Serial-ATA/300.
13. Disk controller: whatever is on the circuit board attached to the disk (probably mainly some FPGA chip); no hardware encryption.
14. Raw media: Western Digital model "ATA WDC WD3200BEVT-7" spinning hard disk, 298 GiB (320 GB), 5400 RPM 2.5" diameter, appears as /dev/sda.

Example 2, a VeraCrypt container mounted in my Linux Mint 19.2 system:

$ df -ahT
Filesystem             Type             Size  Used Avail Use% Mounted on
/dev/sda6              ext4      259G  182G   65G  74% /home
/home/user1/.Private   ecryptfs  259G  182G   65G  74% /home/user1
/dev/mapper/veracrypt1 ext4             2.0G  1.1G  750M  60% /media/veracrypt1

1. Standard Linux organization, except my personal files under /home.
2. Standard Linux filesystem API.
3. Plaintext file "MyBankInfo.txt" is in a 2.0GB VeraCrypt container on /home/user1.
4. It is under "/media/veracrypt1", which is the mount location of an ext4 filesystem stored on "/dev/mapper/veracrypt1".
5. "/dev/mapper/veracrypt1" is using upper filesystem format ???.
   Both VeraCrypt and eCryptfs are in here somewhere.
6. The base filesystem format of "/dev/mapper/veracrypt1" is ext4 ?
7. Manager: none; not using LVM or RAID.
8. Device-mapper and full-volume/block-level encryption: none ?
9. Physical partitions: /home is on /dev/sda6.
   Partition table on /dev/sda is a Master Boot Record (MBR) table.
10. Disk hardware striping/mirroring: none.
11. Intermediate/bridge physical interfaces to media: SCSI.
12. Physical interface to media: SATAII / Serial-ATA/300.
13. Disk controller: whatever is on the circuit board attached to the disk (probably mainly some FPGA chip); no hardware encryption.
14. Raw media: Western Digital model "ATA WDC WD3200BEVT-7" spinning hard disk, 298 GiB (320 GB), 5400 RPM 2.5" diameter, appears as /dev/sda.

My view on it is that there are no layers. There are just different combinations, abstractions, attachments, slices and mirrors of block devices. Upon which you can either build other block devices, or store raw data which could include filesystems.

...

The root of it is that the Linux block device is the base unit and since the other entities present block devices as their product, it gets confusing since the system is making block devices from other block devices and parts of block devices.

...

The first two items in #5 [VeraCrypt containers; eCryptfs] are special types of filesystems, but the 3rd thing [Windows' Encrypting File System (EFS)] is referring to something that becomes a block device. Once it is a block device, then it can be used wherever a block device is used.

#6 is talking about filesystems and "partitions". But it's only a partition if it is referred to in a partition table (GPT, MBR, Sun, SGI, BSD). And even then, the OS only sees that data through the lens of a block device. See "man fdisk".

Trying to represent this as layers breaks pretty fast. For example with LVM, the LV is in a VG. And a VG encompasses one more more PVs. An LV can be spread across multiple PVs.

As I say, in the end actual data is on storage that shows up in Linux as a block device. http://www.haifux.org/lectures/86-sil/kernel-modules-drivers/node10.html

> [me trying to defend layers:]
> For example, can a VeraCrypt container be below (contain) a LVM
> volume ? I don't think so, but maybe I'm wrong.

In Linux, VeraCrypt can encrypt a file. That file can contain general data, a filesystem, or a partition table that divides up the file into partitions.

Also as a file, you can attach it to a loop device and then you can use that as an LVM PV (physical Volume) -- the first bullet here: https://www.veracrypt.fr/en/Home.html

If you boot from USB, how to mount LVM/LUKS hard disk

random neuron misfires' "HOWTO mount an external, encrypted LUKS volume under Linux"
Vivek Gite's "Linux mount an LVM volume / partition"

apt list | grep lvm2/ | grep installed
# if not:
sudo apt install lvm2

lsmod | grep dm_crypt
# if not:
sudo modprobe dm-mod

# encrypted LUKS volume contains an encrypted LVM

# do LUKS
lsblk
cryptsetup luksOpen /dev/sda6 VGNAME  # arbitrary GROUPNAME "VGNAME"
# give passphrase
stat /dev/mapper/VGNAME
sudo mkdir -p /mnt/VGNAME

mount /dev/mapper/VGNAME /mnt/VGNAME/
# should get "mount: unknown filesystem type 'LVM2_member'"

# do LVM
sudo vgscan               # find info about LVM devices
# should see LV == "VGNAME", VG == something like "vgubuntu-mate"
sudo vgchange -a y VGNAME   # activate LVM group "VGNAME"
sudo lvdisplay
sudo lvs
# see VGPATH something like "/dev/dm-2"
ls -l /dev/vgubuntu-mate/   # suppose it has devices home, root, swap
sudo mkdir -vp /mnt/VGNAME/{root,home} # create mount points
sudo mount /dev/dm-2 /mnt/VGNAME/root

df -T | grep VGNAME
ls -l /mnt/VGNAME/root

sudo umount /dev/dm-2
sudo vgchange -a n VGNAME   # de-activate LVM group "VGNAME"

Filesystem Types

# List filesystem types available to load:
ls /lib/modules/$(uname -r)/kernel/fs

# List filesystem types currently loaded:
cat /proc/filesystems

# Can't list all available FUSE filesystem types;
# any app could make a type available at any time.

# List FUSE filesystem types currently loaded:
mount | grep 'fuse\.' | cut -d ' ' -f 5 | uniq

# Descriptions of filesystem types:
man fs

• ext4:
  
  Has journaling to get back to a sane state in case of a sudden crash.
  Does not checksum file contents to catch errors; does checksums on journal and metadata.
  File/directory names can contain any character except null.
  To see filesystem parameters: "man dumpe2fs" and -h option.
  For low-level surgery on a damaged filesystem: "man debugfs", "man ext4magic".
  
  Show bad blocks: "sudo dumpe2fs -b /dev/mapper/data-root"
  Show journal size: "sudo dumpe2fs -h /dev/mapper/data-root | grep 'journal size'"
  
  
  
• ZFS:
  
  Has journaling to get back to a sane state in case of a sudden crash.
  Checksums file contents to catch errors.
  Has hooks to support encryption.
  File/directory names can contain any character except null.
  
  ZFS is a somewhat-new-on-Linux [in 2020] system that integrates several layers (logical volume manager, RAID system, and filesystem) into a unit, and includes features such as checksums and snapshots and copy-on-write. Mostly oriented toward server/enterprise. And it requires letting ZFS manage the entire disk; you can't put ZFS in just one partition.
  
  A "scrub" (on mounted filesystem) reads all blocks and looks for checksum errors. By default, Ubuntu will do a scrub once a month. "sudo zpool scrub bpool" "sudo zpool scrub rpool" Error will be reported in system log ? Also see them by doing "zpool status".
  
  No equivalent of fsck ?
  
  Better to use LUKS full-disk encryption, or ZFS encryption ?

  
  Some factors:
  
  

  • There are well-known ways to automatically-decrypt LUKS (TPM, hardware key). Is possible to do same for encrypted ZFS, but less common.
    
    
  • Encrypted ZFS can do a still-encrypted "send" to an untrusted backup server. And replicate back to do a restore, without backup server ever seeing decrypted data.
    
    
  • Encrypted ZFS leaves some metadata unencrypted.
    
    
  • Maybe some benefit to having one subsystem (ZFS) instead of two (ZFS + LUKS) or three (ZFS + LUKS + LVM) ? [Not sure having three ever would make sense.]
    
    
  • LUKS2 has a better passphrase hashing / key derivation algorithm than ZFS.
    
    
  • LUKS format has no header, so some deniability.
    
    
  • LUKS supports multiple key-slots.
    
    
  • No way to have an encrypted swap partition if you're not using LUKS. But you could use a swap-file, which would be in the encrypted ZFS volume.
    
    
  • [As of mid-2022] Still some minor edge-cases with ZFS being fixed.
    
    

  
  To check for encryption: run "zpool status POOLNAME". If you see something along the lines of gptid/SOMELONGGUID.eli, the .eli is a good sign the pool is encrypted. Or: run "zfs get encryption POOLNAME".
  
  Get sector-size of pool: "sudo zpool get ashift rpool". 12 == 4 KB.
  
  Get full volume name of root fs: "zfs list /".
  
  See used/free space on filesystem:
  "zfs list -o space"
  
  See snapshots:
  "zfs list -t snap"
  or:
  "sudo zpool set listsnapshots=on rpool"
  "zfs list"
  
  "systemctl status zsys-gc.timer zsys-gc.service" cleans up snapshots.
  Also: "apt info zfs-auto-snapshot"
  
  Jim Salter's "ZFS 101 - Understanding ZFS storage and performance"
  zectl
  Jim Salter's "A quick-start guide to OpenZFS native encryption"
  Vladimir Artiukh's "Why is ZFS File System in Linux Ubuntu So Good?"
  Vladimir Artiukh's "How to Open a ZFS Disk in Windows"
  Didier Roche's "ZFS focus on Ubuntu 20.04 LTS: ZSys general principle on state management"
  Chris Siebenmann's "Unfortunately, damaged ZFS filesystems can be more or less unrepairable"
  storytime.ivysaur.me's "Why not ZFS"
  Anarcat's "A ZFS migration"
  kimono-koans' "A Somewhat Opinionated Guide to Effective ZFS Snapshots"
  kimono-koans' "New ways to make ZFS/btrfs snapshots UNIX-y"
  kimono-koans / httm (find files in snapshots)
  Sanoid (also includes syncoid and findoid)
  /r/zfs
  
  
  
  
• Btrfs (pronounced "butter-f-s"):
  
  Has journaling to get back to a sane state in case of a sudden crash.
  If copy-on-write enabled, checksums file contents to catch errors.
  Does not have hooks to support encryption; do encryption outside Btrfs.
  File/directory names can contain any character except null or "/".
  May have up to three copies of the "superblock", for redundancy in case of error ?
  
  Paraphrased from 2.5 Admins episode 03 podcast:
  Btrfs may be fine for non-RAID desktop systems, but there are serious issues with its RAID-type behavior; use ZFS instead for RAID-type situations.
  
  Also, Btrfs may define its RAID types slightly differently than other systems do ? Read the definition carefully if you're going to use one.
  
  Apparently Btrfs has one key feature that both gives it a bad reputation and makes it attractive to me: it checksums all files to verify the contents. So if your disk gets corrupted in a way that affects only data blocks of some files, other filesystem types will ignore those problems, but Btrfs will report them. In the past, some people have interpreted that as "Btrfs has/causes data corruption". [ZFS also has data checksums.]
  
  When creating a Btrfs filesystem, there is a "--data" option; default is "single". If you specify "dup", each data block is written to two locations on disk, so you might be able to recover if some sectors start going bad. An SSD may defeat use of "dup". Specifying "dup" will reduce performance. I always just use the default.
  From someone on reddit:
  "[Dup] only helps you when the storage device messed with your data; if your CPU [or software] is actively writing garbage it'll probably write dup garbage just fine."
  
  When creating a Btrfs filesystem, there is a "--metadata" option; default for single devices is "dup".
  
  On an existing filesystem, I think "sudo btrfs filesystem usage PATH" will show if dup is enabled for either data or metadata.
  
  From someone on reddit 1/2022:
  'In "sudo btrfs filesystem usage MOUNTPOINT", you want "metadata" to be "dup", not "single"'.
  Before Btrfs 5.14, it was "single" on SSDs. You could change settings on an existing volume via "sudo btrfs balance start -mconvert=dup MOUNTPOINT". Took about 3 minutes on a 500 GB USB HDD. [Maybe also do this on any device that has an encryption layer such as LUKS ? https://github.com/kdave/btrfs-progs/issues/319 ]
  
  The minimum size of a Btrfs filesystem is 109 MB, but if you use the "--mixed" option on mkfs.btrfs the limit decreases to 16 MB.
  
  Check status of Btrfs filesystem:

  
  #------------------------------------------------------------
  # Real drive:
  
  # sudo mkfs.btrfs --mixed -b 20000000 /dev/sdb
  # sudo mkfs.btrfs --mixed /dev/sdb		# fill whole device
  
  sudo btrfs filesystem show /dev/sdb
  sudo mount -o noatime /dev/sdb /mnt
  btrfs filesystem df /mnt
  sudo btrfs property get /mnt
  sudo cp /bin/ls /mnt/abc
  lsattr /mnt		# a capital C will show up if COW is DISABLED
  sudo umount /dev/sdb
  
  # partial check (mounted device only):
  sudo mount -o noatime /dev/sdb /mnt
  sudo btrfs scrub start -r /dev/sdb	# returns immediately, work done in background
  sudo btrfs scrub status /dev/sdb	# see status of background processing
  # any bad blocks/files will be reported in output of dmesg ?
  
  # extensive check (unmounted device only):
  sudo umount /dev/sdb
  sudo btrfs check --readonly --check-data-csum --progress /dev/sdb
  
  btrfs-assistant     # Arch ?
  
  
  #------------------------------------------------------------
  # VeraCrypt volume or container with Btrfs filesystem inside:
  
  sudo btrfs filesystem show /dev/mapper/veracrypt5
  btrfs filesystem df /media/veracrypt5
  sudo btrfs property list /media/veracrypt5	# all possible properties
  sudo btrfs property get /media/veracrypt5	# existing properties
  lsattr /media/veracrypt5	# a capital C will show up if COW is DISABLED
  
  sudo btrfs scrub start -r /dev/mapper/veracrypt5
  sudo btrfs scrub status /dev/mapper/veracrypt5
  # any bad blocks/files will be reported in output of dmesg ?
  sudo btrfs check --readonly --check-data-csum --force --progress /dev/mapper/veracrypt5
  sudo btrfs check --readonly --force --progress /dev/mapper/veracrypt5
  

  
  Compression:

  
  Compression is enabled if the filesystem was mounted with the "-o compress" or "-o compress-force" option specified. Also compression can be enabled on a single file via "chattr +c FILENAME".
  
  The type of compression can be specified by an option such as "compress=zlib:1" when the filesystem is mounted. [Consensus of recommended options seems to be "compress-force" and "zstd:1". In /etc/fstab flags, that would be "compress-force=zstd:1". This forces use of zstd's "incompressible data detection feature", which is better than Btrfs's equivalent.]
  
  From someone on reddit:
  "Use zstd:1 for NVMe SSD, zstd:2 for SATA SSD, and default (:3) for HDD.
  For fast SSD, the default zstd (level 3) will bottleneck SSD speeds."
  
  Compression type for an individual file can be specified via "btrfs property set FILENAME compression TYPE". Compression doesn't occur until the next time the file is written.
  
  To see how much a file is compressed: "sudo compsize FILENAME".
  
  Compression imposes a performance penalty.
  
  If you enable or change compression, and want to compress all existing files too: after the filesystem is mounted with new compression specified, do "sudo btrfs filesystem defragment -r MOUNTPOINT".
  
  Btrfs Wiki's "Compression"
  

  
  Jim Salter's "Examining btrfs, Linux's perpetually half-finished filesystem"
  Btrfs FAQ
  Btrfs SysadminGuide
  Ubuntu Community Help Wiki's "btrfs"
  ArchWiki's "Btrfs"
  Willi Mutschler's "Ubuntu 20.04 with btrfs-luks full disk encryption"
  Willi Mutschler's "Fedora Workstation 33 with btrfs-luks full disk encryption ..."
  Troy Curtis Jr's "Choose between Btrfs and LVM-ext4"
  Deep Grewal's "Btrfs Guide: Getting Started with Btrfs"
  nickavem's "Btrfs transparent compression"
  
  Better performance if you mount Btrfs filesystem with "noatime" flag specified, to avoid forcing COW of metadata when you do read-access to files. See Btrfs Wiki and Jonathan Corbet's "Atime and btrfs: a bad combination?". More-aggressive step: also specify "lazytime".
  
  From someone on reddit:
  "btrfs-specific mount options such as nodatacow do not work per subvolume. [For example] You need to use chattr +C on the Downloads directory and swap-file."
  
  Recommendations vary about specifying "autodefrag" when mounting Btrfs filesystem. Default is off; probably best to leave it that way. Definitely don't do it on an SSD. Among other things, a defrag breaks any reflinks/snapshots. Jesse Smith answer
  
  From someone on reddit 12/2020:

  
  Re: Btrfs on SSD:
  
  Modern SSDs (I assume you bought a reputable brand and a decent size greater than 256 GB) have specified lifetimes measured in the several hundred TBW. This means you can write the entire disk once a day for a few years before running out. Everything written 5-10 years ago about SSD lifetimes is obsolete and can be completely ignored (a lot of internet writers still repeat it without thinking however).
  
  Btrfs will automatically implement some basic SSD optimizations, and you should probably enable a weekly (or so) fstrim service (most distros will do this by default). Other than this, you don't need to do anything special at all to keep your SSD healthy for typical desktop usage.
  
  If you bought a small-ish SSD, you might consider enabling Btrfs zstd compression to get a little extra space (this is an advantage over ext4).

  From someone else on reddit: set metadata to single; SSDs likely internally dedupe their blocks.
  
  Apparently some systems (e.g. SUSE with /etc/sysconfig/btrfsmaintenance) do periodic balances. Maybe a good idea to turn those off.
  
  See which features of Btrfs are considered 100% stable in latest released Linux kernel: Btrfs Wiki's "Status"
  
  Driver to let Windows access a Btrfs volume:
  WinBtrfs
  Paragon Software's "Btrfs for Windows"
  
  Sub-volumes:

  
  "Free space on your Btrfs partition (volume) is shared across all its sub-volumes. So it's best to have a single large Btrfs partition."
  
  The unstated secret about how desktop Linux uses Btrfs: In the normal "/" directory of the filesystem are subvolumes called "@" and "@home" or other similar names, and any snapshots.
  What the user sees as "/" really is "@" mounted as "/".
  What the user sees as "/home" really is "@home" mounted as "/home".
  
  Apparently the names and number of sub-volumes differ from one distro to another; there is no standard.
  
  Having many sub-volumes can make it harder to repair a system where you need to do a "chroot" to get access to files ? Does chroot know about nested sub-volumes ?
  
  "The concepts of 'snapshots' and 'subvolumes' in Btrfs are all just virtual different versions of the same filesystem." and "You can make a snapshot of any subvolume."
  
  From someone on reddit:

  A subvolume is a special folder or tree. It can operate as a separate filesystem root, as in, you can mount that subvolume as if it were a separate disk+filesystem.
  
  It's special because it has special semantics, like atomic deletes, snapshots, etc.
  
  Most filesystems have a single root, that is, they present the same data regardless on how you mount it, but as previously mentioned, subvolumes are independent filesystem roots, which can be mounted at will, anywhere.

  
  From Btrfs SysadminGuide:
  "A Btrfs subvolume is an independently mountable POSIX filetree and not a block device (and cannot be treated as one). Most other POSIX filesystems have a single mountable root, Btrfs has an independent mountable root for the volume (top level subvolume) and for each subvolume; a Btrfs volume can contain more than a single filetree, it can contain a forest of filetrees. A Btrfs subvolume can be thought of as a POSIX file namespace."
  
  From Fedora Project Wiki's "BtrfsByDefault":
  "Subvolumes don't have size, they act mostly like directories, space is shared."
  
  Typically sub-volume names start with "@", to distinguish them from directory names. Ubuntu uses this convention, and some apps such as Timeshift depend on it.
  
  From someone on reddit 7/2021:

  > Somehow I've managed to do a manual install of PoP!_OS on Btrfs
  > (on a LUKS-encrypted partition), with no subvolumes.
  
  Every Btrfs filesystem has a root or "top-level" subvolume with the ID 5. This is where you installed your OS. When subvolumes are created for /, /home, /etc ... it is done by the operating system installer based on the decisions of the distro maintainers or manual user configs. If you did not do this, it did not get done.

  
  Jonathan Corbet's "The Btrfs inode-number epic"
  

  
  Backups and snapshots:

  
  Btrfs can do "snapshots", but they're just markers that reside in the filesystem. So they're not "backups" unless you send copies of them to some other device. And Timeshift can be set to do Btrfs snapshots instead of rsync file-copying. Docker images can be set to do Btrfs snapshots too.
  
  Backup tools: btrbk, snap-sync. Maybe snapper ?
  
  No matter how you make snapshots, snapper is a good CLI tool for managing and deleting snapshots. Snapper is already set up by default to do size-based and number-based cleanups; see /etc/snapper/configs/root
  
  Make sure databases are closed (associated apps shut down) when taking a snapshot, else you may get invalid/inconsistent data. True for password manager, email client, database server app, etc. For serious databases, maybe use a specific database-backup utility to make a backup file, then do the snapshot.
  
  Alessio Ciregia's "Incremental backups with Btrfs snapshots"
  nachoparker's "Easy sync Btrfs snapshots with btrfs-sync"
  Alessio Ciregia's "Recover your files from Btrfs snapshots"
  nachoparker's "How to recover a Btrfs partition"
  linux dabbler's "How to install Debian with BTRFS subvolumes and restore with TimeShift" (video)
  EndeavourOS's "Installation on BTRFS"
  kimono-koans / httm (find files in snapshots)
  
  From someone on reddit 10/2020:

  Don't enable quota groups, since this will cause significant performance degradation and even system freezes if you have multiple snapshots. For that reason I ditched Timeshift in favor of snapper, since Timeshift enables quota groups by default and without this Timeshift has an issue deleting old snapshots.

  Also, if you have a large number of snapshots (say, more than 100), some Btrfs operations such as balance or add/remove drive or delete snapshots will become VERY slow.
  
  Problem if you snapshot all of root filesystem: you'll be saving large variable storage such as the package manager cache and log files, which could take a lot of space. How to avoid that ? Lots of work-arounds proposed, but I favor: empty the caches before you do a snapshot.
  
  From someone on reddit 12/2020:

  
  Snapshots are per-subvolume.
  
  I have one subvolume for my root system, and one for my home folder.
  
  These each get snapshotted separately.
  
  The root gets snapshotted regularly and every time I install a package on my system.
  
  My home gets snapshotted at regular intervals.
  
  If a system update borks my system. I can boot from the previous snapshot.
  
  If I accidentally delete a file I can either roll my whole home directory back to less than 1 hour ago, or I can go into the snapshot and pick out the specific file I lost.
  
  Snapshots capture the entire subvolume instantaneously, but you decide how you want to restore, you can restore everything, or just bits and pieces. [By mounting the snapshot read-only and then copying files across to the active volume, I think.]

  
  

  
  # List all subvolumes and snapshots:
  sudo btrfs subvolume list /
  
  # List snapshots:
  sudo btrfs subvolume list -s /
  
  # Make a snapshot on the same disk:
  sudo btrfs subvolume snapshot -r /home /snapshots/home-$(date +%Y%m%d)
  
  # Make a snapshot on another drive:
  sudo btrfs subvolume snapshot -r /home /mnt/SOMEOTHERDISK/snapshots/home-$(date +%Y%m%d)
  
  # Send an existing snapshot to another drive:
  sudo btrfs send /snapshots/home-DDDDDDDD | sudo btrfs receive /mnt/SOMEOTHERDISK/snapshots
  
  # See subvolid's of all mounted volumes:
  mount | grep subvolid=
  
  # Delete a snapshot on the same disk:
  # Get ID from first column of:
  sudo btrfs subvolume list -s /
  # Make sure not mounted:
  mount | grep subvolid=IDNUM
  sudo btrfs subvolume delete --subvolid IDNUM /
  
  # To see REAL root of filesystem, not @:
  # Get "top level number" from:
  sudo btrfs subvolume list /
  # Get disk device name with UUID from:
  mount | grep ' / '
  # Then:
  sudo mkdir /mnt/toplevel
  sudo mount -o subvolid=TOPLEVELNUM /dev/disk/by-id/DISKUUID /mnt/toplevel
  # or:
  sudo mount -o subvolid=TOPLEVELNUM /dev/mapper/DISKUUID /mnt/toplevel
  ls /mnt/toplevel
  umount /mnt/toplevel     # don't delete it, it's all of your files
  
  # To replace @ and @home with old snapshots of them:
  # Get /mnt/toplevel mounted as shown above.
  cd /mnt/toplevel
  sudo bash
  ls -d @* snapshots/*
  # Move the broken stuff aside:
  mv @ @old
  mv @home @homeold
  # Make the snapshot rw:
  btrfs property set -ts snapshots/@SNAPSHOTNAME ro false
  btrfs property set -ts snapshots/@homeSNAPSHOTNAME ro false
  # Replace the current subvolumes with your snapshots:
  mv snapshots/@SNAPSHOTNAME @
  mv snapshots/@homeSNAPSHOTNAME @home
  # Reboot now.
  

  
  From someone on reddit 10/2020:

  If you are using databases (MySQL) or virtualization (KVM / VirtualBox), mark the directory used to store their data/images so as to disable copy-on-write (chattr +C) to prevent fragmentation and potential performance degradation. [Someone else says: you have to mark every existing file in the directory, too. And nodatacow doesn't work if you're using compression. And you can only mark a zero-length file, not one that has contents.]
  
  > Maybe do this on a moderate-sized (300 MB) VeraCrypt container file ?
  
  I would say it's not worth stressing about. It's not strictly required, especially if you are on an SSD (which does not suffer the same seek-time issues due to fragmentation).

  
  "Unlike ZFS send/receive, Btrfs send/receive will not resume an interrupted transaction. So if you pipe over SSH, and your network hiccups, you'll have an incomplete backup with no notification about it."
  
  Willi Mutschler's "Why I (still) like Btrfs"
  kdave / btrfsmaintenance
  
  Display superblock:
  "sudo btrfs inspect-internal dump-super DEVICE >superblock.DEVNAME.txt"
  
  Periodic maintenance:

  
  I'm told that Btrfs does so much verification that a fsck-like check at boot time is unnecessary.
  
  Maybe once per month, do a read-only check of used (not free) data and metadata blocks, using checksums, on mounted filesystem: "sudo btrfs scrub start -r MOUNTPOINT", and then "sudo btrfs scrub status MOUNTPOINT" as it runs. Any bad blocks/files will be reported in output of dmesg ?
  
  Also do "sudo btrfs device stat MOUNTPOINT"
  
  A check of filesystem structure, supposed to be done on an unmounted filesystem: "sudo btrfs check DEVNAME". Really intended as part of repair process after a problem is found.
  
  Scripts and systemd services to do tasks: kdave / btrfsmaintenance
  
  How to monitor BTRFS filesystem raid for errors?
  
  Manual defragmentation: "sudo btrfs filesystem defragment -r MOUNTPOINT". Not sure this is a good idea; it may increase disk space consumed. Definitely don't do it on an SSD.
  
  Some people say do a "balance" of data (not metadata) every now and then; others say do it only if you have problems because filesystem is running out of space in one part. "sudo btrfs balance start MOUNTPOINT", and then "sudo btrfs balance status MOUNTPOINT" as it runs. Operation with no filters takes a VERY long time (about 15 minutes on a 4 GB USB stick with 560 files), and I assume very bad things can happen if you kill it early.

  
  Recovery and repair:

  
  "sudo dmesg | grep BTRFS | grep -E 'error|warning|failed'"
  "sudo btrfs device stat MOUNTPOINT"
  
  If scrub gives uncorrectable errors, do SMART test. If SMART test shows errors, drive is failing. If SMART doesn't show errors, the scrub errors came from elsewhere in system: CPU or RAM bad or overheating or something.
  
  Make sure the drive is not throwing hardware errors. Maybe use a SMART utility.
  Aaron Kili's "How to Check Bad Sectors or Bad Blocks on Hard Disk in Linux"
  Oleg Afonin's "Predicting SSD Failures: Specific S.M.A.R.T. Values"
  
  If filesystem says can't create files but there is free space, maybe do a "balance" to adjust the spaces allocated to data and metadata.
  
  
  [I tried the following on a test filesystem on a USB stick. Maybe you should do the same before using them on any filesystem containing valuable data.]
  
  
  Read-only check of data and metadata blocks, using checksums, on mounted filesystem: "sudo btrfs scrub start -r DEVNAME".
  
  "You can only use scrub if you can still mount the filesystem." and "It's simple, read raw data, compute and check checksum, and nothing else."
  
  Check and fix data and metadata blocks, using checksums, on mounted or unmounted filesystem:
  "sudo btrfs scrub start DEVNAME".
  Fixing can occur only if there is a good copy available (RAID ?).
  Any bad blocks/files will be reported in output of dmesg ?
  
  Show status of in-progress scrub or last scrub done:
  "sudo btrfs-scrub status DEVNAME".
  
  Full read-only check of unmounted filesystem:
  "sudo btrfs check --readonly --check-data-csum --progress DEVNAME".
  
  Check is equivalent of fsck.
  
  A little dangerous, deletes unwritten transactions: remove log on unmounted filesystem:
  [Check man page to see if applies to your situation.]
  "sudo btrfs rescue zero-log DEVNAME.
  
  
  Somewhat dangerous: Full check and repair of unmounted filesystem:
  "sudo btrfs check --repair --check-data-csum --progress DEVNAME".
  
  
  Desperate attempt to repair unmounted filesystem:
  "sudo btrfs rescue chunk-recover DEVNAME" (other sub-commands too).
  
  Show info from mounted filesystem:
  "sudo btrfs inspect-internal dump-tree DEVNAME" (there are other sub-commands too)
  
  Choose one of the internal superblocks to replace primary, on unmounted filesystem:
  "sudo btrfs-select-super -s 1 DEVNAME"
  
  "btrfs restore" ?
  
  nachoparker's "How to recover a Btrfs partition"
  Btrfsck
  
  From Josef Bacik on a mailing list 6/2020:

  
  The UX of a completely f*cked fs sucks, regardless of the file system. Systemd currently (but will soon apparently) does not handle booting with a read-only filesystem, which is essentially what you get when you have critical metadata corrupted. You are dumped to a emergency shell, and then you have to know what to do from there.
  
  With ext4/xfs, you mount read-only or you run fsck. With Btrfs you can do that too, but then there's like a whole level of other options depending on how bad the disk is. I've written a lot of tools over the years (which are in btrfs-progs) to recover various levels of broken filesystems. To the point that you can pretty drastically mess up a FS and I'll still be able to pull data from the disk.
  
  But, again, the UX for this _sucks_. You have to know first of all that you should try mounting read-only, and then you have to get something plugged into the box and copy it over. And then assume the worst, you can't mount read-only. Now with ext4/xfs that's it, you are done. With btrfs you are just getting started. You have several built-in mount options for recovering different failures, all read-only. But you have to know that they are there and how to use them.
  
  ...
  
  The fact is I can make any filesystem unmountable with the right corruption. The only difference with btrfs is that our metadata is completely dynamic, while xfs and ext4 are less so. So they're overwriting the same blocks over and over again, and there is simply less of "important" metadata for the filesystem to function.
  
  The "problem" that btrfs has is its main strength, it does COW. That means our important metadata is constantly being re-written to different segments of the disk. So if you have a bad disk, you are much more likely to get unlucky and end up with some core piece of metadata getting corrupted, and thus resulting in a filesystem that cannot be mounted read/write.
  
  Now you are much more likely to hit this in a data segment, because generally speaking there are more data writes than metadata writes. ...
  
  ...
  
  All of our filesystems in linux are extremely thoroughly tested for a variety of power-fail cases. The only area that btrfs is more likely to screw up is in the case of bad hardware, and again we're not talking like huge percentage points of difference. It's a trade-off. You are trading a slight increased percentage that bad hardware will result in a filesystem that cannot be mounted read/write for the ability to detect silent corruption from your memory, cpu, or storage device.

  
  From someone on reddit 6/2023:

  
  I hate to say it but I've tried Btrfs several times over the years and had a total loss each time ... it has all this cross-checking to DETECT inconsistencies but then at best goes read-only, and at worst flat-out refuses to mount. No fsck, and no manual procedure either to trim out the last, say, minute of writes and at least regain use of the filesystem with just that lost.

  
  
  Reverting back to a snapshot: There is no "revert the filesystem back to as it was when snapshot X was taken". Instead, you could set /etc/fstab to boot from the filesystem as it was when snapshot X was taken:
  From someone on reddit 12/2020:

  
  You can try booting to the snapshot, make sure it is read/write. Use btrfs subvolume list / and find the ID of the snapshot subvolume. Then add rootflags=subvolid=ID and see if it boots. You may need to comment out your root partition from /etc/fstab but I am not completely sure.
  
  Also, I would recommend organizing your system into subvolumes. If your /home exists on the same partition as your / right now, your rollback will also rollback your personal files.
  
  Where rootflag ? You can edit the boot entry in grub at the menu, then add it to the grub config once your system is rolled back.

  
  "btrfs sub set-default" to roll back to a snapshot ?
  
  Changaco / btrfs-undelete
  
  grub-btrfs
  refind-btrfs
  

  
  Lan Tian's "I Moved My SSD to My New Laptop. This is What Happened To My Btrfs Data"
  
  Thom Holwerda's "Windows on Btrfs"
  
  
  
  
  
• bcachefs:
  
  Under development as of 2022.
  Checksums on file contents and metadata.
  Supports compression, encryption, CoW, snapshots.
  bcachefs
  Tony Asleson's "Bcachefs, an introduction/exploration"
  
  
  
• XFS:
  
  Does not checksum file contents to catch errors; does checksums on metadata.
  File/directory names can contain any character except null.
  Appropriate more for enterprise use: good support for highly parallel I/O.
  
  
  
• F2FS:
  
  Intended for flash (and SSD, I think) devices.
  Supports encryption.
  Does not checksum file contents to catch errors.
  
  
  
• NILFS2:
  
  Intended for flash (and SSD, I think) devices.
  Checksums file contents to catch errors.
  
  
  
• exFAT:
  
  Pawit Pornkitprasan's "Notes on exFAT and Reliability"
  
  
  
• FAT32:
  
  Maximum file size is 4 GB.
  
  
• NTFS:
  
  If you want portability between Windows and Linux, file/directory names can contain any character except null or "/\:*"?<>|".
  NTFS Documentation
  
  
  

It's worth mentioning that while btrfs and zfs both have features that make snapshots easier to take (and a bunch of other awesome features), ext4 is actually more resilient. When ext4 breaks, it's almost always fixable without having to reformat. When zfs and btrfs break, the data is usually recoverable, but it can sometimes require a reformat to get the drive operational again.

Source: I do data recovery professionally.

...

[I asked why: better journaling, tools, what ?]

I'm not 100% sure why ... When it comes to repair, I use the same tools as everyone else. And ext4's pretty much always work (which isn't the case for any filesystem on any OS, from what I can tell). I think ext4 being developed and patched by so many more people, for so many years as the default for pretty much all of Linux, and as a result of its ubiquitousness, has resulted in a rock-solid piece of technology that's almost impossible to break.

It's worth noting that NTFS on Windows can be broken beyond repair, and we regularly see that. As can Apple's filesystems, APFS and HFS+ (HFS+ was actually surprisingly fragile).

...

I'm talking about worst-case scenarios anyway. As someone who does data recovery professionally, nobody calls me when things are going well, lol. Btrfs is a fantastic, and almost always stable, filesystem. ZFS even more so (just because it's a more mature code-base).

I also install, configure, and maintain Linux systems professionally (both for standard desktop users and servers). And the majority of filesystem errors on all common Linux filesystems are repairable, even when caused by power outages or hardware failure (which are the worst-case scenarios for a filesystem). My comments were mostly meant to highlight the bulletproof nature of ext4, not to call out the next generation filesystems.

...

[Guessing from a comment by someone else: Advanced filesystems such as Btrfs and ZFS may be totally fine and repairable if you use them as simple filesystems. Perhaps it's when you start using them "full stack" and RAID and such and then have a hardware failure that you can get into rare irreparable situations.]

# Show bind mounts:
awk '$4 != "/" {printf("%-20s -> %-20s\n",$4,$5)}' /proc/self/mountinfo 

Encryption of data at rest

Things you could encrypt

• A single file (maybe use 7zip, GnuPG, ccrypt).
  
  
• An archive of files (maybe use 7zip, rar).
  
  
• A directory tree (maybe use eCryptfs, gocryptfs, zuluCrypt, Plasma's Vault).
  
  
• A container file that you then mount as a data disk (LUKS, VeraCrypt, zuluCrypt).
  
  
• A disk partition (maybe use LUKS, VeraCrypt, zuluCrypt).
  
  
• An entire data disk (maybe use LUKS, VeraCrypt, zuluCrypt).
  
  Danger: if you attach an encrypted disk to Windows, and Windows doesn't recognize the encryption format, it will assume it's a new unformatted disk and ask if you want to format it.
  
  
• Most of a boot/system disk (LVM/LUKS on Linux, or VeraCrypt on Windows).
  
  
• An entire boot/system disk (hardware encryption).
  
  

How is the plaintext data presented to you ?

• A single file. (GnuPG, ccrypt)
  
  
• A directory tree from a mountable filesystem, perhaps encompassing everything you see from "/" on down, or everything from your home directory on down, or other places. (VeraCrypt, zuluCrypt, LUKS, eCryptfs, gocryptfs, AVFS)
  
  
• Files inside an archive manager application, and you can extract them to the normal directories. (7zip, Engrampa, Archive Manager)
  
  

My strategy

For critical security software, I want: open-source, standard part of stock OS, widely used, lots of devs, simpler.

So for system/boot disk encryption, I use whatever is built into the OS. Usually LVM/LUKS.

For data disks and containers, I had been using VeraCrypt. But I got a little spooked about the TrueCrypt devs being anonymous, VeraCrypt's license status, and I wonder how many devs are working on VC. It's not "simple" in that it has a lot of features I don't need: cross-platform, hidden volumes, encrypted boot disk (on Windows), in-place encryption, many cipher algorithms.

I used to think cross-platform was important to me, but I changed my mind. In an emergency, I can boot from a Linux live-session on a USB stick, and copy needed files to a non-encrypted filesystem.

Then I found that LUKS can do container files as well as partitions, so I switched to LUKS for everything. It seems VeraCrypt is mostly a GUI on top of code similar to LUKS1: all the same features are there in LUKS. And in fact VeraCrypt seems to be sticking with older settings (LUKS1) to preserve compatibility with old software. But using LUKS directly, I have LUKS2. And people are telling me that LUKS2 uses more secure algorithms.

On Ubuntu, software automatically detects a LUKS-encrypted disk and asks for passphrase and opens and mounts it, no scripts needed. For container files, the behavior varies by file manager.

Alternatives

• VeraCrypt   

  
  Volume or container-file encryption.
  See VeraCrypt section below.
  
  
  

• zuluCrypt

  
  Volume or container-file encryption.
  zuluCrypt
  zuluCrypt PDF
  Linux-only.
  "zuluCrypt can create and open 5 types of encrypted volumes: LUKS, TrueCrypt, VeraCrypt, PLAIN dm-crypt, and PLAIN dm-crypt volume at a non-zero offset."
  "zuluCrypt can do two types of encryption. It can do single file encryption/decryption or block device encryption."
  Apparently it can do encrypted container files.
  "zuluMount can auto-detect plugged-in devices and auto-mount them."
  
  
  
  

• LUKS encryption   

  
  Volume or container-file encryption.
  See LUKS section below.
  
  
  

• eCryptfs

  
  Directory encryption.
  Really can be used to create an encrypted directory anywhere, but Mint uses it for home-directories.
  Considered to be unsupported and deprecated ?
  Jack Wallen's "How to install and use eCryptfs on Ubuntu"
  ArchWiki's "eCryptfs"
  Chris Hoffman's "How to Recover an Encrypted Home Directory on Ubuntu"
  Linux_Lobo's "HOWTO: Recover files from encrypted ecryptfs home directory"
  AskUbuntu discussion "How to recover ecryptfs encrypted data"
  man page "ecryptfs(7)"
  
  
  
  

• gocryptfs

  
  Directory encryption.
  "sudo apt install gocryptfs"
  Brian Exelbierd's "How to encrypt files with gocryptfs on Linux"
  gocryptfs man page
  
  
  
  

• Tomb

  
  Directory encryption.
  Aaron Kili's "Tomb - A File Encryption and Personal Backup Tool for Linux"
  
  
  
  

• Cryptmount

  
  Directory encryption.
  Aaron Kili's "Cryptmount - A Utility to Create Encrypted Filesystems in Linux"
  
  
  
  

• EncFS

  
  Directory encryption ?
  Aaron Kili's "Cryptmount - A Utility to Create Encrypted Filesystems in Linux"
  
  
  
  

• KDE Vault

  
  Directory encryption ?
  Derrik Diener's "How To Create Encrypted Folders On KDE Linux Desktop With Vaults"
  
  
  
  

• GnuPG (AKA GPG, AKA GNU Privacy Guard)

  
  Single-file encryption.
  

  
  # generate keys:
  # specify MYEMAILADDRESS to associate with public key
  gpg --gen-key
  
  # generate a public key you can give to others:
  gpg --armor --output mypubkey.gpg --export MYEMAILADDRESS
  
  # To encrypt a file test.txt using RECIPIENTEMAILADDRESS's public key:
  gpg --output test.txt.gpg --encrypt --recipient RECIPIENTEMAILADDRESS test.txt
  
  # To decrypt a file test.txt.gpg using your private key:
  gpg --output test.txt --decrypt test.txt.gpg
  

  
  
  

• ccrypt

  
  Single-file encryption.
  

  
  sudo apt install ccrypt
  
  ccencrypt FILENAME		# creates encrypted file FILENAME.cpt
  
  ccdecrypt FILENAME.cpt	# creates decrypted file FILENAME
  

  
  
  

Archive encryption: "zip -e", zipcloak, gpgtar.
Single-file encryption: "vim -x", aescrypt, bcrypt.

"7z" files: "sudo apt install p7zip-full", and then maybe Archive Manager will handle 7z files. If not, "7za x FILE.7z" or "7za x FILE.7z.001" to extract files.

Cryptomator: started as cloud-only, but now supports local folder encryption ? article

VeraCrypt

Choices and policies

If you want an encrypted Container or partition to be accessible on Windows, choose exFAT or NTFS (not ext*) for the filesystem inside it. There are freeware utilities such as Linux Reader that can read ext* filesystems on Windows, and now WSL2 can do ext4, but maybe it's better to just use a filesystem that Windows can understand natively. Relevant discussion. Article. On Linux, you may lose symlinks when copying from ext4 to exFAT or NTFS ? exFAT is best choice: Linux kernel will be adding deeper support for it mid-2020, and Mac OSX also supports it. On Linux 5.4 kernel, "sudo apt install exfat-fuse exfat-utils". But exFAT doesn't allow some characters in filenames, which complicates things when copying from ext4 to exFAT.

For a big external disk, it's far easier/quicker to make a full-disk/full-partition VC-encrypted volume, rather than create a non-encrypted partition with a filesystem in it and then a VeraCrypt container file in that filesystem. Just leave the drive "raw" (no partitions), use VeraCrypt to make an encrypted volume (Linux: on the drive /dev/sdb, not a partition /dev/sdb1), using VC's "quick format" option. You will have to have the VeraCrypt application already installed on any computer where you want to access that drive, which seems okay. One danger: when you attach such a drive (full-volume VC-encrypted) to a Windows machine, Windows will say something like "drive looks unformatted, want me to format it for you ?". You must be VERY careful to say "no" every time you attach that drive. [Some people say you can disable this behavior via "delete the drive letter in Disk Management".]

If you want extra security, when you create an encrypted container or partition, you could choose a stacked cipher instead of the default AES, and some hash function other than the default HMAC-SHA-512. And if you create multiple containers/volumes, you could use different settings for each. You also could use keyfiles, hidden containers, PIM settings. But at some point you are more likely to fool or confuse yourself rather than some adversary. It might be best to just stick to the defaults, or use just one group of settings and apply it to all of your volumes.

Good practice: right after you create an encrypted container, before you put anything in it, turn off compression, disable COW, turn off atime updating: "chattr -c +C +A FILENAME". These settings are ignored and harmless in filesystems that don't support them.

Good practice: after you create an encrypted container or partition, click the "Volume Tools ..." button and select "Backup Volume Header ..." and save the backup to somewhere else. Might save you if the volume gets corrupted somehow. [Although actually VeraCrypt already maintains a backup copy of the header inside the volume. So you'd have to lose a bunch of blocks to lose both the primary and backup headers inside the volume, and need to use your external backup copy.]

Quirks and issues

A couple of quirks with VeraCrypt containers, at least in Linux: Firefox doesn't properly remember "last folder saved to" if it's in a VC container, and Nemo renaming a folder in a VC container doesn't work if all you're changing is capitalization ?

A quirk with VeraCrypt containers related to cloud backup: When you modify a file in the container, the modified-time and checksum of the whole container change. So if you add a 1-KB file to a 10 GB container, the backup software will say "okay, have to write this whole 10-GB file up to the cloud". (Same is true of any other aggregate format, such as ZIP or RAR.)

I used MEGAsync for a while, but had a couple of bad experiences where somehow it appeared that the file on my laptop (what I considered the master) was older than the copy in MEGAsync (what I considered a backup), and MEGAsync synced the old file down to my laptop, and I lost data. Seemed to happen with VeraCrypt containers in use; I would forget to dismount them, and MEGAsync would see them as old. VeraCrypt on Linux has some quirks with updating the modified time on the container file. https://sourceforge.net/p/veracrypt/tickets/277/

Auto-mount functionality is only for encrypted volumes/partitions, not on containers.
Same with quick-format; not available when creating a container.
To mount a container using CLI:

veracrypt --slot=1 /home/user1/.../MYCONTAINERFILENAME /media/veracrypt1

Apparently there is a longstanding performance issue with VeraCrypt, although it may show up only on heavy multi-threaded benchmarks or under extreme loads, mostly with SSD ? reddit thread

Checking and fixing a VeraCrypt volume

It seems there are really just two things that can go wrong:

• VeraCrypt volume header gets damaged somehow.
  
  Even with correct password, you will get a message "can't mount because wrong password or not VeraCrypt volume".
  
  In this case, maybe copy whole volume to a known-good disk, check original for disk errors, then restore VeraCrypt volume header from a header backup.
  
  
• Contents of VeraCrypt volume after the VC header gets damaged somehow.
  
  The volume will be listed in VeraCrypt as mounted properly, but the operating system will say "unrecognized filesystem" (if the filesystem header is damaged) or something else if blocks after the FS header are damaged.
  
  In this case, maybe copy whole volume to a known-good disk, check original for disk errors, then run filesystem-repair utilities.
  
  

VeraCrypt's "Troubleshooting"

From VeraCrypt's "Frequently Asked Questions":

File system within a VeraCrypt volume may become corrupted in the same way as any normal unencrypted file system. When that happens, you can use filesystem repair tools supplied with your operating system to fix it. In Windows, it is the 'chkdsk' tool. VeraCrypt provides an easy way to use this tool on a VeraCrypt volume: Right-click the mounted volume in the main VeraCrypt window (in the drive list) and from the context menu select 'Repair Filesystem'.

I think the procedure (for a VC container) is:

There are TWO filesystems: the one inside the disk partition (call it FS-D) and the one inside the container file (call it FS-C).

Now, a sector goes bad that happens to be used by the container file.
Do this:

1. First, make sure the disk is not throwing hardware errors. Maybe use a SMART utility. article
2. Run fsck/chkdsk or whatever to repair the disk filesystem (FS-D). The OS might do this automatically. The filesystem has to be unmounted while it's being checked and repaired. (Note: "chkdsk /r" checks much more than "chkdsk /f".)
3. Open VC container without mounting filesystem that's inside it (FS-C): Click Mount button, click Options button to expand the dialog, near bottom check the box "Do not mount". Type password, device will appear in list but the "Mount Directory" column will be empty.
4. If opening the VC container fails, both volume headers inside the container are bad. Use a backup copy of the volume header, that you saved elsewhere. VC has a "repair" function to do this ?
5. Right-click on the device in the list and select either the Check Filesystem or Repair Filesystem menu item. A small terminal window will open and FSCK will run. If instead you get an error dialog "xterm not found", go to CLI and run "apt install xterm", then try again.
6. Mount the container in VeraCrypt, and check dmesg to see that there are no error messages. Nemo does not report dirty filesystems (bad).
7. Then you're good, no need to copy or move the container.

John Wesorick's "Running fsck on a Truecrypt Volume"
CGSecurity's "Recover a TrueCrypt Volume"
Silvershock's "Opening/Decrypting VeraCrypt drives without mounting them (for fsck)"

How secure is a VeraCrypt volume ?

My "veracryptcrack" project
My "veracryptcrack2" project
Oleg Afonin's "Breaking VeraCrypt containers"
NorthernSec / VeraCracker
Thynix / veracrypt-bruteforce
If you need to generate a wordlist: crunch

From someone on reddit:

There's essentially two parts to a VeraCrypt volume, the encrypted data and the encrypted header.

When you move your mouse around during volume creation, you are helping add entropy to generate the keys to the encrypted data.

For multiple reasons, there needs to be a header that tells the volume's size and other important data. It is also encrypted to help hide the fact it's specifically a VeraCrypt volume. The key to that is derived from hashing your password in a protocol called PBKDF2. So as usual with cryptographic systems that have a password, it's the password that is the weakest link. Brute-force attacks will never be used against the keys, but the password instead. So the password must be as strong as possible because that gives you access to the encrypted data.

Resizing a VeraCrypt volume

VeraCryptExpander utility. Windows-only.

First, outside of VeraCrypt (Windows Disk Management), resize the partition. Then run VeraCryptExpander to resize filesystem inside VC volume ?

Make a Btrfs filesystem inside a VeraCrypt volume

I'm using Ubuntu GNOME 20.04.

Install software:

sudo apt install btrfs-prog
man mkfs.btrfs

Important: In the following steps, change device names and labels as appropriate for your system. Best to have no extra removable or optional devices attached while doing operations, to avoid confusion.

Try Btrfs first on a real device (no VC) to make sure it works:

# Attach a USB drive to system.

# Check device name:
sudo dmesg		# probably /dev/sdb
# If you look in file explorer and see that the device got mounted, ignore that.

sudo wipefs --force --all /dev/sdb
# If you look in file explorer and see that the device got mounted, ignore that.

# Make Btrfs filesystem:
sudo mkfs.btrfs --force --label LABEL /dev/sdb
# If it says "ERROR: /dev/sdb is mounted", do "sudo umount /dev/sdb".

sudo btrfs check /dev/sdb

# See that new filesystem got mounted, maybe as /media/user1/LABEL:
df | grep sdb			# doesn't show up here ?
blkid | grep sdb		# doesn't show up here !
lsblk --fs /dev/sdb

sudo umount /dev/sdb
# Detach USB drive from system.
# Attach USB drive to system.

sudo chmod a+w /media/user1/LABEL

# Go to file explorer, see that new filesystem got mounted.
# Copy some files to it.
# In file explorer, unmount, remove drive, attach again, see it appear again.

Mount an existing VeraCrypt-encrypted volume to check device names:

# Attach USB drive to system.

# Run VeraCrypt GUI and mount volume.

# Check names:
lsblk --fs /dev/sdb
df -h | grep verac
# On my system, /dev/sdb to /dev/mapper/veracryptN mounted onto /media/veracryptN

# In VeraCrypt GUI, dismount volume.

# Detach USB drive from system.

VeraCrypt-encrypt a volume then add Btrfs:

# Attach USB drive to system.
# In file-explorer, if drive appears, ignore it.

# Run VeraCrypt GUI to create volume.
# Choose filesystem type "Linux Ext4" and "Quick format".
# Choose "I will mount the volume only on Linux".

# In VeraCrypt GUI, mount the volume.

# In file-explorer, in "Other Locations", drive should appear, click to unmount it.

# Check names:
df -h | grep verac		# volume does not appear

# Make Btrfs filesystem:
sudo mkfs.btrfs --force --label LABEL /dev/mapper/veracryptN

sudo btrfs check /dev/mapper/veracryptN

lsblk --fs /dev/mapper/veracryptN

# In VeraCrypt GUI, dismount the volume.

# Detach USB drive from system.

# Attach USB drive to system.
# In file-explorer, if drive appears (shouldn't), ignore it.

# In VeraCrypt GUI, mount the volume.

sudo chmod a+w /media/veracryptN

# In file-explorer, in "Other Locations", drive should appear.
# Copy files to it.

# Dismount in VC GUI, detach, attach, mount in VC GUI.
# Check files.

# Now mount/unmount can be done through VeraCrypt
# as usual; no need to do any special Btrfs commands.

One complication: it's best to mount a non-system Btrfs filesystem with the "noatime" flag specified, to avoid triggering COW on metadata when you read a file. In VeraCrypt GUI, specify that in Settings / Preferences / Mount Options / mount options. In VeraCrypt CLI, add "--fs-options=noatime". I would do this for all non-system volumes, regardless of filesystem type. Probably not a good idea to do it for a system volume, although you could do it to everything under your home directory via "chattr -R +A ~".

To see the flags after a volume is mounted:

mount | grep veracrypt5
# for Btrfs you probably want similar to: rw,noatime,space_cache,subvolid=5,subvol=/

I ended up doing that for all of my VeraCrypt volumes, regardless of filesystem type.

Make an encrypted ZFS filesystem (not using VeraCrypt)

I was going to try making a ZFS filesystem inside a VeraCrypt volume, but ZFS supports encryption natively, so no need to use VeraCrypt.

I'm using Ubuntu GNOME 20.04.

https://itsfoss.com/zfs-ubuntu/
https://wiki.ubuntu.com/Kernel/Reference/ZFS

I've read on /r/ZFS that ZFS is not intended for use with removable/USB drives. It's intended for enterprise large static configurations. It should work with USB, just USB inherently is less reliable.

From someone on /r/ZFS:
"When you mount the ZFS pool to the system, it mounts to a directory in your filesystem. It won't show as a separate volume."

Apparently installing zfs-fuse would remove zfsutils-linux. I'm told zfs-fuse would give lower performance.

Install software:

sudo apt install zfsutils-linux zfs-dkms
# Probably get a license dialog, have to press Return for OK.
# May have to reboot at this point.

man zfs
man zpool
zfs list

Important: In the following steps, change device names and labels as appropriate for your system. Best to have no extra removable or optional devices attached while doing operations, to avoid confusion.

zfs unencrypted first on a real device (no VC) to make sure it works:

# Attach a USB drive to system.

# Check device name:
sudo dmesg		# probably /dev/sdb
# If you look in file explorer and see that the device got mounted, ignore that.

sudo wipefs --force --all /dev/sdb
# If you look in file explorer and see that the device got mounted, ignore that.

zfs list
sudo zpool create MYPOOL /dev/sdb
sudo zpool status MYPOOL
zfs list
df -h | grep POOL
mount | grep zfs

# Make ZFS filesystem:
sudo zfs create MYPOOL/fs1
# If it says "ERROR: /dev/sdb is mounted", do "sudo umount /dev/sdb".
zfs list
df -h | grep POOL
mount | grep zfs
ls -ld /MYPOOL/fs1
lsblk --fs /dev/sdb
# Now filesystem is mounted and usable.
sudo chmod a+w /MYPOOL/fs1

cp /bin/ls /MYPOOL/fs1/aaa		# copy a file to it

sudo zpool scrub MYPOOL		# test data integrity
sudo zpool status -v MYPOOL	# if "scrub in progress", do again

sudo zpool export MYPOOL
sudo zpool status MYPOOL
# Detach USB drive from system.

# Attach USB drive to system.
sudo dmesg
sudo zpool import MYPOOL
zfs list
sudo zpool status MYPOOL
ls -l /MYPOOL/fs1

sudo zpool export MYPOOL
# Detach USB drive from system.

ZFS encrypted on a real device:

# https://www.medo64.com/2020/05/installing-encrypted-uefi-zfs-root-on-ubuntu-20-04/
# https://www.medo64.com/2020/04/installing-uefi-zfs-root-on-ubuntu-20-04/
# https://www.medo64.com/2020/06/testing-native-zfs-encryption-speed/
# https://blog.heckel.io/2017/01/08/zfs-encryption-openzfs-zfs-on-linux/
# https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-encrypted-zfs/

# Attach a USB drive to system.

# Check device name:
sudo dmesg		# probably /dev/sdb
# If you look in file explorer and see that the device got mounted, ignore that.

sudo wipefs --force --all /dev/sdb
# If you look in file explorer and see that the device got mounted, ignore that.

zfs list
sudo zpool create -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase -f MYPOOL /dev/sdb
# Passphrase minimum of 8 chars.

# Make ZFS filesystem:
sudo zfs create MYPOOL/fs1
# If it says "ERROR: /dev/sdb is mounted", do "sudo umount /dev/sdb".
ls -ld /MYPOOL/fs1
lsblk --fs /dev/sdb
# Now filesystem is mounted and usable.
sudo chmod a+w /MYPOOL/fs1

cp /bin/ls /MYPOOL/fs1/aaa		# copy a file to it

# Test data integrity:
sudo zpool scrub MYPOOL
sudo zpool status -v MYPOOL		# if "scrub in progress", repeat

sudo zpool export MYPOOL
# Detach USB drive from system.

# Attach USB drive to system.
sudo zpool import -l MYPOOL
ls -l /MYPOOL/fs1

sudo zpool export MYPOOL
# Detach USB drive from system.

ZFS using an encrypted VeraCrypt volume (FAILED):

# Attach USB drive to system.
# In file-explorer, if drive appears, ignore it.
sudo dmesg
sudo wipefs --force --all /dev/sdb

# Run VeraCrypt GUI to create volume.
# Choose filesystem type "Linux Ext4" and "Quick format".
# Choose "I will mount the volume only on Linux".

# In VeraCrypt GUI, mount the volume.

# Check names:
df -h | grep verac

# In file explorer, Other Locations, find device and unmount it.

sudo zpool create -f MYPOOL /dev/mapper/veracryptN
zfs list
sudo zpool status MYPOOL

# Make ZFS filesystem:
sudo zfs create MYPOOL/fs1
ls -ld /MYPOOL/fs1
lsblk --fs /dev/mapper/veracryptN
# Now filesystem is mounted and usable.
df -h | grep MYPOOL
sudo chmod a+w /MYPOOL/fs1

cp /bin/ls /MYPOOL/fs1/aaa		# copy a file to it

# Test data integrity:
sudo zpool scrub MYPOOL
sudo zpool status -v MYPOOL		# if "scrub in progress", repeat

sudo umount /MYPOOL/fs1
sudo zpool export MYPOOL
# In VeraCrypt GUI, dismount the volume.
# Detach USB drive from system.

# Attach USB drive to system.
# In VeraCrypt GUI, mount the volume by clicking "Mount",
# type password, click "Options", check "Filesystem - do not mount".
#sudo zpool create -f MYPOOL /dev/mapper/veracryptN
sudo zpool create -f MYPOOL
sudo zfs create MYPOOL/fs1
sudo mkdir /MYPOOL/fs1
# FAIL: can't figure out how to get filesystem defined in pool without creating it anew
lsblk --fs /dev/mapper/veracryptN
sudo zfs mount MYPOOL/fs1
sudo mount -t zfs /dev/mapper/veracryptN /MYPOOL/fs1

ls -l /MYPOOL/fs1

sudo zpool export MYPOOL
# In VeraCrypt GUI, dismount the volume.
# Detach USB drive from system.

LUKS encryption

LVM is not encryption. It's "Logical Volume Management", which is basically (very simplified) a smarter way to do partitioning, giving you much more flexibility later, e.g. to modify (grow/shrink) volumes, add new drives to the "pool" of storage, and things like that. LVM is not related to encryption, you can use LVM with or without encryption, and you can use encryption with or without LVM.

On a very technical nit-picky level, LUKS itself isn't encryption either, but a standard on how to manage encryption/keys/encrypted devices. The actual encryption itself is done by dm-crypt. Using dm-crypt without LUKS is a bit more of a hassle and not really common, so in practice "LUKS" is pretty much synonymous with "encryption" in the Linux world.

The most common setup, which is what most installation wizards offer, is as follows: you have two partitions on the drive, a small /boot partition and a large one for everything else. /boot is separate because it must not be encrypted, to be able to boot something that then decrypts the rest. The large partition is then encrypted using LUKS. Inside the LUKS container LVM is used to create whatever volumes ("partitions") are needed/desired. The nice thing about using LVM on LUKS is that you have full flexibility to do whatever you you like inside the encrypted part, and no matter what you do in there you can unlock it with a single password (unlike if you use several partition each with its own LUKS and no LVM).

# See all algorithms available in your kernel:
cryptsetup benchmark

# Dump parameters:
sudo cryptsetup luksDump /dev/sda6 >saved.luksDump.sda6.txt

# See what cipher algorithm is used on data:
sudo cryptsetup luksDump /dev/nvme0n1p3 | grep cipher

# See what key-derivation function is used (argon2id best):
sudo cryptsetup luksDump /dev/nvme0n1p3 | grep PBKDF

# Make backup of header:
sudo cryptsetup luksHeaderBackup /dev/sda6 --header-backup-file saved.luks_backup_sda6

# See if passphrase works:
printf "MYPASSPHRASE" | sudo cryptsetup luksOpen --test-passphrase /dev/sda6 && echo "Works"

# Find out device for encrypted partition:
lsblk

# Map encrypted partition to /dev/mapper/* and /dev/dm-* devices:
sudo cryptsetup open /dev/sda6 OLDROOT

# Mount partition:
sudo mkdir /mnt/OLDROOT
sudo chmod 777 /mnt/OLDROOT
sudo mount /dev/dm-9 /mnt/OLDROOT

# Now you can access the filesystem:
cd /mnt/OLDROOT
ls

# Get out:
cd /
sudo umount /mnt/OLDROOT
sudo cryptsetup close OLDROOT

# Shut down OS on USB, remove USB, boot from internal system.

lsblk
sudo cryptsetup open /dev/nvme0n1p3 OLDROOT
sudo mkdir /mnt/OLDROOT
sudo chmod 777 /mnt/OLDROOT
sudo mount /dev/mapper/ubuntu-root /mnt/OLDROOT
sudo chroot /mnt/OLDROOT

Creating a LUKS encrypted full-disk volume

# started with Disks app showing no partitions

sudo cryptsetup --type luks2 --iter-time 4100 --verify-passphrase luksFormat /dev/sdb
# I added the --iter-time setting; just wanted something not default.
# will ask twice for passphrase

# Make a backup of the LUKS header:
sudo cryptsetup luksHeaderBackup /dev/sdb --header-backup-file LUKS2.HeaderBackup.MYVOL1

# MYVOL1 is an arbitrary, temporary name.
# Device /dev/mapper/MYVOL1 will appear.
sudo cryptsetup luksOpen /dev/sdb MYVOL1
# will ask for passphrase

# Format the volume.
# Use --mixed if size less than 109 MB.
# VOLNAME is a permanent name for the filesystem.
sudo mkfs.btrfs --label VOLNAME --mixed /dev/mapper/MYVOL1

# If you forget the label (I did), later do:
sudo btrfs filesystem label /dev/mapper/MYVOL1 VOLNAME

sync

sudo cryptsetup luksClose MYVOL1

eject /dev/sdb		# probably get "unable to open"

# unplug the USB drive, and plug it in again

# File manager should detect the drive, see that it is LUKS,
# ask for passphrase, and mount it.

# https://unix.stackexchange.com/questions/319592/set-default-mount-options-for-usb
# Change mount options:
# Run Disks app.
# Select the device in the left pane.
# Select the filesystem (lower "Volume") in the main pane.
# Click on the "gears" button below.
# Select "Edit mount options". 
# Slide "User Session Defaults" to left to turn it off.
# Un-click "Mount at system startup".
# Add a name in the "" field, it shows up in fstab later.
# Edit mount options (maybe add "noatime").
# Click Save.
# Quit out of Disks app.

cat /etc/fstab		# to see mods made by Disks app
# note the UUID of the new disk

# In file manager, dismount filesystem.
# Unplug drive and plug it in again.
# Filesystem may get mounted automatically.

mount				# to see the new mount flags
sudo chmod 777 /mnt/THEUUID

Jordan Williams' "Encrypt an External Disk on Linux"

Creating a LUKS-encrypted container file

From Oleg Afonin's "Breaking LUKS Encryption":
"LUKS can be used to create and run encrypted containers in a manner similar to other crypto containers such as VeraCrypt."

Following article1 (seems best to me):

dd if=/dev/zero of=vol1.luks conv=fsync bs=1 count=0 seek=50M

sudo cryptsetup --type luks2 --iter-time 4100 --verify-passphrase luksFormat vol1.luks
# I added the --iter-time setting; just wanted something not default.
# Will ask twice for passphrase to set on volume.

# Make a backup of the LUKS header:
sudo cryptsetup luksHeaderBackup vol1.luks --header-backup-file LUKS2.HeaderBackup.MYVOL1

# MYVOL1 is an arbitrary, temporary name.
# Device /dev/mapper/MYVOL1 will appear.
sudo cryptsetup luksOpen vol1.luks MYVOL1
# will ask for passphrase

# Format the volume.
# Use --mixed if size less than 109 MB.
# VOLNAME is a permanent name for the filesystem.
sudo mkfs.btrfs --label VOLNAME --mixed /dev/mapper/MYVOL1

# If you forget the label (I did), later do:
sudo btrfs filesystem label /dev/mapper/MYVOL1 VOLNAME

# vol1 is an arbitrary, temporary mount-point name.
sudo mkdir /mnt/vol1
# I like to use noatime; maybe you don't.
sudo mount -o defaults,noatime /dev/mapper/MYVOL1 /mnt/vol1
sudo chown -R $USER /mnt/vol1

sudo umount /mnt/vol1
sudo cryptsetup luksClose MYVOL1

Following article2:

SIZE=500
FILE=xxx.luks
fallocate -l ${SIZE}M $FILE
dd if=/dev/urandom of=$FILE conv=fsync bs=1M count=$SIZE

# run Disks utility
# select "Attach Disk Image"
# un-check "Set up read-only loop device"
# select file xxx.luks
# click Attach
# Now the file is attached as if it were a hard drive.

# Select the new "drive", click the Gears icon,
# choose type Encrypted (LUKS + ext4), set passphrase etc.
# Click Format.

# Eject the drive in Disks or in file manager.

Use Disks application to Attach the container volume when you want to use it.

Or: associate file-extension ".luks" with "Disk Image Mounter" application, then you can double-click on any "*.luks" container file to mount it. BUT: it will mount read-only ! You have to REMOVE the association to "Disk Image Mounter" and create an association to:

'/usr/bin/gnome-disk-image-mounter' --writable

# Note: This is not available in KDE, can't find any equivalent.
# Maybe clevis-luks, clevis-luks-bind ?
# Maybe create a new Dolphin "service menu" ?
# I created a new "Service Menu" to do it: lukscontainerfile.

Or to mount and then un-mount:

sudo cryptsetup luksOpen xxx.luks xxx
sudo mount -o defaults,noatime /dev/mapper/xxx" /mnt/xxx

sudo umount /mnt/xxx
sudo cryptsetup close xxx

Or to mount and then un-mount:

udisksctl loop-setup -f xxx.luks
udisksctl unlock -b /dev/loop0
udisksctl mount -b /dev/dm-0

udisksctl unmount -b /dev/dm-0
udisksctl lock -b /dev/loop0
udisksctl loop-delete -b /dev/loop0

Change mount options to add noatime: Run Disks application.

Meer-Web's "Create an encrypted container in linux"
Luksman CLI utility

How secure is a LUKS* volume ?

My understanding is that there is no stored hash of the password (passphrase) in the LUKS header. So there is nothing to extract and then try to match with hashcat or similar. What is stored is an encrypted value V of the key K used to encrypt the actual data. So when the user opens the volume, the passphrase P and a salt S is used to transform V to produce K, and then K is used to decrypt the data. If P is wrong, you don't find out until you get the decrypted data and find out that it's not a valid filesystem.

From https://blog.elcomsoft.com/2020/08/breaking-luks-encryption/ :
"Unlike TrueCrypt/VeraCrypt, LUKS does store the information about the selected encryption settings in the encryption metadata, making it possible to detect the encryption settings prior to launching the attack."

Mike Fettis' "Cracking linux full disc encryption, luks with hashcat"
Forensic Focus's "Bruteforcing Linux Full Disk Encryption (LUKS) With Hashcat"
Diverto's "Cracking LUKS/dm-crypt passphrases"
milosz's "How to perform dictionary attack on LUKS passphrase"
Darell Tan's "Bruteforcing LUKS Volumes Explained"
glv2 / bruteforce-luks (uses cryptsetup API to be faster)
Terahash chart of random brute-force alphanum password-cracking times
Cryptsetup Wiki
Milan Broz's "LUKS2 On-Disk Format Specification 1.0.0"

Make a LUKS1 volume, and dictionary-attack it with hashcat:

sudo apt install hashcat	# or hashcat-nvidia
hashcat --version			# has to be 3.5.0 or better

# LUKS1 container file will be vol1.luks
dd if=/dev/zero of=vol1.luks conv=fsync bs=1 count=0 seek=50M

# Cracking will NOT work if you specify --iter-time
sudo cryptsetup --type luks1 --verify-passphrase luksFormat vol1.luks

# Cracking will NOT work unless you make a filesystem inside the container.
sudo cryptsetup luksOpen vol1.luks MYVOL1
sudo mkfs.ext4 /dev/mapper/MYVOL1
sudo cryptsetup luksClose MYVOL1

# LUKS1 container file is vol1.luks
# Dictionary is dict.txt; have container's password on one of the lines in it.
hashcat --force -m 14600 -a 0 -w 3 vol1.luks dict.txt -o luks_password.txt
# Keep typing "s" for "status" until done.
# See "Status ......: Cracked".
sudo cat luks_password.txt

Apparently hashcat 5.1.0 (and 6.2.4) does not support attacking LUKS2 volumes.

Apparently hashcat 6.2.4 on Manjaro requires use of a discrete GPU. Adding "-d 1" or "-D 1" to try to get it to use CPU didn't work. Installing "opencl-amd" (I have AMD CPU) got it going. But then it didn't crack the password, eventually screwed up the GUI, had to reboot.

Solid-State Drive (SSD)

Trim

Why is trim needed, and what is it ?

From Wikipedia:
"A trim command (known as TRIM in the ATA command set, and UNMAP in the SCSI command set) allows an operating system to inform a solid-state drive (SSD) which blocks of data are no longer considered to be 'in use' and therefore can be erased internally."

When a "delete" operation is done at the filesystem level, there is no corresponding "delete" at the device level; the device API has "read" and "write" operations, but not "delete".

A trim command notifies the SSD that some block is no longer being used.

The alternative to trim is "live discard", enabled by "discard" in the mount options. "Live discard" is becoming (kernel 6.2) the default for SSDs (article). "async=nodiscard" in the mount options would disable "live discard", and then you should be using TRIM.

Alan Formy-Duval's "Extend the life of your SSD drive with fstrim"
Justin Ellingwood's "How To Configure Periodic TRIM for SSD Storage on Linux Servers"
Chris Siebenmann's "SSDs and their support for explicitly discarding blocks"
Chris Siebenmann's "Using TRIM on SSDs with ZFS"

From reddit: "Does external SSD drive support TRIM ?"
"It depends on the usb-to-sata/nvme adapter. You need to make sure that the controller in the external SSD (or SSD enclosure) you want to buy supports trim passthrough. Do internet search for info about the model you want to buy."

Testing TRIM:
ICTBanking article

TRIM and LUKS

Do "lsblk --discard". If DISC-GRAN and DISC-MAX values for LUKS volume are non-zero, TRIM should work and no change is needed.

See which filesystems have been trimmed: "journalctl | grep trimmed".

To fix DISC-GRAN and DISC-MAX values:
From Jay Ta'ala's "Enable periodic TRIM - including on a LUKS partition":
In /boot/grub2/grub.cfg (in openSUSE, /etc/default/grub), add ":allow-discards" to the cryptdevice argument on the GRUB_CMDLINE_LINUX_DEFAULT, and add the kernel parameter "rd.luks.options=discard". Then rebuild GRUB menu.
"Kernel Parameters" section
[Another way is to create /etc/crypttab.initramfs ? Containing something like 'btrfsCRYPT PARTLABEL="LUKS" none timeout=0,discard' ? Then "mkinitcpio -P".]


"sudo dmsetup table /dev/mapper/luks-XXXXXXXXXXXX --showkeys" should show "allow_discards".

"mount | grep luks" does NOT need to show "discard"; that's for "live discard", not weekly TRIM.

"Live discard" is becoming (kernel 6.2) the default for SSDs (article). "async=nodiscard" in the mount options would disable "live discard", and then you should be using TRIM.

If using LUKS on SSD (probably only affects non-root volumes):
Add "discard" to end of line (4th field) in /etc/crypttab
Probably that field should contain "luks,discard".
See "For LUKS devices unlocked via /etc/crypttab" in Arch Wiki.

TRIM and Swap

Most distros run a TRIM via a systemd service about once/week.

To see what filesystems will be trimmed by fstrim service, "sudo fstrim -an".

From Fedora docs:
"The Linux swap code will issue TRIM commands to TRIM-enabled devices ..."
But I don't see the kernel code doing so, such as in swapfile.c

And from ArchWiki:
"If using an SSD with TRIM support, consider using discard in the swap line in fstab. If activating swap manually with swapon, using the -d/--discard parameter achieves the same."
Note: swap is not mounted as a filesystem, so mount's "discard" flag does not apply, and fstrim does mounted filesystems.

sudo apt install nvme-cli
man nvme

sudo nvme list
sudo nvme fw-log /dev/nvme0      # can store multiple versions
sudo nvme fw-log -o json /dev/nvme0
sudo nvme smart-log /dev/nvme0
sudo nvme error-log /dev/nvme0 | less

# See if an "NVMe fabric" is running ?
sudo nvme discover -t rdma -a IPADDRESS

# Get block size, but it may lie to you:
sudo blockdev --getbsz /dev/sdaN
sudo blockdev --getbsz /dev/nvme0n1
cat /sys/block/sda/queue/logical_block_size
cat /sys/block/sda/queue/physical_block_size
cat /sys/block/nvme0n1/queue/logical_block_size
cat /sys/block/nvme0n1/queue/physical_block_size

# See what sector sizes are supported:
sudo nvme id-ns -H /dev/nvme0n1 | grep 'LBA Format'
# You could reformat to the desired sector size:
#sudo nvme format /dev/nvme0n1 -l DESIREDLBAFID
# This destroys access to the existing filesystems on the device !
# Maybe run it in distro A, just before installing distro B.
# Or run it while booted from USB stick.
# Maybe do performance tests before and after.

sudo nvme id-ctrl /dev/nvme0 -H | less
sudo nvme show-regs /dev/nvme0 -H | less

# Trim command is known as TRIM in the ATA command set,
# and as UNMAP in the SCSI command set.

# See if SATA drive supports TRIM in ATA command set:
sudo hdparm -I /dev/sda | grep TRIM
# See if NVME drive supports Data Set Management (DSM), which should include Deallocate:
sudo nvme id-ctrl /dev/nvme0 -H | grep 'Data Set Management'
# USB external HDD: hdparm will work if USB interface identifies
# drive as UAS (USB-attached SCSI), but not if interface
# identifies drive as only "Mass storage device".
lsusb      # to find device's numbers
sudo lsusb -D /dev/bus/usb/BUSNUMBER/DEVICENUMBER | grep bInterface


# See if drive supports UNMAP in SCSI command set:
man sg_vpd		# in package sg3 or sg-utils
# See if supports TRIM command:
sudo sg_vpd -a /dev/nvme0n1p2 | grep 'Unmap'
# If returns nothing, then TRIM not supported ?

# Non-zero DISC-GRAN and DISC-MAX == TRIM support:
lsblk --discard

# Read paragraph in /etc/lvm/lvm.conf about
# parameter "issue_discards".  But
# https://wiki.archlinux.org/title/Solid_state_drive
# says no change needed.
# And want default value thin_pool_discards = "passdown".

# Trim supported at LUKS level ?
sudo dmsetup table
# See "allow_discards".

# Is OS doing TRIM via "discard" option in /etc/fstab,
# cron job, or systemctl fstrim.timer ?
mount | grep discard
sudo systemctl status fstrim.service fstrim.timer --full --lines 1000
sudo systemctl status btrfs-trim.service btrfs-trim.timer --full --lines 1000

# If using LUKS on SSD (maybe applies only to non-root device;
# root device will be handled in GRUB kernel command line):
# Add "discard" to end of line (4th field) in /etc/crypttab
# Probably that field should contain "luks,discard".
# See "For LUKS devices unlocked via /etc/crypttab" in
# https://wiki.archlinux.org/title/Dm-crypt/Specialties#Discard/TRIM_support_for_solid_state_drives_(SSD)
# Also
# http://blog.neutrino.es/2013/howto-properly-activate-trim-for-your-ssd-on-linux-fstrim-lvm-and-dmcrypt/

# Manually do a TRIM ?
sudo fstrim -a -v
# should take 30+ seconds and not return an error

# Periodic scrub ?
ls /etc/systemd/system/btrfs*
ls /usr/lib/systemd/system/btrfs*
# Edit the timer file to modify frequency:
EDIT /usr/lib/systemd/system/timers.target.wants/btrfs-scrub@.timer
# Then:
sudo cp /usr/lib/systemd/system/btrfs-scrub@.service /etc/systemd/system/
sudo cp /usr/lib/systemd/system/timers.target.wants/btrfs-scrub@.timer /etc/systemd/system/
sudo systemctl daemon-reload
systemd-analyze verify btrfs-scrub@.timer
# Have to put the mount-point path in the name when enabling.
# "-" is encoding for "/".
# Also could use "btrfs-scrub@$(systemd-escape -p MOUNTPOINT).timer"
sudo systemctl enable btrfs-scrub@-.timer
systemctl list-timers --all
# Run it right now to test it:
sudo systemctl start btrfs-scrub@-.timer
sudo systemctl status btrfs-scrub@-.timer --full --lines 1000
# But I don't see anything to indicate that the
# service ran or a scrub happened.
# Maybe have to wait until the timer fires, a week from now.
systemctl list-timers --all
journalctl -u btrfs-scrub@-.service

# Is the firmware updatable ?
sudo fwupdmgr get-devices
# Look for updates:
sudo fwupdmgr refresh

# Test sequential read performance:
sudo hdparm -t /dev/nvme0n1p2

Reducing writes to disk

Reducing writes really is not necessary for longevity; modern SSDs have 100 TBW (or multiples of that) rated lifetime, which would be 10 years while writing 30 GB per day.


On filesystem mounts, use "noatime" option, and maybe "lazytime" option.


You could have no swap, or use zram, instead of a swap partition or swap-file. You could change "swappiness" to 10 or 0. See "Swap" section of my Using Linux page.


[Most of following assumes you have plenty of RAM.]

Firefox and Thunderbird have settings to put cache in RAM instead of disk. Doing so would reduce use of SSD. Can't find similar in chromium browsers.

/tmp and /var/crash and ~/.cache (also /var/log, but I don't recommend that) can be mounted in RAM (tmpfs) instead of disk. Doing so would reduce use of SSD. I have seen cautions that you should not put /var/tmp on tmpfs. [Note: if you have swap, there are times when data from tmpfs can be swapped out.] [Note: "echo $TMPDIR" may show another place used for temp files.]

Do Timeshift backups to external disk, not to system disk (safer, too).

Don't have a swap partition or swap-file on SSD.

System journal can be kept in RAM, not disk, but then you lose past history. Edit /etc/systemd/journald.conf to have "Storage=volatile" in [Journal] section.

There is no standard way for a SSD to report its page size or erase block size. Few if any manufacturers report them in the datasheets. (Because they may change during the lifetime of a SKU, for example because of changing suppliers.)

...

Best leave about 5% to 10% of unallocated space outside any partitions. Having overprovisioned space is of great help to SSDs in maintaining their performance in time.

[This] will leave a number of blocks on the "never touched by the OS" list. This will simplify the life of the microcontroller which needs to juggle block allocations and erasures; remember that the poor little microcontroller has no notion of file systems, and from its point of view all the blocks touched by the operating system are in use, unless of course the OS is kind enough to trim unused blocks from time to time. This is especially important for external SSDs, which quite often may not even expose a trim interface.

... so the microcontroller has more space to move around stuff coming from within the partitions, without touching the interior of the partitions in the intermediate stages.

> I plan to have one partition for / and /home.
> I think I will always have 20% free and be
> doing weekly TRIMs. Should I just use all of the space
> visible to me as the single big partition ?

... just go ahead and use the entirety of space ... Modern SSDs have more space than advertised specifically for balancing, they do it on their own.

...

Always use LVM. Create the logical volumes as you need/want. But given that it is very easy to extend logical volume, create them with the size you need. Say, have a 30 GB logical volume for root filesystem, 1 GB volume for swap and 150 GB for home. You do not have to fill up all your SSD. If in a few months time you see that you need more space in /home, extend it.

There is a sanitize command (erases all blocks), and a secure-erase / format command (changes encryption key; two forms of it; you want "User Data Erase"). Use one of them. But you can't be sure what the command actually is doing. There have been cases where manufacturers lied about what their drives were doing. I'm told all SSDs are encrypted these days, so secure erase mainly wipes the key and sets a new one, making all data in chips useless, but still there.

"Sanitize is a newer SCSI command, intended as an extension to the existing ATA Secure Erase command. If the drive supports Sanitize, use that."

What is the relationship among the commands hdparm, sdparm, smartctl, nvme, and blkdiscard, especially when used with an SSD ? There seems to be some overlap. Some apply to all devices that support command set X, while others apply to all devices with interface type Y ? Just try to use each one, see if it says "unsupported device type" ?

tinyapps.org's "NVMe Sanitize"
tinyapps.org's "NVMe Secure Erase"
tinyapps.org's "ATA Sanitize and hdparm"
Mendhak's "Securely wipe an SSD with its built in commands"
LSU Grok's "Erasing SATA Drives by using the Linux hdparm Utility"
"hdparm --sanitize-status /dev/sdx"
"hdparm -I /dev/sdx"
"sudo nvme id-ctrl /dev/nvmex -H"
"man nvme-format"
"man blkdiscard" and maybe then trim.


You could try overwriting the whole device, but that's not guaranteed to get all the data in the chips. The problem is that the device is doing over-provisioning (reporting less capacity than it actually has) but it's using all of the actual space more or less evenly (wear-leveling). So even if you make the OS write the "full capacity" of the drive, it's missing some percentage of the blocks. And those missed blocks are spread all over the device.

Still, you could try. Boot from an OS on a different device (maybe USB stick), maybe change SSD partition table to have one full-disk partition, turn off any compression if applicable, and write random data to the raw device of the SSD until it is full. Do that 2 or 3 times. You're probably overwriting 99% of the data. But you're shortening the life of the device, too.

Chris Siebenmann's "When securely erasing disks, who are you trying to stop?"
Chris Siebenmann's "How secure is merely discarding (TRIMing) all of a SSD's blocks?"

Miscellaneous

Periodically check the health of your drives

Run SMART utility (Disks application, or smartctl).

If HDD:
"sudo apt install smartmontools" and then "sudo smartctl -a /dev/sda | less"
"sudo apt install libatasmart-bin" and then "sudo skdump /dev/sda | less"


If drive is an NVMe SSD:
"sudo apt install nvme-cli" and then "sudo nvme smart-log /dev/nvme0"
"sudo apt install smartmontools" and then "sudo smartctl -a /dev/nvme0n1p2 | less"

How to interpret important values:
"Percentage Used": percent used of estimated lifetime of the device.
(100 / "Percentage Used") * "Data Units Written (in TB)" == rated lifetime in TBW.
When "Available Spare" falls to "Available Spare Threshold", drive goes read-only ?
"Unsafe Shutdowns": lost power to device before data in cache was written to media.
"Media and Data Integrity Errors": number of unrecovered data integrity errors.
"Warning Composite Temperature Time": minutes run above warning temp.
"Critical Composite Temperature Time": minutes run above critical temp.
https://www.intel.com/content/www/us/en/support/articles/000056596/memory-and-storage/client-ssds.html
http://www.cropel.com/library/smart-attribute-list.aspx (HDD)
https://en.wikipedia.org/wiki/S.M.A.R.T.#Known_ATA_S.M.A.R.T._attributes

To get more info about SSD SMART errors:
"sudo nvme error-log /dev/nvme0n1p2"

How to interpret errors, especially cmdid values ? No "opcode" field ?
https://www.smartmontools.org/ticket/1300
https://github.com/linux-nvme/libnvme/tree/master/src/nvme
nvme_io_opcodes
https://metebalci.com/blog/a-quick-tour-of-nvm-express-nvme
https://cooboos.github.io/post/nvme-amdin-cmd/
https://www.smartmontools.org/static/doxygen/nvmecmds_8cpp_source.html start at line 294

If the drive is external (USB) and does not show up in SMART utility, maybe see SAT with UAS under Linux (especially the "Temporary Settings" section).

Thomas-Krenn's "SMART tests with smartctl"
Wikipedia's "S.M.A.R.T."
Oleg Afonin's "Predicting SSD Failures: Specific S.M.A.R.T. Values"
ZAR's "S.M.A.R.T. basics"
Chris Siebenmann's "SMART Threshold numbers turn out to not be useful for us in practice"
Chris Siebenmann's "Disk drives can have weird SMART values for their power on hours"
Chris Siebenmann's "NVMe disk drives and SMART attributes (and data)"
Chris Siebenmann's "The names of disk drive SMART attributes are kind of made up (sadly)"

I get an email every day from smartd daemon about "number of Error Log entries increased" on my SSD. Support for my laptop said "Those errors are not related to the SSD drive status and SMART status is OK, it's just telling you that there are two additional entries to the log." and pointed me to openmediavault thread

How to enable smartd:

ls /etc/systemd/system/smartd.*
less /etc/smartd.conf

sudo pamac install smartmontools
ls /usr/lib/systemd/system/smartd.*
sudo cp /usr/lib/systemd/system/smartd.service /etc/systemd/system/
less /etc/smartd.conf
sudo systemctl daemon-reload
sudo systemctl enable smartd
sudo systemctl start smartd

sudo systemctl status smartd --full --lines 1000
journalctl -u smartd

Disk test in BIOS or GRUB/EFI ?
See if your drive's vendor has a vendor-specific utility.

Drive Interfaces

Form-factors

• 2.5 Inch:
  
  Standard form-factor for hard disks in laptops.
  
  
• M.2:
  
  Smaller form-factor; based on the mSATA (Mini-SATA) standard. Too small to hold a HDD. Other peripherals such as Wi-Fi cards may also make use of M.2 connectors.
  
  From Josh Covington's "NVMe vs. M.2 vs. SATA - What's the Difference?":
  M.2 is just the form factor. M.2 drives can come in SATA versions and NVMe versions, which describes the bus they use to electrically communicate with the other PC components. SATA M.2 SSD drives and 2.5" SATA SSDs actually operate at virtually identical spec. NVMe M.2's on the other hand, definitely do not ...
  
  From Chris Siebenmann's "Getting NVMe and related terminology straight":
  The dominant consumer form factor and physical connector for NVMe SSDs is M.2, specifically what is called 'M.2 2280' (the 2280 tells you the physical size). If you say 'NVMe SSD' with no qualification, many people will assume you are talking about an M.2 2280 NVMe SSD, or at least an M.2 22xx NVMe SSD.
  
  From ExplainingComputers' "M.2 SSD Adapters & Enclosures" (video):
  M.2 is a physical connector.
  The interface through it may be USB, SATA, PCIe, more.
  M.2 SSDs have either a SATA or PCIe interface.
  PCIe M.2 SSDs use a data transfer protocol called NVMe.
  
  
  
  

Electrical interfaces and protocols

• AHCI SATA:
  
  Communicates through SATA controller before getting to CPU.
  
  Limited to 1 command queue and 32 commands per queue.
  
  From Chris Siebenmann's "Getting NVMe and related terminology straight":
  Traditional SATA SSDs are, well, SATA SSDs, in the usual 2.5" form factor and with the usual SATA edge connectors (which are the same for 2.5" and 3.5" drives). If you simply say 'SSD' today, most people will probably assume that you mean a SATA SSD, not a NVMe SSD. Certainly I will. If I want to be precise I should use 'SATA SSD', though. SATA comes in various speeds but today everyone will assume 6 Gbits/s SATA (SATA 3.x).
  
  
  
• NVMe (Non-Volatile Memory Express):
  
  Can communicate directly to CPU.
  
  Up to 64K command queues and up to 64K commands per queue.
  
  From Chris Siebenmann's "Getting NVMe and related terminology straight":
  NVMe, also known as NVM Express, is the general standard for accessing non-volatile storage over PCIe (aka PCI Express). NVMe doesn't specify any particular drive form factor or way of physically connecting drives, but it does mean PCIe ...
  
  From Josh Covington's "NVMe vs. M.2 vs. SATA - What's the Difference?":
  NVMe ... developed to allow modern SSDs to operate at the read/write speeds their flash memory is capable of. Essentially, it allows flash memory to operate as an SSD directly through the PCIe interface rather than going through SATA and being limited by the slower SATA speeds.
  
  ...
  
  NVMe drives provide write speeds as high as 3500 MB/s [PCI Express Gen 3 bandwidth]. That's 7x over SATA 3 SSDs and as much as 35x over spinning HDDs!
  
  Every NVMe drive has an M.2 form-factor.
  
  Faster NVMe chips and drives coming in 2021 or later will support PCI Express Gen 4 bandwidth, maybe 5000 to 7000 MB/s.
  
  Figure out how many PCIe lanes the SSD (controller) is using:

  
  sudo lspci -vv | grep 'Non-Volatile'                   # get device number
  sudo lspci -vv -s NN:NN.N                              # see all info for device
  sudo lspci -vv -s NN:NN.N | grep -E 'LnkCap:|LnkSta:'  # see lane info
  # you could search for datasheet for NVMe controller chip to see total lanes
  

  
  Wikipedia's "NVM Express"
  
  
  

Christopher Harper's "NVMe vs M.2 vs SATA: Which is the best for your SSD?"
Anthony Spence's "NVMe vs SATA vs M.2 : What's the difference when it comes to SSDs?"