16:49:08 <cmurf> #startmeeting Workstation WG (2020-06-16)
16:49:08 <zodbot> Meeting started Thu Jun 18 16:49:08 2020 UTC.
16:49:08 <zodbot> This meeting is logged and archived in a public location.
16:49:08 <zodbot> The chair is cmurf. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:49:08 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:49:08 <zodbot> The meeting name has been set to 'workstation_wg_(2020-06-16)'
16:49:11 <cmurf> #meetingname workstation
16:49:11 <zodbot> The meeting name has been set to 'workstation'
16:49:13 <cmurf> #chair cmurf
16:49:13 <zodbot> Current chairs: cmurf
16:49:21 <cmurf> #topic Rollcall and introductions
16:49:22 <cmurf> #info present: cmurf, aday, kalev, bcotton, felipe, otaylor, igor, james, mclasen, neal, robert handlin, tpopela, mcatanzaro, langdon
16:49:25 <cmurf> Chris welcomes our guests: Michel Salim, Davide Cavalca, Josef Bacik, Alex Gartrell
16:49:27 <cmurf> #topic Discussion of Facebook's usage of btrfs
16:49:29 <cmurf> #link Agenda: https://docs.google.com/document/d/1u5it1dXxem-DJ8gN9jn9z9y7g9Uliirac1oUWH3CqPY/edit
16:49:31 <cmurf> Owen: clarifies the scope for the call: implications of btrfs by default for Fedora workstation, rather than btrfs for other use cases. Focus on developer laptops.
16:49:33 <cmurf> Michel: btrfs experience is mostly in production rather than in laptops. For laptops they have use cases where snapshots and rollbacks would be useful. Not partitioning into root and home would really help - hard to predict space usage.
16:49:35 <cmurf> Davide: gives an overview of btrfs usage in production. "It mostly works to be honest." Doesn't remember issues with btrfs. Main advantages: transparent compression (reduces space usage), snapshotting (used for containers).
16:49:37 <cmurf> #info snapshotting and rollbacks could be useful for updates on workstation.
16:49:39 <cmurf> (said not so useful in production because it interacts badly with configuration management.)
16:49:41 <cmurf> Ability to expand storage online - attach volumes, devices, and make use of the space.
16:49:43 <cmurf> Thinks that these are compelling features from a user perspective.
16:49:45 <cmurf> #info resource control via cgroupsv2 is valuable - btrfs provides IO isolation
16:49:47 <cmurf> Can specify baseline i/o for a particular process. ext4 has resource inversion issues such that I/O limitation does not work well. (Michael: note I/O limits are a goal for F33/F34.)
16:49:49 <cmurf> Alex: a lot of people at Facebook are managing their own Fedora installs. It's useful to use btrfs for the build process; it's "overlay-ish". [Implication is that btrfs is useful for container development.] Can be used for throwaway space.
16:49:51 <cmurf> Owen: it's hard to translate from the Facebook custom container setup. Does Facebook distribute containers as btrfs volumes?
16:49:53 <cmurf> Alex: yes... possibly as a send stream.[chris: this is a btrfs feature to create a stream of metadata+data from a subvolume snapshot. 'man btrfs send']
16:49:55 <cmurf> Owen: do you see errors that are being collected at the device level, and do you have statistics for that?
16:49:57 <cmurf> Josef: haven't had issues that are directly btrfs's fault. Issues have been due to hardware. Disks are awful; btrfs has helped to discover issues. (Switching to btrfs revealed a problem where a RAID controller would write random stuff into the middle of disk - the system previously with ext4 mysterious corruption.)
16:49:59 <cmurf> Compression reduces write-amplification issues; allows extending the lifetime of SSDs.
16:50:01 <cmurf> #info compression extends lifetime of SSDs
16:50:03 <cmurf> Alex: they haven't had data corruption issues. Says that btrfs has been reliable, despite it being stress tested.
16:50:05 <cmurf> Owen: reviews some of the user experience features that have been touched on so far: compression, subvolumes to avoid space issues with partitions
16:50:07 <cmurf> Davide: snapshotting could be used for backups
16:50:09 <cmurf> Davide: btrfs will expose more hardware errors which aren't picked up by other filesystems. That could make people more aware of those issues ("hardware wonkiness"). Thinks that this is a positive thing.
16:50:11 <cmurf> #info most issues aren't btrfs itself, it discovers problems resulting from hardware
16:50:13 <cmurf> Josef: has written tools to recover data. btrfs is vulnerable to crappy hardware. Until recently these were userspace tools to copy data out of unmounted filesystems, but very recently the kernel has gained the ability to mount filesystems in "recovery mode". However, it still isn't a pleasant experience. ext4 will just skip/return the bad blocks.
16:50:15 <cmurf> Neal: is there are recovery tooling that will be available to regular users?
16:50:17 <cmurf> Josef: that's the plan, but there could always be kinds of corruption that they haven't seen before. Talks about potentially enabling a "do not care" mode. The distro needs to be able to boot while the filesystem isn't mounted read/write, particularly if the filesystem fails early in the boot process.
16:50:19 <cmurf> Langdon: asks about the timeline for this work.
16:50:21 <cmurf> Josef: the work is already done, but the OS needs to cooperate - requires integration.
16:50:23 <cmurf> Dusty: SUSE already uses btrfs. Do we have any lessons to learn there?
16:50:25 <cmurf> Josef: their story is similar to Fedora, and will want this degraded behavior
16:50:27 <cmurf> Zbigniew: from the systemd side, there is a PR to flip mounts for read-only block devices to read only. "It almost boots correctly." They hope to be able to have machine booting in readonly mode.
16:50:29 <cmurf> #info near term "do not care" mode to make recovery and degraded operations easier
16:50:31 <cmurf> Owen: any other advantages of btrfs?
16:50:33 <cmurf> Matthias: what about desktop integration? What would need to be exposed in the UI? File manager, etc. Does anything exist?
16:50:35 <cmurf> Davide: compression is transparent.
16:50:37 <cmurf> Neal: there's a gui for snapper, which allows taking snapshots. There's a tool called timeshift. None of them are shipped in Fedora, but they do exist.
16:50:39 <cmurf> Owen: it's clear that we'd need to do integration work if we wanted to use snapshots. That would apply to any feature we wanted to expose. What's been done to make the cli make sense with subvolumes? Intelligible results from `df` aren't new, but there is a question regarding integration with the standard command line tools.
16:50:41 <cmurf> Josef: there's the btrfs command, which follows the cli conventions, and has documentation. See for example `btrfs filesystem usage`, which gives a nice presentation of where data is. There's been work into cli usability.
16:50:43 <cmurf> Igor: There's a DNF plugin for snapper to automatically take Btrfs snapshots.
16:50:45 <cmurf> Josef: `df` and `du` aren't going to lie, but they don't know about compression. Dusty: the version that lies is just fine in his experience.
16:50:47 <cmurf> Davide: subvolumes show up as directories - so should work with standard tooling. They won't appear as subvolumes though.
16:50:49 <cmurf> #info 'df' 'du' mostly report correctly but don't know about compression, subvolumes behave mostly like directories
16:50:51 <cmurf> Owen: encryption. Are there best practices for encryption+btrfs?
16:50:53 <cmurf> Josef: dmcrypt is the answer and works just fine. Someone is working on built-in subvolume encryption.
16:50:55 <cmurf> Neal: one of our problems is how to convert from unencrypted to encrypted
16:50:57 <cmurf> Josef: it will use the same paths as compression - it rewrites. The same will work for encryption. They've done this is production - converting machines to compression on the fly without redoing everything.
16:50:59 <cmurf> #info btrfs works fine with dm-crypt, native encryption is being worked on and live conversion is an expected feature
16:51:01 <cmurf> Zbigniew: main worry for the workstation case is btrfs not being tolerant of flaky hardware.
16:51:03 <cmurf> Josef: the worry is crappy hard drives. Read only mode + recovery options will be perfect for that.
16:51:05 <cmurf> Alex: we should be careful with phrases like "pushing hardware harder"  - their experience hasn't been that hardware hasn't lasted longer or failed more. They're using consumer grade drives.
16:51:07 <cmurf> Neal: asks for people's experiences using btrfs on their desktops
16:51:09 <cmurf> Davide: has been running btrfs on his laptop for ~3 years. It just works. Compression is nice. Snapshotting is useful - for example, can systemd-nspawn into a mutable snapshot of / to try things out.
16:51:11 <cmurf> Langdon: if you were the workstation, would you recommend switching to btrfs for F33?
16:51:13 <cmurf> Josef: describes himself as conservative and says that it would be fine.
16:51:15 <cmurf> Alex: if you can land it with the killer features (compression, better ssd behavior) - "Fedora makes SSDs last longer". That's a killer feature.
16:51:17 <cmurf> Neal: Commentor points out more advanced file systems (APFS, NTFS, ZFS) are starting to become the norm for workstation OSes and btrfs seems like a very good option for Fedora.  Is probably triggered by bad SSDs.
16:51:19 <cmurf> Langdon: buying smaller (read: cheaper) ssds is also a pretty good feature
16:51:21 <cmurf> Davide: even if you don't expose the features in the UI, it would help people to make those features discoverable, you will help kickstart the features and integration, if it's not there by default the UI integration will not happen
16:51:23 <cmurf> Michel: Facebook will begin updating their imaging for laptops/desktops to use Btrfs in the fall
16:51:25 <cmurf> #info desktop experience, it just works, has a number of advantages, solves problems
16:51:27 <cmurf> #info Facebook can recommend btrfs by default, will be updating laptops/desktops this fall to use it
16:51:28 <cmurf> #endmeeting