15:00:12 #startmeeting Fedora Base Design Working Group (2014-08-22) 15:00:12 Meeting started Fri Aug 22 15:00:12 2014 UTC. The chair is pknirsch. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:12 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:17 <- 15:00:21 #meetingname Fedora Base Design Working Group 15:00:21 The meeting name has been set to 'fedora_base_design_working_group' 15:00:27 heya haraldh :) 15:00:32 hey 15:00:47 hey! 15:00:51 hah! sent two kernel patches today :) 15:00:53 #chair haraldh vpavlin msekleta 15:00:53 Current chairs: haraldh msekleta pknirsch vpavlin 15:01:00 #chair dgilmore masta 15:01:00 Current chairs: dgilmore haraldh masta msekleta pknirsch vpavlin 15:01:04 hey! 15:01:08 heya everyone :) 15:01:25 * vpavlin just needs to lock the bike;) 15:01:33 heh :) 15:01:50 haraldh, which subsystem? 15:02:01 efi 15:02:05 nice haraldh ! 15:02:08 and early microcode 15:02:14 hello 15:02:14 that's a hell of a thing to say 15:02:15 * jreznik_q10 is here 15:02:23 #chair jreznik_q10 15:02:23 Current chairs: dgilmore haraldh jreznik_q10 masta msekleta pknirsch vpavlin 15:02:49 pjones, well yeah... 5h of debugging, why the initramfs is not uncompressing 15:02:51 ok, lets get started then. vpavlin asked me to pull the later 2 topics earlier as he was traveling 15:03:00 ugh 15:03:03 :/ 15:03:13 #topic Subpackage config file discussion 15:03:29 * masta notes pjones appears when one mentions efi 15:03:37 * pjones thought everybody knew that :P 15:03:41 :) 15:03:44 alright, so first up the subpackage discussion for config file 15:04:15 vpavlin i think wanted to send out an email about that. we discussed that in the hallway at Flock 1 1/2 weeks ago 15:04:32 have you seen what systemd now provides? a way to populate /etc and users with files from /usr 15:04:54 ah 15:04:59 yeah, sorry I wasn't able to do that yet..damn you Docker! 15:05:04 heh :) 15:05:15 so, yeah 15:05:16 * pknirsch is running RHEL-7 docker images on RHEL-6 these days ;) 15:05:29 subpackage the /etc files and additionally add the systemd things to the main package 15:05:39 hey all 15:05:45 you have my blessing for that 15:05:58 haraldh, would that need additional changes to specfiles other than the subpackages? 15:06:03 * vpavlin nods 15:06:15 yes 15:06:19 hm 15:06:27 but if you touch them, better only touch them once 15:06:32 in one go 15:06:33 true dat 15:07:07 haraldh: wassup? 15:07:12 have you and vpavlin talked about the details? so when we send out that email to f-d people will understand :) 15:07:12 pknirsch, I invited poettering :) 15:07:18 ah, cool, thanks poettering ! 15:07:23 and thanks haraldh 15:07:26 we talked about this with haraldh on flock that we should do it as one thing and I agree 15:07:47 poettering, the new /etc and user populating mechanism of systemd 15:07:57 poettering, want to give a short summary? 15:08:05 haraldh let's talk about it next week and I'll write it up and send 15:08:08 poettering, maybe also about the rpm macros you added? 15:08:23 doesn't hurt, if they are mentioned here 15:08:58 dont forget rpm macros and changes here will need fpc arrpoval and packaging guideline changes 15:09:01 poettering, we want to subpackage the /etc files in "*-config" subpackages 15:09:06 approval 15:09:07 * pknirsch nods to dgilmore 15:09:18 hmm? "the /etc files"? 15:09:37 yeah, config files of packages 15:09:46 dgilmore: tehre's already an fpc ticket for the rpm macros for sysuers, if those are what you are talking about? 15:10:02 poettering, if we touch those packages, we might also want to add your macros in one go 15:10:05 haraldh: you want to split *all* rpms in two? 15:10:16 haraldh: which macros precisely are you talking of? 15:10:21 haraldh: the sysusers macros? 15:10:24 poettering, yes 15:10:27 poettering: thats part of it. if we want to put config files in separate packages like haraldh is talking about that will need changes as well 15:10:31 i am missing some context here apparently 15:10:39 poettering, and the factory mechanism 15:10:39 ENOCONTEXT 15:10:45 :) 15:10:48 poettering: I thing all of us except for haraldh are 15:11:00 hmmm 15:11:03 Maybe haraldh should do the summary then ;) 15:11:12 i am not actually arguing for splitting up all RPMs in two, i think they should continue to ship the config 15:11:14 vpavlin, you go first :) 15:11:16 poettering we would like to move config files stored in /etc to subpackages 15:11:29 but we should also ship them in /usr/share/factory... 15:11:44 in the main package, if needed/wanted 15:11:48 so that users can simply installed their *-config package to replace co figuration 15:12:02 or package their own 15:12:08 hmm, so suddenly i am the conservative here? ;-) 15:12:13 hahahah 15:12:14 or remove them completly 15:12:27 and let s.th. else than rpm handle the config 15:12:28 as long as the configs are in /usr they can stay in main pkg 15:12:42 vpavlin, right.. /usr/share/factory.. 15:12:56 or as with systemd, are overridable with /etc 15:12:57 does this really make sense? i mean, if people want to install without config, then maybe that should become an rpm switch, after all rpm knows already exactly what is config and what isn't due to %config 15:13:12 good point 15:13:19 like with %docs you mean? 15:13:20 anyway, let me maybe summarize what i had in mind 15:13:22 aka --nodocs? 15:13:34 so, i want stateless systems 15:13:39 so that we can boot up without /etc around 15:13:46 and populate automatically what is necessary 15:13:51 but only what is strictly necessary 15:13:54 they want to install configs, but they have to override them by one shot scripts which is error prone 15:14:03 we haven't figured out in all details how this all will map to fedora 15:14:11 but what we do now is this: 15:14:28 we want all RPMs to ship the pristine vendor configuration in /usr/share/factor/etc/* 15:14:40 that we can easily diff vendor defaults with user changes 15:14:51 simply by diffing /etc/ against /usr/share/factory/etc/ 15:14:58 poettering: that has some value 15:15:13 but the main goal we have here is the "golden container" setup 15:15:18 we want /usr to be fully self sufficient 15:15:19 * vpavlin agrees that having config in /usr would be awesome 15:15:21 do replicate systems 15:15:34 so, basically, if you want to "instantiate" an OS 15:15:57 you just mount /usr as pristine vendor directory to some directory name /usr 15:16:01 and then you can boot that up 15:16:05 and /var and 7etc are added in 15:16:12 now think about this, if you do this for 100x containers 15:16:19 from the same /usr tree 15:16:34 so, with systemd upstream this actually works pretty fine now 15:16:39 at least with the basic sets of tools 15:16:49 however, there are a couple of packages that currently are allergic to missing /etc 15:16:54 atcually quite a few 15:17:05 from the core OS that's most importantly dbus1 and pam 15:17:10 how many? approximately 15:17:11 dbus1 chokes on missing configuration 15:17:31 and dbus1 is something we probably could move easily, the devs are friendly to our ideas 15:17:43 but so far we didn't prep patches for that 15:17:58 also becuase we mostly use kdbus now ;-) 15:18:03 PAM is the bigger problem 15:18:18 for PAM we prepped patches 15:18:26 they are hanging since months in the PAM trac 15:18:40 but i figure the pam maintainers are not so friendly to our ideas 15:18:49 anyway, those are the two biggies from the basic OS 15:18:56 from my point of view, we can roll out the thing in two steps: first move config to subpackages and add the factory to the main package 15:19:06 so you suggest to rather invest in moving things from /etc to /usr/share/factory and toss the subpackage thing, right 15:19:07 we are fully aware that we cannot patch all software we ship to be compatible with empty /etc 15:19:10 then in the far future we just drop the subpackages 15:19:14 wait i am not done yet 15:19:15 please 15:19:16 wait 15:19:30 so, acknowledging the fact that a lot of software doesn't like /etc empty 15:19:39 we beefed up tmpfiles a bit 15:19:57 so that it is able to copy whole trees from /usr/share/factory to /etc 15:20:07 you basically just add a line "C /etc/pam.d" 15:20:25 and that means that /usr/share/factory/etc/pam.d" is copied → "/etc/pam.d" if it doesn't exist there yet 15:20:49 it's our strategy to work-around upstream maitnainers who think empty /etc is stupid 15:20:56 of coruse, we really see that as last resort only 15:21:06 we really want softwrae to work without /etc by default 15:21:08 that also means that once you do an install, you'll never thereafter get any changes to the pam config dir that are made in the upstream defaults, right? 15:21:24 ok, so it sounds like rpm be made to default config install to the factory location path, and some sub-routine near the end copy them into /etc (possibly as rpmnew) 15:21:25 but if we cannot have that, then we can just add a tmpfiles line to work around that, and copy if missing 15:21:45 so, regarding updates 15:21:50 it's actually a difficult problem 15:22:01 3-way merge :) 15:22:02 the "golden container" image thing i mentioned earlier is only half useful 15:22:06 if we cannot actually update them 15:22:19 so, we needed a strategy to facility "offline" updates 15:22:28 where we can basically update a /usr tree with RPM somewhere 15:22:39 and then, this gets replicated into the 100x instances of it 15:22:58 for example, stuff like /etc/ld.cache and things need to be rebuild 15:23:03 if /usr changes 15:23:11 so, we added a very simple scheme for that 15:23:21 howe certain tools can be run on a reboot 15:23:31 if we figure out that /usr has changed since the last time we booted up 15:23:51 the idea is then that we can update /usr at any time, in a snapshot kind of way 15:24:02 then we simply restart the container instances with the new /usr version 15:24:07 * pknirsch nods 15:24:20 and we run a handfull of additional services that will replicate changes to /etc and /var that we really have to do 15:24:38 of course, we updated systemd itself to make use of that 15:24:49 for example, we now ship an ldconfig service that rebuilds the cache 15:24:57 and sysusers and stuff is run like this 15:25:00 but for the scenario of config files, are you thinking that say gdm would ship a systemd service that would compare its new config file versus the old and copy over manually if the config file hasn't been modified? 15:25:21 so that if a package wants a new user to be added, it just adds the sysuser file, and on next reboot of the instances, we get the new user added to /etc in early boot 15:25:25 shouldn't that be done by a general framework, walters ? 15:25:30 this all is implemented using normal services btw 15:25:43 pknirsch, it's how ostree works 15:25:47 but there's a new ConditionUpdateNeeded= that conditionalizes these services 15:26:13 i don't think packagers would like the idea to modify all their packages to check for modified config files, something rpm has "taken care of" for them in the past 15:26:20 walters: well, i really hope we never have to patch config files 15:26:23 (arguably badly, but at least it did in some form) 15:26:44 walters: but yeah, if we really really have to patch some, then there would have to be a service that is conditionalized on upgrades and does that 15:26:47 poettering, from this time forward, /etc/pam.d/cockpit should never change? 15:27:03 walters: but in general, there shouldn't be a gdm config file anyway in /etc, right? 15:27:19 walters: so if an admin adds one, then that coutns and we shouldn't fuck with it on upgrades 15:27:41 walters: well, /etc/pam.d/cockpit should be /usr/lib/pam.d/cockpit or so 15:27:41 but you were suggesting the system copy /etc/pam.d, that's not the admin's doing 15:27:50 and if people want to edit it, they copy it 15:27:54 and edit it, and we won't fuck with it 15:28:06 but the entire idea here is that /etc is minimal 15:28:12 poettering: how could we then still notify the admin that the default config file changed though? so that he has a chance to actually check if he needs to do something about it? (see rpmsave resp rpmnew files previously) 15:28:26 and vendor config is either built into programs, or lies around in /usr/share and /usr/lib 15:28:43 and that /etc is really primarilly admin territory 15:28:49 and we try to stay away from it 15:28:55 and if the admin drops-in a file there 15:28:58 that's the file that counts 15:29:04 and we will not patch it 15:29:08 or fuck it up otherwise 15:29:16 pknirsch: well, that's what we have the factory stuff for 15:29:26 pknirsch: we want a powerful diff tool 15:29:30 ahh 15:29:31 right 15:29:39 that can inform users about differences between vendor config in factory 15:29:43 and what he actually is running 15:29:44 * pknirsch nods 15:29:51 systemd-delta in a way is already that 15:29:54 that would be awesome actually 15:29:55 (as ostree can do today) 15:30:01 but doesn't actually understand /usr/share/factory 15:30:03 much better than the rpmsave crap we have 15:30:26 but i figure we'll update systemd-delta accordingly to also diff all of /etc to /usr/share/factory 15:30:35 * pknirsch nods 15:30:40 so, anyway, it's a lot of pieces of a puzzle to get right 15:30:43 yea 15:30:51 but i think in the end we have very clear definitions of what is in /etc 15:30:54 and what is in /usr 15:31:06 mhm 15:31:09 we have nice tools for admins to figure out what chanegs they made 15:31:17 we can replicate systems nicely 15:31:23 so, what would you suggest tobdo from Fedora pov now, poettering? 15:31:25 and services or apps that use subdirs for configuration would be even better of. 15:31:28 we can "offline" update instances nicely, and so on 15:31:41 * pknirsch thinks about unmodified httpd.conf and http/conf subdirs 15:31:54 first thing i would want from fedora 15:32:09 is that RPM implicitly copies all files marked %config into /usr/share/etc 15:32:18 that would already bring us a big step ahead 15:32:34 I asked ffesti to do that.. 15:32:34 also, we should get started making the most basic packages compatible with /etc being totally empty 15:32:37 notable PAM 15:32:44 * vpavlin nods 15:32:47 mhm 15:32:48 oh, and one clarification 15:33:01 but I don't think we will get it in the next year... except pknirsch makes some pressure :) 15:33:02 things like apache of course will not work without /etc containing their configs 15:33:21 but i think that that is actually totally fine, since apache is one of those things that are not useful without admin config anyway 15:33:33 so, we really should focus on packages that are supposed to "just work" 15:33:38 right 15:33:39 and that nobody installs explicitly 15:33:52 yeah, agreed with that 15:34:00 there are a couple of non-obvious things that need to be made compatible with this scheme too 15:34:04 like for example /etc/services 15:34:06 how do you handle /etc/passwd and /etc/shadow ? 15:34:09 (note intersection with the issue of wheteher or not services start by default) 15:34:10 Will require changes to SELinux. but it is something we have wanted to do for a while. 15:34:24 /etc/services should move to /usr/share/services or so 15:34:41 and glibc should first use /etc/services, and on ENOENT fall back to /usr/share/services 15:34:49 so that admins can make changes 15:34:55 even though probably nobody does anymore 15:35:07 i mean, that's the good thing about /etc/services, nobody uses that anymore... 15:35:11 yeah, we should git grep through glibc and add fallbacks from /usr 15:35:13 hence we can ignore it for a while 15:35:20 but for full funcitonality we do need to cover that 15:35:26 this means changes to glibc 15:35:26 /etc/resolv.conf? 15:35:48 poettering: it quite possibly means changes to a lot of packages 15:36:18 dgilmore, one way or the other... but better progress :) 15:36:58 dgilmore: yupp! 15:37:00 yeah, I expect this to have a good bit of churn 15:37:09 dgilmore: but then again, tmpfiles is a way out for the hard cases 15:37:20 however proposal for splitting config files to separate subpackages was in a way about something different, if I am not mistaken 15:37:29 dgilmore: tmpfiles is basically our way how we can make this transition more slowly 15:37:44 some configs are expected to never be edited 15:37:44 msekleta you wouldn't need that if all goes well 15:37:44 dwalsh: /etc/resolv.conf should be a symlink to some file in /run 15:37:52 dwalsh: it's awful that NM touches /etc all the time 15:37:56 perhaps we should have somewhere to put them 15:38:09 dwalsh: /etc really should be considered read-only for services, unless they are configuration management daemons, which NM is not 15:38:28 vpavlin, so will not pursue this further and will focus on getting lennart's idea to work then? 15:38:45 ah, here regarding RPM now 15:38:51 poettering, I agree 100% 15:39:08 i think actually, that most RPMs should continue to install their configuraation to /etc, since most admins expect that 15:39:11 poettering: making /etc/resolv.conf breaks many many things and is going to have to be handled carefully 15:39:15 where should NM write state instead? /var ? 15:39:31 however, i think we should work on fixing these packages to not actually require them during runtime 15:39:50 and the config files shoudl alaways end up in both /etc *and* in /usr/share/factory/etc 15:40:06 and it might be nice to have an RPM mode, where it doesn't bother with dropping them in /etc 15:40:13 and only puts them in /usr/share/factory/etc 15:40:21 walters: /run 15:40:24 Random tools/services writing random crap into /etc has been a bane to SELinux existance. 15:40:26 walters: networkd already does that 15:40:37 poettering do you thing it's possible to get it to F22? Of not, I'd still pursue subpackages.. 15:40:38 poettering, er...i want to be able to have my wireless network choices be persistent 15:40:38 walters: /run/systemd/resolved/resolve.conf 15:41:00 walters: I agree there 15:41:15 walters: ok, you are talking about those. they should probably be in /var for system-defined ones, and in $HOME for user-defined ones 15:41:21 walters: and vpn connections 15:41:39 poettering, you do realize NM started out that way, and went through a *really* painful transititon to system connections 15:41:46 walters: but honestly, the fact that NM currently stores wlan passwords unencrypted in /etc is really awful 15:42:14 well 15:42:14 poettering: you do realise that NM is running in a legacy compat mode 15:42:22 and was designed to do things differently 15:42:40 it turns out a lot of people want the ability to have their network connect before they've logged in 15:42:41 walters: well, i am totally fine if NM stores wlans in /etc, if it wants to. what i am not fine about is that it keeps touching /etc/resolv.conf 15:42:53 yeah no one disagrees about resolv.conf 15:42:54 walters: i mean, in a way if you connect to a *new* wlan, this is kinda configuration 15:42:59 walters: hence writing to /etc might be fine 15:43:04 right 15:43:16 walters: but when i reconnect to known networks, that's not configuration changes, that's just reconnecting 15:43:26 and reconnecting should cause /etc to be modified 15:43:37 * vpavlin will have to go in 15mins :( 15:43:49 anyway, that's kinda what i have in mind 15:43:58 so, what i'd really like to see is: 15:44:09 a) rpm populates /usr/share/factory by default 15:44:17 b) rpm gets a new switch --noconfig or so 15:44:40 c) we start fixing PAM, glibc to always look in /usr/lib if a file is missing in /etc 15:44:56 d) we start working on adding tmpfiles snippets for the other packages 15:45:11 but as mentioned, even without all this, things work surprisingly well already 15:45:19 btw, i updated nspawn with some siwtches to test this all 15:45:43 in the version in rawhide there is now a new switc --volatile 15:45:52 it takes three values: 15:46:20 --volatile=no is the default, an means you get /usr, /var, /etc from whatever you point it to 15:46:22 haraldh - you said you are talking to rpm guys, right? 15:46:40 --volatile=yes you get a tmpfs as /, and /usr mountd in from where you point it 15:46:45 maybe it would help vpavlin if you talked with jzeleny, too 15:46:53 Will /etc/machine-id be moving? 15:46:57 sure thing 15:47:00 coolio 15:47:01 --volatile=state you get /usr and /etc from where you point it to, plus /var being tmpfs 15:47:20 these three models are kinda what we want to focus on 15:47:31 i also plan to add a kernel cmdline option to map this 15:47:31 vpavlin, I said, I was.... I am not constantly nagging.. 15:47:39 heh 15:47:39 dwalsh: it already moved 15:47:45 Where is it now? 15:47:48 dwalsh: oops, sorry 15:47:51 dwalsh: i am confused 15:47:52 vpavlin, nagging from different sides would definitely help :) 15:47:57 /etc/machine-id actually stays where it is 15:48:06 /etc/os-release howver moved to /usr/lib 15:48:13 /etc/machine-id is inherently per-instance information 15:48:20 it is the identity of a machine instance 15:48:23 hence belongs in /etc 15:48:31 haraldh ok:) I nag somebody all the time..no problem to add few more people;) 15:48:35 poettering, somewhat as an aside, i'd also like initramfs to be immutable, so it'd be quite nice if the machine id was a kernel commandline argument (or dracut knew how to pick it up from a secondary initramfs) 15:48:42 we moved /etc/os-release however, since that realy just describes the OS as it is vendor-supplied in /usr 15:48:56 walters: eww 15:49:03 (more generally, everything dracut computes locally and stashes in the initramfs were either a kernel cmdline, or a secondary initramfs) 15:49:10 there is things in the initramfs I wish were not 15:49:11 walters: if /etc is found empty, we will generate one add write it there 15:49:35 walters: where "generate" means, either randomize it, or pull it out of qemu's bios supplied machine uuid, or nspawn's --uuuid= cmdline 15:50:06 dgilmore, like? 15:50:07 How about a environment variable? 15:50:13 walters: i mean, if we allow provisioning of the uuid via qemu's -uuid or nspawns --uuid= switch, then i figure we can also allow it on the kernel cmdline 15:50:19 though i do wonder what your usecase is... 15:50:24 Trying to figure a way to populate for docker images. 15:50:42 dwalsh: i think docker should just implement the ContainerInterface stuff 15:50:51 dwalsh: http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/ 15:50:57 dwalsh: it should just set $container_uuid 15:51:02 dwalsh: that's where we pick it up from 15:51:03 poettering, Upstream is not very responsive :^( 15:51:13 haraldh: all the mount info. ive had quite a few times changing disks and ive had to use a resuce environment to recover things, and by rescue booting media in rescue mode. as the rescue kernel initramfs have not worked 15:51:27 Ok so we set container_uuid, systemd would populate. 15:51:39 dwalsh: only when /etc/machine-id is missing! 15:51:47 Right. 15:51:54 dwalsh: our docker base images will not have systemd in them 15:51:55 dgilmore, install dracut-config-generic 15:51:55 dwalsh: i.e. if you set container_uuid but there's already an /etc/machine-id from a previous one, we won't change it 15:52:17 as that instance already has it's id 15:52:23 Only last problem is the UUID of docker is much longer then /etc/machineid is allowed. 15:52:45 dgilmore, or use --no-hostonly-cmdline / hostonly_cmdline="no" 15:53:02 dwalsh: hmm, then it's probably not a uuid 15:53:08 dwalsh: uuids are always 128bit... 15:53:17 dwalsh: but you can just truncate it 15:53:26 dwalsh: but why would they ever use a longer one? 15:53:38 poettering: because they can 15:53:57 00eece37864040d3f5ca77e51a779859908277f43bb98f8bb00685f77bae9b35 15:53:57 i mean, do they really believe that they need mor docker containers than grains of sand on this world? 15:54:05 they are stupid, if i may say so 15:54:09 ok, I'll leave, I am not going to be here for next topic anyway..thanks for explanation poettering, I'll talk to jzeleny 15:54:28 poettering, I'll pass that along. :^) 15:54:29 seeya vpavlin 15:54:34 have a nic weekend all 15:54:37 cya vpavlin ! 15:54:52 have a nice weekend vpavlin! 15:55:34 so now that vpavlin is gone, we can all agree he's doing all the writeup of this, right? ;) 15:55:45 hehe 15:55:47 anyway, so far our ideas 15:55:53 or actually 15:55:57 one more thing: sysuers 15:56:03 we already had a discussion on fedora-devel 15:56:10 which resulted in a couple of things 15:56:18 which i all implemented in the most recent systemd 15:56:23 now i just need to update the FPC bug 15:56:36 there's one nasty thing there 15:56:36 nice! 15:56:39 which is this: 15:56:47 sysusers is supposed to cover both usecases: 15:57:14 "offline" updates as i decribed above, where a new file is dropped into /usr/lib/sysusers.d/, and then is worked on on next boot 15:57:30 and "online" updaets as we always did them with RPM, i.e. create users from scriptlets 15:57:40 now, for the latter case we actualy supply two RPM macros 15:57:45 but there's the nastiness: 15:58:04 we want to create these users based on definitions in the files we ship in /usr/lib/sysusers.d/ 15:58:24 but there might be files in the RPM that shall be owned by users that we want to create 15:58:33 so there's a chicken and egg problem 15:58:35 ugh 15:58:40 we want to create the users based on files we just dropped in 15:58:54 but before dropping in the files we want to create the users, so that the files can be owned by them 15:59:05 so, this is awful 15:59:11 we thought about this for quite a while now 15:59:16 our proposed solution is this: 15:59:31 we added a sysusers switch so that users can also be created based on the user defintions from stdin 15:59:46 the idea is then that packages can choose between two macros: 15:59:57 %sysusers_create() 16:00:18 which should be added to %post, and which just adds in all users from the just-dropped-in sysuers files 16:00:28 if you use that macro, you cannot make files owned by these users 16:00:38 and then there's %sysusers_create_inline() 16:00:53 where you can specify the precise "sysusers" line directly inside of the RPM macro invocation 16:01:00 that's what people should use in %pre 16:01:09 to create users before we drop in the first files 16:01:20