20:04:44 <sgallagh> #startmeeting serversig
20:04:44 <zodbot> Meeting started Tue Mar 13 20:04:44 2018 UTC.  The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:04:44 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:04:44 <zodbot> The meeting name has been set to 'serversig'
20:04:50 <sgallagh> #meetingname Server SIG
20:04:50 <zodbot> The meeting name has been set to 'server_sig'
20:05:21 <sgallagh> #topic Roll Call
20:05:53 <dperpeet> .hello dperpeet
20:05:54 <zodbot> dperpeet: dperpeet 'None' <dperpeet@redhat.com>
20:08:50 <adamw> .hello adamwill
20:08:51 <zodbot> adamw: adamwill 'Adam Williamson' <awilliam@redhat.com>
20:09:36 <sgallagh> Low attendance this week, it seems
20:10:03 <sgallagh> Well, I only have one topic to discuss
20:10:19 <sgallagh> #topic The Future of FreeIPA
20:10:53 <sgallagh> adamw: how many releases in a row has FreeIPA been broken until after the first Freeze date?
20:11:16 <adamw> well, 27 and 28 at least.
20:11:24 <adamw> i don't have info from before that right on hand, i'd have to go look it up.
20:11:51 <sgallagh> I'm pretty sure 26 also
20:11:55 <dperpeet> I'm not sure how salient this is, but my team is currently working on fedora CI for ipa
20:12:04 <sgallagh> dperpeet: That's... definitely relevant
20:12:15 <adamw> although...finding out that it's broken in fedora isn't the problem
20:12:24 <adamw> we already test it nightly, and know when it's broken.
20:12:42 <adamw> (we also test updates.)
20:12:52 <sgallagh> The question before us is whether FreeIPA is something that we (Fedora Server) want to continue using as our flagship feature and, by extension, whether we continue blocking for it in F29+
20:13:38 <adamw> for f26, it started working around 2017-04-13, i think.
20:14:01 <adamw> which was a month after Alpha freeze.
20:14:29 <dperpeet> well, is it a flagship feature from a user perspective?
20:14:46 <sgallagh> dperpeet: Define "user perspective" in that sentence, please
20:15:08 <adamw> i am not sure i'd claim to know anything at all about what anyone who downloads a fedora server image actually does with it, to be honest.
20:15:20 <dperpeet> sgallagh, how important is it to users?
20:15:21 <sgallagh> From the perspective of the Server SIG, it's a flagship feature in that it's how we showcase that Fedora is the place where server development happens
20:15:49 <adamw> i tend to suspect a lot of them just use it as a network install and deploy whatever the hell they were deploying on it before we ever invented Server, ignoring cockpit, rolekit, and whatever else we've come up with. but i am basing this on no actual data, just general cynicism. :P
20:16:08 <adamw> there are definitely users of FreeIPA on fedora, in that people complain when it doesn't work.
20:16:08 <sgallagh> adamw: Well, we have positive feedback on Cockpit at least.
20:16:42 <sgallagh> dperpeet: So, a brief history lesson
20:16:44 <adamw> but anyway, sgallagh's question is more about our POV than the users'
20:17:11 <adamw> practically speaking, at present, the bits of Server that really matter a lot are: cockpit, FreeIPA, postgres, rolekit, and some firewall integration things.
20:17:14 <dperpeet> I guess it boils down to: is it worth blocking on?
20:17:16 <sgallagh> When we began Server Edition, the intention was to stop shipping the bag of lego blocks and instead start providing something closer to Windows Server, in that we would guide people towards preferred implementations
20:17:23 <adamw> from our perspective in terms of building it and shipping it. those are the things we block on.
20:17:29 <adamw> dperpeet: right, that's a large chunk of it.
20:17:29 <sgallagh> FreeIPA was the de-facto answer to Active Directory
20:17:47 <dperpeet> I remember long meetings about phrasing the goals :)
20:18:19 * nirik arrives
20:18:51 <sgallagh> Furthermore, we made decisions that FreeIPA was important in that we had decided at the time that we were willing to have our offerings require a functioning IPA domain for full effect
20:20:32 <dperpeet> well, if we don't block... does that mean we'll have *some* older version that works?
20:20:38 <sgallagh> dperpeet: No
20:20:48 <dperpeet> and thank you for the brief history :)
20:21:00 <sgallagh> It likely means that FreeIPA will be in whatever state the maintainers can scrape together
20:21:13 <sgallagh> Which recent history suggests is "who the hell knows"
20:21:18 <adamw> so, up front, i don't think we're at 'burn it all down' stage yet
20:21:20 <nirik> I still think it's pretty nice/important... shows that we can ship a complex application... (or not I guess if we are not able to)
20:21:40 <adamw> i do think overall we've not done *badly* here, we've delivered i think 6 straight releases with freeipa actually working pretty well. which really isn't a bad achievement.
20:21:43 <dperpeet> I'm leaning toward a "remedial" mode
20:21:48 <adamw> i just think we can maybe go about doing it more smoothly.
20:22:16 <sgallagh> I tried today to get the FreeIPA maintainers to commit to shipping it as a module stack that we can keep the old version running on
20:22:21 <adamw> and, i mean, we have upstream/downstream relations to think about. DS is a big product for RH. i think it demonstrates pretty good interaction/value that we ship essentially a full working version of it in every Fedora release.
20:22:27 <sgallagh> But they were unwilling to commit to that
20:22:30 <adamw> i know freeipa folks have said they value the feedback they get from fedora testing.
20:22:56 <adamw> bit more background: this is kinda coming out of me flagging up that freeipa is frequently broken for long periods in Fedora
20:22:58 <dperpeet> I for one can pledge to help integrate fedora and their upstream testing a bit more
20:23:14 <sgallagh> adamw: I know it has value and I really want to keep shipping it. However, I'm becoming more and more convinced that RHT is not committing the resources necessary to make this happen.
20:23:16 <adamw> and that it's currently broken for f28, and we're deep into the beta freeze.
20:23:26 <sgallagh> And I wonder if it isn't time for us to put some of the pressure back on them
20:23:28 <dperpeet> one concrete measure: with fedora ci getting results in fedora, we could leverage the centos ci infrastructure to give feedback on their github pull requests
20:23:35 <adamw> sgallagh: i don't think i'm yet convinced this is really a resource problem, as opposed to an organization one.
20:23:36 <dperpeet> so they can see fedora breakage early
20:23:40 <dperpeet> like cockpit
20:24:03 <sgallagh> adamw: The FreeIPA people specifically cited it as a resource problem. There are zero Dogtag people working in Fedora right now, among other things
20:24:14 <adamw> dperpeet: real talk: is this actually likely to get done, to a state where it's reliable enough that upstream can use it, before all these deadlines you guys have?
20:24:47 <dperpeet> adamw, people on my team have been working on ipa tests in the standard test interface for a while now
20:24:54 <dperpeet> >= 2 weeks
20:25:21 <dperpeet> but we will likely need some more upstream dev help
20:25:23 <dperpeet> from freeipa
20:25:31 <dperpeet> to make some of the deadlines
20:25:54 <adamw> sgallagh: well, i'm not convinced what's needed is "more work", exactly. it seems more like just better co-ordination and understanding of what we can do to resolve problems.
20:26:04 <sgallagh> adamw: Perhaps
20:26:14 <adamw> i'm particularly interested for e.g. in how this nss compatibility issue wound up blocking us for, like, months. i'm sure there should have been a better path there.
20:26:28 <dperpeet> from what I'm hearing, a lot of this is about the feedback loop
20:26:42 <dperpeet> i.e. there is interest in having this work, but we have a "broken" state for a long time
20:27:32 <nirik> is that a bug on the nss side? or the freeipa side adjusting to nss?
20:27:34 <sgallagh> dperpeet: The fact that it was broken wasn't unknown
20:27:48 <sgallagh> nirik: NSS made a backwards incompatible change when it switched database formats
20:27:58 <nirik> ah, the sqlite thing?
20:28:08 <sgallagh> This broke FreeIPA badly, as many of its subcomponents needed to be made capable of dealing with it
20:28:11 <dperpeet> sgallagh, I say we work on closing the feedback loop
20:28:22 <dperpeet> and that these factors will weigh in "blocking" decisions
20:28:24 <dperpeet> when the time comes
20:28:28 <sgallagh> sure
20:28:35 <sgallagh> nirik: Yes, the SQLite thing
20:28:54 <nirik> yeah, it sounds to me like they could have raised the alarm there and nss could have held off landing that until freeipa was ready. I didn't hear anything at all about sqlite change causing any problems...
20:28:59 <dperpeet> and we could clearly state that the likelihood of blocking on one thing is lower, with increased release complexity
20:29:04 <sgallagh> And because of course Firefox updates get marked for stable within minutes of Bodhi update request, everything went to hell
20:29:36 <nirik> "It's up to developers of NSS applications, if they accept the new default and an automatic conversion, or if they prefer to continue to use the classic DBM storage format. Although not recommended, developers can easily do so, by adding a "dbm:" prefix to the storage parameter they provide to NSS at NSS library initialization time. "
20:30:06 <sgallagh> nirik: Except that FreeIPA interoperates with the system database, which got upgraded
20:30:16 <sgallagh> *system certificate database
20:30:38 <nirik> fun
20:30:50 <adamw> yeah, the system cert stuff complicates anything that involves it
20:30:50 <sgallagh> I don't know all of the details, only that it was apparently very complicated to resolve
20:31:15 <adamw> i don't think there's a break in the feedback loop *exactly*, but i'm not sure *all* information gets through it
20:31:18 <sgallagh> And they had to do it all reactively because that change landed as a Firefox update
20:32:28 <adamw> what i'm primarily interested in is, were the right communication links set up such that freeipa folks talked to nss folks and considered possibilities like whether the nss change could be reverted in fedora, or whether some sort of compat layer could be added?
20:32:46 <adamw> or was the change basically taken as a fait accompli and the *only* courses considered were 'how do we make all the freeipa bits work with the new nss'?
20:33:07 <adamw> it's stuff like that where i feel we could at least look at whether we can make things better, before we decide it's futile
20:34:13 <sgallagh> Alright, we're 30 minutes in. Do we want to table this for now?
20:34:20 <sgallagh> I think dperpeet had another topic for this week
20:34:37 <dperpeet> sgallagh, well
20:34:53 <dperpeet> I think there is consensus that the bar will be higher for blocking, right?
20:34:58 <dperpeet> that might be worth a message
20:35:06 <dperpeet> for what it's worth :)
20:35:49 * nirik would prefer to have more info. It would seem pretty sudden to drop it as blocking without exploring ways to make it better first
20:36:00 <dperpeet> ok, sounds like tabling is better then
20:36:06 * sgallagh nods
20:36:06 <dperpeet> I'll have more info in 1-2 weeks
20:36:11 <dperpeet> with progress on tests
20:36:28 <sgallagh> dperpeet: Thanks
20:36:30 <sgallagh> dperpeet++
20:36:30 <zodbot> sgallagh: Karma for dperpeet changed to 2 (for the f27 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
20:37:09 <dperpeet> sgallagh, do you want me to tag the topic? =)
20:37:18 <sgallagh> #chair dperpeet
20:37:18 <zodbot> Current chairs: dperpeet sgallagh
20:37:19 <sgallagh> Please
20:37:25 <sgallagh> #chair nirik adamw
20:37:25 <zodbot> Current chairs: adamw dperpeet nirik sgallagh
20:37:28 <adamw> yeah, table for now, i'm curious to see where these threads i've been cced on go
20:37:40 <sgallagh> ack
20:37:55 <dperpeet> #topic Packages in Fedora Server
20:38:15 <dperpeet> so as most of you probably know, we've been testing packages that are part of Fedora Atomic Host
20:38:54 <dperpeet> and Fedora will soon have a CI pipeline running packages on non Atomic packages
20:39:06 <dperpeet> it exists already, but not in production
20:39:17 <dperpeet> so the question is, which packages we should run on
20:39:22 <dperpeet> and gather statistics
20:39:37 <dperpeet> rasibley, do you want to add something that describes the issue better?
20:40:17 <rasibley> dperpeet, I think you described it well, we just need to know which set of base packages we should prioritize the testing on
20:40:57 <sgallagh> So what you're looking for is for us to provide a list of critical packages or for us to approve a handy list you may have already made...? :)
20:41:39 <sgallagh> Or just make a blanket statement like "If it's part of the default install, plus these X important extras"...?
20:41:52 <adamw> this would also require that some useful tests have already been committed to the package, right?
20:42:31 <sgallagh> Or are you asking for a list of things that your team would be creating tests for?
20:42:35 <nirik> do we have a list of packages with existing tests?
20:42:37 <dperpeet> well, packages on this list can expect some priority regarding help from fedora-ci
20:42:47 <sgallagh> dperpeet: OK, that's good to know
20:42:50 <adamw> or is this about nominating packages that *should get their tests migrated*?
20:43:12 <adamw> nirik: https://fedoraproject.org/wiki/CI/Tests/stat , first column, I believe - rasibley can confirm or deny :P
20:43:42 <nirik> ah ha
20:43:49 <dperpeet> well, ideally the list will also have some relation to downstream stuff happening
20:43:59 <dperpeet> to put it bluntly
20:44:34 <adamw> i mean, a good starting point is that list i wrote earlier - cockpit, freeipa, postgres, firewalld
20:44:55 <dperpeet> I would set the scope of everything that a basic os should contain
20:44:56 <rasibley> adamw, yeah those are the atomic hosts packages we currently track, although the list is wrong at the moment :)
20:45:03 <dperpeet> more than Atomic, for sure
20:45:18 <adamw> ah, okay, bigger scope than i realized.
20:45:18 <rasibley> but ideally a separate dashboard for non atomic host packages we would prioritize for CI
20:45:27 <nirik> well, I guess for us, everything on the server dvd ?
20:46:08 <dperpeet> sgallagh, back to your question: I would say we gather some info and propose a list
20:46:15 <adamw> i mean, i guess i'm worried that this list is going to be so large as to be impractical
20:46:32 <adamw> would a non-prioritized list of...i'm guessing at least ~1000 packages...be useful?
20:46:39 <nirik> yeah, and many things with no tests...
20:46:59 <adamw> (that's what you'd get if you asked Server and Workstation both this question, I believe)
20:47:25 <dperpeet> rasibley, how many packages were on the list we brainstormed?
20:47:26 <adamw> or should we start with the big list and then try to help prioritize it?
20:48:06 <rasibley> around ~1000 :)
20:48:57 <dperpeet> rasibley, do you have a list you can paste a link to?
20:49:19 <rasibley> dperpeet, this is the initial list we are working from: https://paste.fedoraproject.org/paste/IctMEHLdbnM3g53m-3C-CA
20:49:25 <dperpeet> thank you
20:49:25 <adamw> i mean, i think https://kojipkgs.fedoraproject.org/compose/branched/Fedora-28-20180313.n.0/logs/x86_64/createrepo-Server.rpm.x86_64.log should list all packages in the server DVD (i'd have to check that)
20:49:40 <sgallagh> I think a reasonable place to start might be the Server default install which is around 500 packages
20:49:42 <dperpeet> I suggest we just put the list out there for now as a suggestion and everyone think about their input regarding priorities
20:49:52 <adamw> and https://kojipkgs.fedoraproject.org//work/tasks/5549/25685549/livemedia-out.log lists every package in the Workstation live image
20:49:57 <sgallagh> (Actually, that's runtime packages; the number of associated SRPMs is lower)
20:50:14 <dperpeet> let's call our list the working set
20:50:18 <sgallagh> sure
20:50:22 <dperpeet> and we're happy to entertain "focus lists"
20:50:31 <dperpeet> i.e. we could have a server subset
20:50:53 <dperpeet> that we can use to measure CI coverage and health for Server
20:51:21 <adamw> sgallagh: i'd suggest maybe all the packages installed after you deploy both release-blocking roles (successfully)
20:51:42 <sgallagh> adamw: Yeah, reasonable
20:51:53 <dperpeet> adamw, would you have the capacity to compare what you suggested against our proposal?
20:52:25 <dperpeet> especially interesting if it turns out not to be a true subset
20:52:33 <adamw> dperpeet: i think so, yes. once we dig out from under this giant mine collapse labeled 'fedora 28 beta'. :P
20:52:47 <dperpeet> :D
20:52:58 <adamw> eh, i can probably do it now before i forget about it...
20:53:14 <dperpeet> I will note to sync up with you further on the list
20:53:42 <dperpeet> we have to iron out some kinks, but we should be getting statistics up soon-ish
20:56:39 <dperpeet> I think we're done on this topic for now
20:57:08 <sgallagh> OK, sounds good
20:57:14 <sgallagh> #topic Open Floor
20:57:44 <sgallagh> Going once
20:57:56 <sgallagh> Going twice
20:58:02 <adamw> practically speaking, server is go/no-go next week, allegedly
20:58:10 <adamw> and we still don't have working freeipa, or aarch64 testing at all
20:58:20 <nirik> and who knows about modulatiry. ;)
20:58:23 <adamw> oh, and there's this rolekit systemd bug. so, yeah.
20:58:26 <adamw> everything is fine!
20:58:40 * sgallagh looks up from pouring gasoline everywhere. Hmm?
20:58:57 * adamw hands sgallagh a match, runs
20:59:26 <adamw> if anyone asks, we were all having dinner at dperpeet's place when the fire started
21:00:03 <dperpeet> sounds fair
21:00:10 <adamw> you'll cover for us, right? :)
21:00:11 <sgallagh> I suspect strongly that we're looking at a slip next week, but I'm open to being pleasantly surprised
21:00:25 <nirik> yeah.
21:00:32 <dperpeet> adamw, I'll vouch for everyone who was here!
21:01:02 <dperpeet> and technically, if you ordered pizza to be delivered to my house, you were here in spirit
21:01:03 <dperpeet> :)
21:01:10 <adamw> ooh, good one.
21:03:42 <sgallagh> Thanks for coming, folks
21:03:44 <sgallagh> #endmeeting