15:01:06 <abompard> #startmeeting Bodhi stakeholders (2019-06-26)
15:01:06 <zodbot> Meeting started Wed Jun 26 15:01:06 2019 UTC.
15:01:06 <zodbot> This meeting is logged and archived in a public location.
15:01:06 <zodbot> The chair is abompard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:06 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:06 <zodbot> The meeting name has been set to 'bodhi_stakeholders_(2019-06-26)'
15:01:14 <abompard> #meetingname bodhi_stakeholders
15:01:14 <zodbot> The meeting name has been set to 'bodhi_stakeholders'
15:01:23 <abompard> #topic salutations
15:01:57 <abompard> #chair bowlofeggs adamw kparal masta mboddu nirik puiterwijk Kellin cverna abompard nils mkonecny
15:01:57 <zodbot> Current chairs: Kellin abompard adamw bowlofeggs cverna kparal masta mboddu mkonecny nils nirik puiterwijk
15:02:00 <cverna> hello o/
15:02:13 <bcotton> .hello2
15:02:14 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com>
15:02:19 <pingou> ó/
15:02:25 <abompard> .hello2
15:02:26 <zodbot> abompard: abompard 'Aurelien Bompard' <aurelien@bompard.org>
15:02:37 * nirik waves, sort of here.
15:02:47 <cverna> .hello2
15:02:48 <zodbot> cverna: cverna 'None' <clems.verna@gmail.com>
15:02:54 <cverna> I am None :P
15:04:44 <nils> .hello nphilipp
15:04:45 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
15:05:14 <abompard> Hi everyone, let's get started
15:05:20 <abompard> #topic announcements and information
15:05:27 <abompard> #info Bodhi 4.1 beta deployed to staging
15:05:48 <abompard> #info Bodhi 4.0.2 is still the latest prod release
15:05:58 <abompard> And last but not least:
15:06:05 <abompard> #info Randy is a father now :-)
15:06:13 <cverna> \o/
15:06:29 <cverna> bowlofeggs++
15:06:31 <abompard> Mother and baby are OK
15:06:36 <nils> _o| \o/ |o_
15:06:47 <abompard> bowlofeggs_family++
15:06:58 <nirik> many congrats!
15:07:31 <abompard> He'll be away for 9 weeks and will come back to us recharg^H scratch that
15:07:42 <nils> :D
15:07:46 <abompard> he.... will do what he can
15:08:11 <abompard> #topic Looking forward
15:08:20 * mboddu kinda here
15:08:26 <abompard> At the moment we are testing 4.1 in staging.
15:08:53 <abompard> It should have the features to do rawhide gating for single-package updates, but it will not be enabled when it goes production
15:09:23 <abompard> If you want to follow the rawhide gating tickets:
15:09:25 <abompard> #info Rawhide Gating https://github.com/fedora-infra/bodhi/projects/3
15:10:00 <cverna> lot of PR reviewed the past few weeks :-)
15:10:03 <abompard> There should not be any backwards-incompatible changes in 4.1 (otherwise it'd have been a 5.0, right ;-) )
15:10:12 <abompard> cverna: indeed
15:10:58 <abompard> cverna: should we ask people to test single-package gating now? While in staging? When in prod?
15:11:26 <cverna> abompard: I think we can ask then while it is in staging, and maybe a targeted number of people
15:11:48 <cverna> abompard: like friendly people :P
15:12:13 <abompard> cverna: we have tons of that :-)
15:12:21 * mboddu sending an email to devel@ list :P
15:12:28 * cverna notes that if we don't ask you to test it does not mean your are not friendly :)
15:12:31 <cverna> :D
15:12:43 <abompard> We are currently fixing a couple bugs in staging and it would probably be better to wait for the second beta to be deployed
15:12:52 <abompard> which should happen, hopefully tomorrow
15:13:01 <cverna> +1
15:13:31 <cverna> are we still planning to deploy 4.1 in production this week ? or beginning of next week ?
15:14:24 <abompard> I don't know, it feels like it has slipped a bit. If we can't deploy tomorrow, then the next day is a Friday
15:14:31 <abompard> and we all know what not to do on a Friday
15:14:57 <cverna> yeah and then you go on PTO next week ? it becomes to be a bit risky
15:15:00 <abompard> And next week I will only be here at the beginning and then I'll escape this world for a 2 weeks PTO
15:15:42 <abompard> #info Aurelien will be gone from July 3rd to July 19th
15:15:49 * cverna thinks that if we decide to push 4.1 to prod we should probably test rolling back to the previous version in staging.
15:15:54 * nirik would prefer a few days to test updates pushes before everyone disappears.
15:16:20 <cverna> then have a sysadmin familiar with rolling back the application
15:16:23 <nirik> hardest part of that would be if there's database migrations.
15:16:25 <abompard> There are DB schema updates in 4.1
15:16:32 <nirik> ding
15:16:35 <cverna> yep
15:17:11 <cverna> there was a critical fix in 4.1 ? something currently hotfixed ?
15:17:23 * cverna can't really remember
15:17:25 <abompard> cverna: yes there was a fix, and it has been hotfixed
15:17:29 <abompard> it's a patch in the RPM
15:17:54 <cverna> ok so 4.1 is not that critical I would say
15:17:57 <abompard> it seems a bit dangerous if you ask me
15:18:11 <cverna> as long as we have it in staging for people to play with rawhide
15:18:26 <abompard> (dangerous to try to squeeze 4.1 prod deployment before I leave)
15:18:33 <cverna> agreed
15:18:34 <nils> mhm
15:19:56 <abompard> Anyone objects to not deploying 4.1 to prod before next Wednesday?
15:20:11 <abompard> (that's two negatives)
15:20:28 <abompard> Anyone really wants 4.1 in prod before next Wednesday? :-)
15:20:58 * nirik isn't in a hurry, except to get rawhide gating done/working
15:21:00 <abompard> Alright, we'll just have a very long testing time in staging then
15:21:13 <cverna> which is not bad :)
15:21:34 <nirik> so, just a random thought...
15:21:59 <abompard> #action 4.1 will be deployed to prod when Aurelien comes back, so starting on July 22nd
15:22:02 * mboddu agrees with nirik, not in a hurry, but get rawhide gating working
15:22:18 <nirik> do we keep rawhide gating working for mass rebuilds? would be a lot of stress, but that might be a good test.
15:22:45 <mboddu> nirik: I prefer no gating for mass rebuilds, it will be a lot
15:23:07 <cverna> just to clarify things currently rawhide gating as sold in the change proposal is not ready
15:23:10 <mboddu> And, we dont rebuild based on dep solved order, so if people start adding more tests, that will eventually fail
15:23:21 <cverna> we neeed multi package update and that's not finished
15:23:26 <nirik> if we don't want to at first sure, but I think we should try and cover it
15:23:29 <nirik> right
15:23:49 <nirik> if tests fail, then the thing is broken and we don't want it in. :)
15:24:11 <nirik> but just wanted to note mass rebuild as thats scheduled for july 24th
15:24:18 <mboddu> Right, I wonder how many fail like that
15:24:51 <mboddu> Well, if we think we are ready, all that I have to do is change the script to tag it to a different place than fxx-pending
15:25:03 <abompard> it's unlikely we'll have multi-package update gating by july 24th
15:25:04 <mboddu> So, we could change it any time
15:25:11 <cverna> I would say we are not ready :P
15:25:23 <cverna> yes what abompard says :P
15:25:36 <nirik> thats fine... but next time...
15:25:51 <cverna> yes that's something good to have in mind
15:25:54 <mboddu> nirik: +1
15:25:59 <abompard> yeah
15:26:07 <abompard> Okay!
15:26:13 <abompard> #topic triage
15:26:22 <cverna> mass rebuild  would basically do only single package update right ?
15:26:25 <cverna> oops sorry
15:26:29 <abompard> no go ahead
15:26:50 <nirik> cverna: yes
15:26:51 <mboddu> cverna: No, its all the active packages in entire Fedora
15:27:01 <nirik> mboddu: but it's one at a time...
15:27:18 <nirik> we aren't building stacks/collections, it's just one build after another.
15:27:22 <cverna> ok so we could maybe do a mini mass rebuild in staging
15:27:42 <nirik> possibly...
15:27:44 * mboddu thinking
15:27:50 <abompard> yeah that would be a good test. I like the idea of "mini mass" rebuild ;-)
15:28:03 <mboddu> Since mass rebuilds are done in a side tag, which bodhi has no idea about
15:28:14 <cverna> that could be a good use of the extra time in staging we have to test things
15:28:15 <mboddu> But we could submit it as a multi-package update
15:28:31 <cverna> oh man mboddu you scare me :P
15:28:34 <nirik> I think it should be single package updates... one per package
15:28:51 <mboddu> cverna: haha :D
15:28:58 <nirik> "we rebuilt this package, is it ok"
15:28:59 <mboddu> Yeah, what nirik said might be better and safe
15:29:22 <cverna> let's keep this idea and try to do something like that in staging
15:29:31 <mboddu> +1
15:29:33 <cverna> I think that would be a real good test
15:29:35 <nils> That would be a right mess, one humongous update for a mass rebuild. :)
15:30:06 <cverna> nils: yes :) I think we would find all the performance bottle neck of bodhi for sure :P
15:30:24 <abompard> Great
15:30:38 <mboddu> If multi-package update works as "if everything or nothing" then we will never get mass rebuild builds if submitted as multi-package update
15:30:45 <abompard> #info a mass rebuild will be attempted in staging using single-package gating
15:30:45 <nils> Mass rebuild builds could just be tagged into the normal tag for single pkg updates to be kicked off I guess.
15:31:14 <mboddu> nils: Yes, thats the plan
15:31:20 <cverna> I guess we can move on to the next topic :)
15:33:37 <abompard> Alright!
15:33:47 <abompard> #topic triage
15:33:53 <abompard> #info Bodhi's high priority issues: https://github.com/fedora-infra/bodhi/issues?q=is%3Aopen+is%3Aissue+label%3A%22High+priority%22
15:34:09 <abompard> Those are being a little left out by the Rawhide Gating initiative I think
15:34:40 <nirik> there's a lot of em
15:34:47 <cverna> I would not say that the first issue is High priority tho
15:35:10 <abompard> Well I wonder if the High Priority tag is not being abused a little bit
15:35:35 <nils> take that one up with Randy, he tagged it high prio :o)
15:35:37 <abompard> cverna: and I woulnd't say either that Troy's RFE are high prio
15:35:56 <nils> maybe we need "even higher than high prio"
15:36:02 <cverna> :P
15:36:05 <abompard> haha
15:36:21 <cverna> abompard: the Fedora CI one will be needed for rawhide gating
15:36:33 <nirik> it would be nice to have more granularity
15:36:49 <nirik> 63 "top priority" issues are hard to work on...
15:37:03 <cverna> I think the High Priority and Crash label can be a good start :P
15:37:04 <nils> abompard, the one at the top is covered by your Celery PR even. Perhaps split out the commit into its own PR?
15:38:26 <abompard> nils: right, ideally I would reference that bug in my commit message and be done with it
15:39:24 <abompard> I won't be able to do much of that but maybe while I'm away the bodhi team can try to address those?
15:39:30 <abompard> well, some of them
15:40:25 <nirik> are there multipackage update things people can be working on while you are away?
15:40:51 <nils> yeah, I don't think I'm running out of work soon
15:41:36 <cverna> nirik: yes these are tracked in this https://github.com/fedora-infra/bodhi/projects/3 in the Ready and To do column
15:41:40 <abompard> Yeah I'm only a bottleneck for releasing and deploying I think
15:41:45 <nils> there's also at least one pertinent issue in the list, https://github.com/fedora-infra/bodhi/issues/3061
15:41:46 <cverna> so still quite a few things to do
15:42:24 <nils> it looks like a prett substantial blocker for not so small multi pkg updates
15:42:29 <abompard> Cool. Anything else on the tickets?
15:42:30 <nils> *pretty
15:43:31 <abompard> nils: yeah, let's get Celery landed first and the we'll be able to fix that one
15:43:51 <abompard> OK!
15:43:54 <abompard> #topic Open floor
15:44:30 <nirik> so, as a bodhi-admin member I have been getting error messages from bodhi... but it's unclear if they are important/worth reporting or not...
15:44:41 <nirik> or should I try and go through them with someone sometime?
15:45:03 <nils> a specific kind of error messages?
15:45:38 <nirik> well, some repeats for sure...
15:45:38 <abompard> nirik: I'm recently subscribed to that, I think randy mentionned that most of those are Greenwave not responding to requests
15:45:52 <nirik> https://paste.fedoraproject.org/paste/Txod2epHhiwcN9vCGGCrVA for example is a recent one
15:45:57 <abompard> I haven't taken the time to analyze them yet
15:46:11 <nirik> there's also bugzilla faults...
15:46:26 <abompard> yep
15:46:34 <abompard> nirik: yeah that one's a bug I think
15:46:42 <abompard> we're not handling an empty list
15:46:45 <nirik> when it's doing pushes and waiting for the master mirror to update it gets 404s...
15:46:47 <cverna> yes hopefully greenwave performance will be really improved in 4.1 release
15:47:05 <nirik> and yeah, greenwave too
15:47:39 <nirik> so if you are gonna look at those I can just leave it at that...
15:47:53 <nirik> ideally, I'd prefer not to get emails that I can't do anything about fixing. ;)
15:48:03 <abompard> agreed
15:48:14 <nirik> there's this one too:
15:48:18 <nirik> 2019-06-26 04:04:28,222 ERROR [bodhi.server][f30-modular-updates] Cannot find mailing list address for update notice
15:48:39 <abompard> I think some should be reported to our issue tracker
15:48:44 <abompard> but I don't know which ones yet
15:49:21 <abompard> OK we're reaching the end of the meeting
15:49:27 <cverna> yes I think we need to at least have issues
15:49:55 <cverna> thanks abompard for running the show
15:49:58 <abompard> #endmeeting