14:37:29 <mboddu> #startmeeting RELENG (2017-07-10)
14:37:29 <zodbot> Meeting started Mon Jul 10 14:37:29 2017 UTC.  The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:37:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:37:29 <zodbot> The meeting name has been set to 'releng_(2017-07-10)'
14:37:29 <mboddu> #meetingname releng
14:37:30 <zodbot> The meeting name has been set to 'releng'
14:37:30 <mboddu> #chair dgilmore nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu Kellin
14:37:30 <zodbot> Current chairs: Kellin dgilmore masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll
14:37:30 <mboddu> #topic init process
14:38:01 <bowlofeggs> suuup
14:38:12 <dustymabe> o/
14:38:22 * threebean waves
14:38:28 <mboddu> bowlofeggs: you are here, yay
14:38:48 <bowlofeggs> ☺
14:39:17 <nirik> morning
14:39:50 <maxamillion> .hello maxamillion
14:39:50 <mboddu> #topic #6877 Refine cleaning up packages with broken deps
14:39:50 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
14:39:58 <mboddu> #link https://pagure.io/releng/issue/6877
14:40:19 <mboddu> sorry, hello everyone :)
14:40:38 <mboddu> nirik: how did we use to clean up broken deps before?
14:41:09 <mboddu> nirik: We didn't do it in f25, so I am not sure how we used to do it
14:41:32 <nirik> we didn't. Or manually
14:41:40 <contyk> .hello psabata
14:41:41 <zodbot> contyk: psabata 'Petr Šabata' <psabata@redhat.com>
14:42:24 <mboddu> nirik: oh
14:42:46 <mboddu> https://pagure.io/releng/issue/6877#comment-447244 is what I suggested, but everyone has their opinions
14:44:06 * nirik hasnt really had time to think about it.
14:47:04 <mboddu> bowlofeggs: do you have any idea how deps are calculated in updates-testing?
14:49:05 <kushal> .hellomynameis kushal
14:49:06 <zodbot> kushal: kushal 'Kushal Das' <mail@kushaldas.in>
14:49:11 <nirik> calculated? you mean broken deps listed?
14:49:27 <mboddu> nirik: yeah
14:49:48 <mboddu> nirik: I was thinking if we can block them in updates-testing phase
14:49:50 <nirik> repoclosure I think is used still.
14:50:09 <nirik> well, it's sometimes hard...
14:50:11 <mboddu> nirik: so that they wont ever show up in stable
14:50:40 <nirik> you have say an update with 20 packages that changes a library, and breaks 1 other package... you block the 20?
14:50:44 <bowlofeggs> mboddu: i do not
14:51:33 <mboddu> nirik: okay, not a good idea :(
14:51:41 <bowlofeggs> there is a taskotron test that checks for broken deps
14:51:45 <nirik> yeah, its not an easy problem
14:52:02 <bowlofeggs> with greenwave and bodhi, we could set a test that requires deps to be met before reaching updates-testing…
14:52:17 <nirik> yeah, but that has it's problems too...
14:52:20 <bowlofeggs> (not with current bodhi, but we're working on integrating with greenwave)
14:52:54 <nirik> urgent CVE on foo, update foo, bar, baz, but this leaf package foobar breaks... it will have a fix later, but we have to push this security update now.
14:53:15 <bowlofeggs> it might be possible to allow an admin override or something
14:53:22 <bowlofeggs> or maybe even just let the packager override
14:53:28 <bowlofeggs> some kind of "i know what i am doing" thing
14:53:34 <nirik> sure. just saying that would end up with broken deps getting in.
14:53:39 <bowlofeggs> yeah
14:53:54 <mboddu> there is a no neat way of solving this thing
14:56:27 <mboddu> How about doing it once in a release cycle and block the dep issue pkgs and either people will fix them or file a ticket to unblock them(which we can review and unblock them)?
14:57:16 <nirik> I think we may want to propose whatever to the devel list before doing it... and see if we can reach consensus...
14:57:55 <threebean> yeah - if we go the taskotron route with greenwave
14:58:14 <threebean> ... the waiverdb system is the thing that provides a way to override the test failure.
14:58:20 <threebean> you "waive" the failure.
14:59:28 <maxamillion> nirik: +1
14:59:33 <nirik> one advantage of that is that someone would have to go do that... so at least would be more aware...
15:00:49 <mboddu> "[10:58:20] <threebean> you "waive" the failure" who is "you" here, maintainer or releng/infra team?
15:02:02 <threebean> depends on the policy configured in greenwave.  the current thinking is that we'd give releng rights to waive on every package.  package owners could additionally waive, but only on their own packages.
15:04:03 <mboddu> threebean: okay, this happens through bodhi though, right?
15:07:02 * threebean nods
15:07:12 <mboddu> threebean: okay, thanks
15:07:19 <dustymabe> next topic? kushal and I have have something for open floor
15:07:36 <threebean> (technically, we'll have a button in the bodhi ui to waive.  when you click it, it will really redirect you to a separate waiverdb.fedoraproject.org site which accepts the request.)
15:07:54 <mboddu> threebean: okay
15:08:01 <mboddu> dustymabe: sure, in couple of min
15:08:19 <mboddu> nirik: so, what do you think of waiverdb with bodhi solution?
15:09:04 <nirik> not sure. Propose to devel list and see what people think? ;) Not sure it solves all the issues, but might help if we block at branching first
15:09:27 <mboddu> if everyone is okay with it, I can put that in an email and send it to devel list for any other thoughts and update the ticket
15:10:04 <mboddu> Okay, I will put both of the options and we can look at the responses
15:11:45 <mboddu> #info There isn't any good way to solve this issue, we have couple of thoughts. 1. Blocking the pkgs at branching and unblock them as necessary 2. Blocking the pkgs through bodhi->waiverdb and maintainers can unblock their own pkgs or releng can unblock every pkg
15:13:30 <mboddu> Lets do alt arches and open floor and if we have any more time, we can look at other topics
15:13:46 <mboddu> #topic Alternative Architectures updates
15:14:01 <mboddu> sharkcz : any updates?
15:14:38 <sharkcz> hi, I was on PTO last week, F-26 GA compose for s390x is on my to-do for the next hours/day
15:15:00 <sharkcz> I'm not aware of any issues
15:15:09 <sharkcz> it should be smooth process
15:15:21 <dustymabe> sharkcz: famous last works :)
15:15:26 <dustymabe> words*
15:15:32 <mboddu> sharkcz: when are you planning on releasing f26 ga for s390x?
15:15:43 * nirik will ping admins about additional capacity. They said they were going to double machines, but hasn't happened yet that I know of.
15:15:54 <mboddu> sharkcz: in a week or two weeks after the f26 ga?
15:15:55 <sharkcz> mboddu: I guess on Wednesday
15:16:12 <mboddu> sharkcz: okay, sounds great
15:17:32 <mboddu> sharkcz: I think you need to let mattdm know about it, last week he was asking about s390x release
15:17:57 <sharkcz> mboddu: ok
15:19:03 <mboddu> #info sharkcz is planning to release F26 s390x GA on Wednesday - Jul 12th 2017
15:19:07 <mboddu> sharkcz: thanks
15:19:10 <sharkcz> np
15:19:13 <mboddu> #topic Open Floor
15:19:16 <kushal> I am working on removing mash from bodhi, and use pungi instead. Last Friday, puiterwijk helped to understand the current codebase (which I was having trouble with). Today I have written blog post explaining the current code https://kushaldas.in/posts/story-of-mashing-in-bodhi-fedora-updates-system.html
15:19:21 <mboddu> that is quick
15:19:53 <kushal> Now, if we use directly use pungi for everything related to mashing, we could remove a lot of code (related to ostree generation) from the current masher.
15:20:00 <maxamillion> kushal: awesome!
15:20:04 <dustymabe> so kushal believes he can have bodhi calling pungi to do mashing in 1 month
15:20:11 <mboddu> kushal: but pungi also has mash code in it, if I am not wrong
15:20:13 <dustymabe> are there any blockers that we know of that would prevent this from happening?
15:20:33 <kushal> Previously dgilmore told that rel-eng will provide the pungi configs (for various tags).
15:20:41 <kushal> https://pagure.io/releng/issue/6893 is the ticket I opened for the configs
15:21:18 <kushal> I still have to write a few parts related to reading the config, WIP PR is available here (for bodhi) https://github.com/fedora-infra/bodhi/pull/1676
15:21:30 <kushal> mboddu, We will just use the pungi-koji command itself.
15:21:57 <kushal> mboddu, This way, the nightly and bodhi, both will use the same codebase (what ever is inside on pungi).
15:22:38 <kushal> Now, bowlofeggs asked to make sure for the start this should work based on a config, if we change it, then we should be able to use the old code (without any redeploying).
15:22:53 <kushal> I am writing the code in that way (means if else block).
15:23:05 <kushal> Only when things will be working fine, we will remove the old code.
15:23:19 <kushal> mboddu, Now, we will need your help to get those pungi-configs done.
15:23:22 <dustymabe> right, so we will add code that can be enabled or disabled via a config option
15:23:34 <dustymabe> if it doesn't work we just disable the code using the config option
15:23:37 <dustymabe> and use the old code
15:23:41 <mboddu> kushal, dustymabe : I can help however way possible
15:24:10 <maxamillion> kushal: +1
15:24:13 <dustymabe> does anyone know of any blockers (technical) for why this won't work?
15:24:22 <mboddu> kushal: but I thought the problem with bodhi and pungi for ostree generation is timing/race condition
15:24:23 <dustymabe> bowlofeggs: had a question last week about updateinfo.xml
15:24:34 <kushal> mboddu, Thanks, I also a few open questions in that PR.
15:25:10 <kushal> mboddu, Yes, but if we just use pungi to generate everything, then it will take care of the things (than we trying to control).
15:25:42 <kushal> mboddu, based on the pungi config of course (which is scary long) :)
15:25:45 <dustymabe> mboddu: yeah - no more timing issues there if we create the ostree and the ostree_installer and image_builds all at the same time
15:26:09 <bowlofeggs> dustymabe: i did?
15:26:32 * kushal has nothing else to talk.
15:26:44 <mboddu> kushal, dustymabe : so, everything will happen in nightlies or bodhi will call pungi to do stuff?
15:26:55 <dustymabe> bowlofeggs: yes
15:26:57 <dustymabe> 10:07:18   bowlofeggs | kushal: does pungi know how to make drpms and updateinfo.xml files?
15:26:59 <maxamillion> the pungi config is bonkers
15:27:19 <dustymabe> mboddu: bodhi will call pungi to do stuff
15:27:20 <mboddu> maxamillion: +1 :)
15:27:22 <kushal> mboddu, nightlies will continue, and when bodhi will generate composes, it will also call pungi to do so.
15:27:33 <kushal> maxamillion, yeah, nightmarish
15:27:33 <bowlofeggs> ah yes
15:27:50 <dustymabe> maxamillion: i actual understand the pungi configs much better now
15:28:17 <dustymabe> bowlofeggs: are those still open questions?
15:28:22 <dustymabe> and who can help us answer those?
15:28:35 * dustymabe wishes dgilmore was around to bless/curse this proposal
15:28:39 <bowlofeggs> yeah i guess - we need to make sure that we have updateinfo.xml and drpms as a result of this
15:28:45 <mboddu> kushal, dustymabe : I have more questions about it, we can discuss about it later
15:28:52 <bowlofeggs> however they are made, they need to still exist ☺
15:29:03 <kushal> mboddu, okay, I also have many questions for you :)
15:29:47 <dustymabe> mboddu: can we set up a meeting time to discuss this further?
15:30:30 <mboddu> #info There is proposal on removing mash from bodhi, and use pungi instead. https://pagure.io/releng/issue/6893
15:30:45 <dustymabe> i'm thinking dgilmore mboddu nirik bowlofeggs kushal and myself would be required
15:31:01 <mboddu> dustymabe: sure, probably later this week. I will be busy until Wed
15:31:06 <dustymabe> mboddu: ok
15:31:14 <maxamillion> I'd like to be there too, I'm going to be porting mash to python3 before long and I'd like to know it's fate
15:31:19 <mboddu> dustymabe: sounds great
15:31:31 <dustymabe> kushal: still would like to see details of this proposal sent as a mail to the lists
15:31:34 <dustymabe> infra + releng
15:31:44 <dustymabe> we'll set up a meeting for wednesday
15:31:46 <kushal> dustymabe, As I said, I will :)
15:31:50 <threebean> oh man.  sorry I missed the convo
15:32:03 <threebean> but mcurlej is also working on modifying bodhi to do mashing with pungi instead of mash.
15:32:04 <mattdm> sharkcz: awesome thanks for that update. are power and aarch64 tomorrow?
15:32:05 * nirik is out later this week.
15:32:08 <mboddu> dustymabe: after Wed please :)
15:32:18 <mboddu> dustymabe: Thu or Fri :D
15:32:19 <mattdm> (ooops sorry that was a long way back.)
15:32:37 <sharkcz> mattdm: yes, aarch64 and ppc64 + ppc64le go out tomorrow, they are part of the regular process
15:32:40 <bowlofeggs> maxamillion: i was under the impression that mash was going to be abandoned? i could be very wrong
15:32:41 <mboddu> mattdm: yes, power and aarch64 are part of primary koji now
15:32:41 <threebean> although... only for the modular use case.  we talked about not touching the mainline mash so we can experiment with pungi-in-bodhi without risking the primary mashes.
15:33:24 <maxamillion> bowlofeggs: I've been told it is and it isn't about a dozen times which is why the porting work keeps getting pushed back (it was originally on my TODO list back in January)
15:33:45 <bowlofeggs> kushal: you should probably coordinate with mcurlej about the pungi work. i didn't realize he was doing that too until threebean said so just now
15:33:48 * nirik thinks everyone needs to be on the same page here before we start doing a bunch. ;)
15:33:54 * threebean nods
15:34:02 <mboddu> maxamillion, bowlofeggs : we didn't got what we wanted from dist repos, so we need to port mash to python 3 :(
15:34:02 <maxamillion> bowlofeggs: most recently I've been told "this needs to be done by F27" so I was planning to start it as soon as the OSBS multi-arch stuff is done
15:34:27 <bowlofeggs> oh wow
15:34:28 <bowlofeggs> fun
15:34:41 <maxamillion> mboddu: question though, would it be better to just add what we need to dist-repos than to port mash and keep limping it along?
15:34:54 <kushal> threebean, Does he come online to regular fedora channels?
15:34:56 <maxamillion> I don't really care either way, just curious
15:35:27 <nirik> dist-repos that worked for our updates would have advantages over pungi for our updates...
15:35:29 <mboddu> maxamillion: its not in the priority list of koji dev team, so....
15:35:33 <threebean> kushal: yeah - usually #fedora-modularity is where we've been discussing.
15:36:05 <maxamillion> mboddu: right, but if I'm going to have to write code to port mash anyways ... would it be better for me to just write code to make dist-repos do what we need instead?
15:36:09 <kushal> threebean, ah, we should ask him to come in the #fedora-releng too :)
15:36:19 <maxamillion> SO MANY IRC CHANNELS
15:36:21 <maxamillion> >.>
15:36:47 <threebean> kushal: i'll start an email thread to introduce you two.  fyi, for background -> https://fedoraproject.org/wiki/Infrastructure/Factory2/Focus/Bodhi
15:36:50 <nirik> maxamillion: I'd say yes, but I don;t know how hard that would be.
15:37:24 <mboddu> maxamillion: I agree with nirik here
15:37:34 <kushal> threebean, Thank you, please use mail AT kushaldas.in if possible :)
15:38:14 <maxamillion> nirik: me either, but porting mash isn't going to be easy because there's various parts of dnf vs yum api that just aren't compatible in any way so I'll be ripping reasonable sized chunks of code out and rewritting anyways .... I just want to make sure the effort goes towards the better option, whichever that may be
15:39:35 <mboddu> Okay lets close it, we can take remaining discussion to #fedora-releng channel
15:39:40 <mboddu> thanks guys for joining
15:39:50 <mboddu> #endmeeting