17:01:31 <mboddu> #startmeeting RELENG (2018-09-06)
17:01:31 <zodbot> Meeting started Thu Sep  6 17:01:31 2018 UTC.
17:01:31 <zodbot> This meeting is logged and archived in a public location.
17:01:31 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:31 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:01:31 <zodbot> The meeting name has been set to 'releng_(2018-09-06)'
17:01:31 <mboddu> #meetingname releng
17:01:31 <mboddu> #chair nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu Kellin dustymabe
17:01:31 <mboddu> #topic init process
17:01:32 <zodbot> The meeting name has been set to 'releng'
17:01:32 <zodbot> Current chairs: Kellin dustymabe masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll
17:01:42 <nirik> morning
17:01:52 <mboddu> nirik: How is it going?
17:01:55 <bcotton> .hello2
17:01:56 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com>
17:01:59 <dustymabe> .hello2
17:02:00 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
17:02:41 <nirik> I can't complan, but sometimes I still do. ;)
17:02:49 <mboddu> nirik: haha
17:02:58 <mboddu> bcotton, dustymabe : hello guys
17:03:04 * dustymabe waves
17:03:06 <bcotton> hello, mboddu
17:03:12 <dustymabe> just eating lunch and bjn meeting at the same time
17:03:14 <dustymabe> don't mind me
17:03:46 <nirik> dustymabe: need to be also riding a unicycle and juggling... ;)
17:04:09 <mboddu> haha :)
17:04:27 <mboddu> Okay, I got one topic to discuss today
17:04:32 <mboddu> So, lets get started
17:04:43 <mboddu> #topic #7763 define and implement retention policy for nightly composes
17:04:49 <mboddu> #link https://pagure.io/releng/issue/7763
17:05:26 <mboddu> This thing came up last week's meeting, now we want to come up with a solution
17:07:40 <mboddu> I added my idea to the ticket, but I am open to ideas
17:07:58 <nirik> I'm just a bit concerned how much resources this might take up...
17:08:05 <nirik> but I guess we can try and watch it and see.
17:09:36 <mboddu> nirik: Are you talking about my idea or in general implementing a longer retention policy?
17:09:53 <nirik> just the idea of keeping more things around...
17:10:03 * nirik reads up on the latest comments
17:12:24 <nirik> ah, dunno if just being able to make old composes would meet sharkcz's needs...
17:12:44 * sharkcz is here
17:12:53 <sharkcz> recreating old compose should be OK
17:12:59 <nirik> I guess it would, but would need a 'make me this old compose' and wait for it.
17:13:35 <mboddu> Also, one thing to remember, the compose might get affected with BRO's
17:13:44 <mboddu> It wont be the exact thing, but close to it
17:14:39 <nirik> so, can you describe the workflow here?
17:16:33 <sharkcz> do you ask /me or mboddu ? :-)
17:16:49 <nirik> either one. ;)
17:17:14 <nirik> do we tag all packages in a compose so we can find them later? how do we get pungi to use those?
17:18:09 * cverna waves
17:18:31 <mboddu> nirik: Its kinda explained in https://pagure.io/releng/issue/7763#comment-529648
17:19:19 <sharkcz> we tag package/builds with a "nightly" tag, then if there is a need we call pungi with pkgset_koji_tag = nightly tag and do only limited variants (one), some minimal pungi config
17:19:19 <mboddu> Inheritance is not an option, we have to clone the tag, but correct me if I am wrong, koji uses hardlinks to tag builds if a tag gets cloned, right?
17:19:41 <sharkcz> I think this is just a koji db action
17:20:01 <nirik> so we will need to adjust garbage collection here... and signing...
17:20:53 <mboddu> nirik: Definitely garbage collection, but do we need to do about signing?
17:20:59 <sharkcz> signing - probably not, because everything will be signed already (content of regular nightly)
17:21:08 <mboddu> what do we need to do about signing?*
17:21:20 <sharkcz> ack for garbage collection
17:21:27 <nirik> until it's garbage collected...
17:21:45 <nirik> signed copies could be gone...
17:22:15 <sharkcz> are they removed only post-GA?
17:22:27 <sharkcz> aren't
17:23:36 <nirik> there's a ruleset for them...
17:23:42 <sharkcz> ok
17:23:52 <nirik> I don't recall how long it keeps everything.
17:24:11 <mboddu> I dont remember the rules on top of my head, but its not "removing only post-GA"
17:24:15 <sharkcz> or we can allow unsigned rpms for such special req compose
17:24:26 <mboddu> Yeah, thats my thought
17:24:49 <mboddu> If the builds are still around, we can use "None" to the sigkeys option in pungi and create a compose
17:25:13 <nirik> so I wonder... you want this to test the last good compose against current ones?
17:26:06 <nirik> could we just save the last good compose? but I guess 'good' is reletive.. we could say the last "FINISHED" one, but we never have had a real one of those...
17:26:32 <sharkcz> not last good, but a selected compose from the past, right now I would like to try f29 before it got py 3.7
17:27:41 <sharkcz> there is a chance py 3.7 is doing something wrong on 3.7 (mem leak?)
17:27:54 <sharkcz> wrong on s390x
17:29:54 <sharkcz> eg. compose from 20180627 because on 28th f29 got py 3.7
17:30:08 <nirik> ah ok. so we can't really know in advance...
17:30:40 <nirik> so yeah, being able to recreate might work... or is at least worth a try
17:31:09 <mboddu> I can talk to brew developers and ask them how much resources this might take
17:31:10 <sharkcz> right, so the policy would pick some of the compose to have a tag for recreating in the future
17:31:13 <mboddu> Just to give us an idea
17:31:36 * mboddu hoping its just a db operation
17:31:51 <nirik> yeah, good to check on
17:32:04 <sharkcz> IMO it is, unless you have a build target with the tag
17:32:22 <mboddu> sharkcz: Yeah, build target is a whole different thing
17:32:59 <sharkcz> and internally they mark the snapshots before GA with tags, etc, so I would say it's cheap
17:33:24 <sharkcz> there are builds with dozens of tags
17:34:06 <sharkcz> or check the koji sources :-)
17:35:06 <mboddu> #info The plan is to clone the base tag based on the koji event that is stored in pungi logs. But to do so, mboddu will talk to koji developers to get a rough idea on how much resources it might take.
17:35:44 <sharkcz> ack
17:36:12 <sharkcz> what composes to tag is another part
17:36:12 <nirik> ack
17:36:40 <nirik> I think only FINISHED or INCOMPLETE?
17:37:38 <mboddu> nirik: Yes, that is what I am thinking, because, not many will care whats in a DOOMED compose, everyone will try to fix the issue
17:37:55 <sharkcz> yes, but all of them won't be necessary IMHO, so it could be the "recommended for further testing"
17:38:17 <nirik> perhaps the ones that pass tests? but I am not sure if we have any currently. ;)
17:38:51 <sharkcz> simply to have some points in the past the could be used to "bisect" issues
17:38:55 <nirik> mboddu: reminds me, I was thinking perhaps for those tests as a first step, could we provide the link(s) at the top of the email ? then people would see them/be able to look at them at least
17:39:22 <mboddu> Well, based on whats the koji developers response we can decide what compose to tag
17:39:31 <sharkcz> ack
17:40:05 <mboddu> nirik: I didn't get it, sorry
17:40:32 <mboddu> #info Based on koji developers response about resources, we will decide what composes to tag
17:40:38 <nirik> that resultsdb 'go' 'nogo' stuff we were talking about with adamw the other day?
17:41:29 * nirik looks for a url
17:42:19 <nirik> greenwave.decision.update -- greenwave says NO-GO on "something" for "rawhide_compose_sync_to_mirrors" (fedora-rawhide) https://taskotron.fedoraproject.org/resultsdb/results?productmd.compose.id=Fedora-Server-dvd-x86_64-Rawhide-20180831.n.0.iso
17:42:26 <mboddu> nirik: Okay, about pushing the compose to mirrors
17:42:31 <mboddu> Yes
17:43:02 <nirik> I think I am against using that to decide if we sync or not, unless we are pretty consistent with GO already
17:44:31 <nirik> anyhow, just thought it would be good to add to the email so people can look at it.
17:44:35 <nirik> (as a first step)
17:44:46 <mboddu> nirik: Yeah, if it says NO-GO most of the times then people wont get the updates
17:45:15 <mboddu> nirik: Sure, but that would be a different email
17:45:15 <nirik> right, we already have gaps where we are trying to fix compose issues... if we blocked more people would get even less things.
17:45:35 <mboddu> Agreed
17:46:55 <nirik> anyhow, I had 2 hopefully quick items...
17:47:14 <mboddu> nirik: Sure, go ahead
17:47:21 <mboddu> #topic Open Floor
17:48:24 <nirik> so first... there was some complaints on the devel list about fedora-packager... I see that mboddu and kellin are the only maintainers... would you all like more? ;)
17:49:03 <nirik> Dennis merged a bunch of pending pr's... so I think it's caught back up
17:49:11 <mboddu> nirik: Yeah, I saw them and Dennis picked them up
17:49:23 <mboddu> nirik: I didn't know Dennis made us as the maintainers when he left
17:49:39 <mboddu> So, I was not paying attention to it
17:49:50 <nirik> yeah... easy to do, there's so much stuff. ;(
17:49:59 <mboddu> But since I know now, I will pay more attention to it
17:51:08 <nirik> cool. Let me know if you want more maintainers. :) We may want to setup some time once a week or something to go over outstanding PR's and such on all the releng repos... I know stuff piles up and I miss things from time to time
17:51:26 <mboddu> nirik: Sure, will do, thanks
17:52:15 <nirik> Second item: I am going to try and setup some more automation for beta/final changes... I know we have a SOP, but we shouldn't have to do all this manual stuff. I'm going to start by trying to abstract out thigns in the infra ansible repo so we just need to adjust things in 1 place and let templates take care of the rest.
17:53:02 <nirik> if that works out, we can perhaps look at a ansible playbook that makes at least all the proposed changes for releng/fedora-pungi, etc... so you run the script and then it makes those changes and you can submit pr from that
17:53:23 <mboddu> nirik: Oh, that helps a lot
17:53:36 <nirik> hopefully this will save us from remembering things and let computers do the work. ;)
17:53:51 <nirik> likely I will land stuff after beta freeze and we can see how it can do for final. ;)
17:54:09 <mboddu> nirik: Thanks and I am very much interested in working on that stuff
17:54:24 <mboddu> nirik: So, let me know if you need any help
17:54:38 <nirik> yeah, help welcome. I was hoping to work with you on it.
17:54:52 <mboddu> Awesome :)
17:54:54 <mboddu> nirik++
17:55:03 <nirik> thats all I had I think...
17:56:50 <mboddu> #info nirik is working on automating some of the releng's work using ansible and mboddu is happy to help :)
17:56:53 <mboddu> thanks nirik
17:57:04 <mboddu> #topic Alternate Architectures Updates
17:57:18 <mboddu> Any update on the ppc thing?
17:57:27 * mboddu haven't looked at it in the last 2 days
17:57:32 <nirik> I think it's solved at this point
17:57:38 <dustymabe> i think it's working
17:57:39 <nirik> there's a new qemu
17:57:45 <dustymabe> yep. thanks nirik and sinny
17:57:55 <mboddu> nirik, dustymabe : Thanks guys for the update
17:58:09 <nirik> yeah, many thanks to sinny for tracking down thta one package... it was driving me crazy. ;)
17:59:08 <mboddu> I guess thats all we have for today
17:59:16 <mboddu> Thanks everyone for joining
17:59:32 <nirik> thanks for running things mboddu!
17:59:35 <mboddu> #endmeeting