15:59:02 <mboddu> #startmeeting RELENG (2019-08-01)
15:59:02 <zodbot> Meeting started Wed Jul 31 15:59:02 2019 UTC.
15:59:02 <zodbot> This meeting is logged and archived in a public location.
15:59:02 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:59:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:59:02 <zodbot> The meeting name has been set to 'releng_(2019-08-01)'
15:59:02 <mboddu> #meetingname releng
15:59:02 <zodbot> The meeting name has been set to 'releng'
15:59:02 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu ksinny jednorozec
15:59:02 <zodbot> Current chairs: jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
15:59:02 <mboddu> #topic init process
15:59:14 <nirik> morning
15:59:23 <mboddu> morning nirik
15:59:24 <smooge> mroning
15:59:33 <smooge> will be dropping off in 5 for 1x1
15:59:44 <mboddu> #chair smooge
15:59:44 <zodbot> Current chairs: jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz smooge
15:59:49 <mboddu> Oh well
16:00:01 <smooge> #unchair smooge
16:00:01 <zodbot> Current chairs: jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
16:00:06 <smooge> thanks :)
16:00:13 <mboddu> Haha :)
16:00:18 <mboddu> #topic EPEL-8
16:00:52 <mboddu> #info We did a test compose of epel8-playground and it was successful in the first try
16:01:07 * mboddu cant believe it, but sometimes miracles happen :)
16:01:56 <mboddu> nirik: On that note, do you wanna discuss https://pagure.io/fedora-comps/pull-request/396?
16:02:31 <nirik> congrats on the compose. ;)
16:02:33 <nirik> sure, looking
16:02:37 <nirik> ah right...
16:02:47 <nirik> well, my understanding of the way comps works:
16:03:11 <nirik> all the enabled repos you have, dnf/yum gets comps (if available) and merges them together.
16:03:36 <nirik> so if you do 'yum groupinstall foo' and foo is in rhel and epel8, it will merge them and install everything in both.
16:03:50 <mboddu> Right
16:03:50 <nirik> so,, it seems to me to be not needed to duplicate the rhel8 one
16:04:41 <nirik> we should just add things as they are added in epel8...
16:05:20 <nirik> ie, perhaps things like 'fedora-packager' ?
16:06:04 <nirik> or kde once it's in, etc.
16:06:14 <smooge> yes
16:06:22 <mboddu> nirik: ^ Are you sure? Lets say foo comp group has 'blah' package in rhel8 comps, but we dont specify it in fedora-comps, then when I run "yum groupinstall foo" where can it get the foo group list of packages?
16:06:58 <nirik> it gets them from the rhel8 comps... whereever the group exists?
16:08:10 <nirik> if rhel8 has a foo group with blah in it and epel8 also has a foo group with blarge in it, if you do 'yum group install foo' you will get both blah and bargle
16:08:25 <nirik> it was setup that way so rpmfusion to extend groups in fedora.
16:08:26 <mboddu> nirik: Hmmm, how do the compose (the repo) know that the group exists?
16:08:31 <nirik> or at least thats the effect.
16:08:33 * mboddu bbiab, call
16:09:18 <nirik> for composes, we don't use comps at all I don;'t think... it doesn't use them, it's only for end users.
16:09:31 <nirik> but perhaps I misunderstand... whats an example of a compose using groups?
16:09:52 * nirik runs to get coffee while mboddu is on the phone. ;)
16:12:54 <nirik> back
16:12:54 <mboddu> nirik: I thought composes does use comps in generating the repo - https://pagure.io/pungi-fedora/blob/master/f/fedora.conf#_9
16:13:04 <mboddu> Or else why would it be there?
16:13:23 * mboddu sorry for the call, it was an urgent call that I needed to take
16:13:24 <nirik> thats just it inserting the comps file into the repodata
16:13:46 <nirik> I don't think it does anything else with it... just makes it available to users in the repodata.
16:14:34 <mboddu> So, as an end user when I run "yum groupinstall" it wont check ^ this repodata?
16:14:53 <mboddu> If not, where does groupinstall look at?
16:15:48 <nirik> it will yes. along with all other enabled repos comps files
16:16:10 * nirik thinks we are not understanding each other... sorry if I am explaining poorly
16:17:25 <nirik> I don't think we should duplicate the rhel8 comps groups...
16:17:35 <nirik> because they will change them and then it will get weird.
16:17:46 <nirik> we should only have the things that we have in epel8.
16:17:50 <mboddu> nirik: I think I am getting it, the end users will have both the repos enabled (rhel and epel), not just epel
16:17:56 <nirik> right.
16:18:31 <nirik> if we shipped rhel8 comps, and someone just had epel8 enabled... it would find all those groups, but none of them would be installable.
16:19:00 <nirik> so really what we should do is look at all the built things in epel8 we have... and what fedora groups they are in, and add those groups to our comps...
16:19:01 <mboddu> nirik: Sorry, now I understood it, I was thinking only epel is enabled
16:19:15 <nirik> or just do fedora-packager and let other people add other stuff as they make it.
16:19:22 <mboddu> nirik: Sure, got it
16:19:43 <mboddu> nirik: Okay, I will make the changes
16:19:51 <mboddu> nirik: Thanks for correcting me and the explanation
16:19:55 <nirik> awesome. thanks.
16:21:36 <mboddu> #action mboddu will update the comps file with only epel8 comps groups
16:22:11 <mboddu> Another point to discuss is epel8 bodhi enablement
16:22:22 <mboddu> #topic EPEL8 Bodhi Enablement
16:22:41 <mboddu> So, I think we need to do this asap, and I am leaving on Fri
16:22:51 <mboddu> So, I am planning to work on it today in the afternoon
16:22:52 <nirik> yeah, I think we can anytime for epel8
16:23:17 <mboddu> Cool, I just wanted to check if there are any objections before I do so
16:23:25 <nirik> can't think of any...
16:23:41 <mboddu> Awesome
16:23:58 * mboddu has a lot on my plate for today :)
16:24:23 <nirik> busy busy. :)
16:24:46 <mboddu> #action mboddu will try to enable bodhi for epel8 release today
16:24:58 <mboddu> If not I will complete it before EOD tomorrow
16:25:22 <mboddu> Anything else on epel8 we need to discuss?
16:25:40 <nirik> mirrormanager needs setup...
16:25:58 <nirik> and announcement, and thats all I can think of. I think smooge was going to do those things.
16:25:58 <mboddu> Yes, smooge said he will talk to Adrian about it
16:26:32 <mboddu> nirik: On that note, I dont know when MM side of things will be done, do you think we should start syncing them before MM setup is done?
16:26:56 <mboddu> syncing them to /pub/....?
16:28:31 <nirik> we could sure...
16:28:38 <nirik> hopefully setup there is pretty easy
16:32:29 <mboddu> nirik: Okay, that means I need to update the script that epel8-playground composes to start syncing them once the compose is done
16:32:38 <nirik> yep.
16:32:51 <nirik> well, or just in the cron job like rawhide, whatever.
16:34:12 <mboddu> rawhide also does it in the script, it will generate fullfiletimelists in the cron job
16:34:42 <nirik> ok. consistent with that sounds fine...
16:35:06 <mboddu> nirik: Thinking of creating 8 and 8-playground dirs in /pub/epel/
16:35:18 <nirik> +1
16:35:24 <mboddu> Cool
16:37:05 <mboddu> nirik: https://pagure.io/fedora-comps/pull-request/396#request_diff
16:37:09 <mboddu> Updated the PR
16:37:16 <nirik> ok, can look...
16:37:36 <mboddu> thanks nirik
16:37:43 <mboddu> #topic Mass Rebuild Status
16:37:44 <nirik> does that validate? there's a check thing there... but I can look after meeting
16:40:35 * nirik has an item for open floor... but has to take a dog out right now.
16:42:18 <mboddu> nirik: Sure
16:42:30 <mboddu> I just want to give the status on mass rebuild
16:42:50 <mboddu> #info mass rebuild for rpms are done and all the builds are merged into main tag
16:43:00 <mboddu> #info mboddu started filing ftbfs today
16:43:18 <mboddu> Although it failed a few min, I need to look into it
16:43:49 <mboddu> nirik: FYI -https://pagure.io/fedora-infrastructure/issue/8048#comment-585573
16:44:12 <mboddu> we can look at it later, but just wanted to let you know what happened and the ticket associated with it
16:45:40 <mboddu> ftbfs failed with:
16:45:43 <mboddu> "requests.exceptions.HTTPError: 502 Server Error: Proxy Error for url: https://bugzilla.redhat.com/xmlrpc.cgi"
16:46:20 <mboddu> So I am guessing a random network error, I will rerun it
16:46:55 <mboddu> #topic Open Floor
16:47:11 <mboddu> nirik: You wanna go?
16:47:42 <mboddu> #info There wont be a meeting next week as most of the people might be traveling and I will be on PTO
16:48:20 <nirik> yeah, so fedora_koji space.
16:48:32 <nirik> we need to free up what we can and start moving things to other volumes.
16:49:06 <nirik> do we need the 27 28 dirs here: https://kojipkgs.fedoraproject.org/compose/
16:49:09 <mboddu> nirik: Okay, I never played with change-build-volume, but I am happy to help
16:49:28 <nirik> ok. I think I have a good handle on what we need to do... just need to make time to do it. :)
16:49:51 <nirik> but the storage we have is reaching it's limit... they can't fit a larger than 75TB volume anywhere.
16:49:56 <mboddu> nirik: We dont need them, I will clean them up along with some others on mirrors
16:50:17 <nirik> also we might move ostree to it's own volume
16:50:24 <mboddu> +1
16:50:41 <nirik> that might not be too much space tho
16:51:23 <mboddu> Can we do split koji as well along with this clean up?
16:51:32 <nirik> split koji?
16:51:56 <nirik> you mean move builds to other volumes, yes, I think we will have to do that
16:52:21 <mboddu> nirik: Yes, so that the recent builds are on a faster storage
16:52:50 <mboddu> nirik: https://pagure.io/releng/issue/6805 split fedora_koji
16:53:08 <nirik> sure, but ideally we just leave the 'active' stuff unmoved... and then that entire volume can be put where it is best
16:53:24 <nirik> yeah, I filed that ticket. ;)
16:53:32 <mboddu> Okay and yup :)
16:54:06 <nirik> my plan was still to at least start with volumes per release, and see whats left at the end thats not active
16:54:11 <mboddu> #info nirik will be moving/cleaning some storage stuff around, mboddu will be helping him in anyway possible
16:54:21 <mboddu> ack
16:54:37 <smooge> back
16:55:01 <mboddu> Almost done smooge :(
16:55:39 <ksinny> wanted to add that F29 AH imagebuild on ppc64le composes are running successfully since 20190724 . This is realted to isue https://pagure.io/releng/issue/8407
16:55:40 <mboddu> smooge: But, I need to work on bodhi epel8 enablement and start syncing the composes to /pub/epel/8 and /pub/epel/8-playground dirs
16:55:41 <smooge> mboddu, also see if we can remove koji/rhel?
16:55:49 <ksinny> Thanks nirik for that
16:56:21 <nirik> ksinny: glad to hear it. if thats solved we could close the bug.
16:56:22 <smooge> mboddu, so currently we are on the fastest storage. we are actually going to be moving old stuff to slower storage
16:56:31 <mboddu> Thanks to nirik that our composes are so fast now, 4:30 hrs, yaya
16:57:08 <ksinny> nirik: did you do any additional changes  on those builders or adding additional memmory and cpu was the trick?
16:57:09 <mboddu> smooge: Okay, thanks for the info, I thought we were always on slower storage
16:57:31 <ksinny> nirik++
16:57:31 <zodbot> ksinny: Karma for kevin changed to 26 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:57:32 <nirik> ksinny: cache mode unsafe made them about 10x faster
16:57:38 <smooge> no they moved us to faster at one point because we were eating so much IOPS on slow
16:57:39 <nirik> mboddu: nope...
16:57:50 <ksinny> nirik: nice
16:58:01 <mboddu> smooge, nirik : Okay
16:58:07 <ksinny> nirik: I will add that in comment and will close the ticket for now
16:58:13 <mboddu> Thanks ksinny
16:58:26 <ksinny> np :)
16:58:41 <mboddu> Okay, thanks everyone for joining
16:58:48 <mboddu> See you all next week at FLOCK
16:58:52 <mboddu> Except smooge :P
16:59:01 <ksinny> yay, see you all at Flock
16:59:03 <smooge> yep. I will be tending the fires
16:59:15 <ksinny> smooge:  we will miss you
16:59:19 <smooge> making sure the milk doesn't burn
16:59:33 <nirik> stir that milk
16:59:49 <mboddu> Until it becomes butter :D
16:59:51 <nirik> thanks mboddu
16:59:53 <mboddu> #endmeeting