16:04:10 <mboddu> #startmeeting RELENG (2019-10-31)
16:04:10 <zodbot> Meeting started Wed Oct 30 16:04:10 2019 UTC.
16:04:10 <zodbot> This meeting is logged and archived in a public location.
16:04:10 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:04:10 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:04:10 <zodbot> The meeting name has been set to 'releng_(2019-10-31)'
16:04:10 <mboddu> #meetingname releng
16:04:10 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec
16:04:10 <mboddu> #topic init process
16:04:10 <zodbot> The meeting name has been set to 'releng'
16:04:10 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
16:05:19 <nirik> morning
16:05:37 <mizdebsk> hello
16:05:47 <mboddu> Hello everyone
16:06:22 <mboddu> #topic #8865 create tags for EPEL-8 module builds in staging
16:06:29 <mboddu> #link https://pagure.io/releng/issue/8865
16:06:31 <mboddu> So...
16:06:55 <mboddu> https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/roles/mbs?id=d2fd1ce6e4a640f1f178f8c34e3259225ad635e6
16:07:08 <mboddu> nirik, mizdebsk : ^ if you can confirm that, I can run the playbook
16:07:28 <nirik> I don't know much about mbs... but that looks ok at a quick glance.
16:07:32 <mboddu> And, while merlinm is getting prepared for modules in epel8
16:07:43 <dustymabe> hello
16:07:52 <mboddu> He filed - https://pagure.io/fedora-infrastructure/issue/8334
16:07:59 * mboddu waves at dustymabe
16:09:55 <nirik> yeah, we need to do a sync, but...
16:10:15 <mizdebsk> the modulemd look good to me
16:10:54 <nirik> not sure how to do that without disrupting people using it now... likely we will have to re-add some tags/etc... and also... we need a good backup to sync over.
16:11:17 <mizdebsk> groups could probably be cleaned up, but that is minor
16:11:54 <mizdebsk> eg. i don't think that fedpkg-minimal should be in build group, only in srpm-build
16:12:36 <mboddu> mizdebsk: We could change it and thanks
16:12:39 <mboddu> for finding it
16:12:51 <mboddu> nirik: When do you think we could be ready for it?
16:13:59 <nirik> well, I need to try and look at the backup situation again.... once thats sorted or made better we could do the sync anytime
16:14:09 <mizdebsk> koji sync takes time, especially when there are db schema changes - sql script needs to be updated and that may require several db syncs
16:14:33 <nirik> yeah
16:14:37 <mizdebsk> in order not to disrupt people testing it would be good to spin a second db (db-koji02.stg) and sync db there
16:14:56 <mizdebsk> once db sync is ready we could quickly re-point koji hub at the newly-synced db
16:15:03 <mizdebsk> that would at least reduce stg koji downtime
16:15:07 <nirik> sure, still needs a good backup tho.
16:15:34 <mizdebsk> and that too, but i thought that replication fixed backups, didn't it?
16:15:37 <mboddu> Yeah, if there is schema change its a headache and backup is still a problem
16:15:59 <nirik> mizdebsk: sadly not... sometimes they finish, sometimes not.
16:18:06 <mizdebsk> we could try to take advantage of lvm snapshots for backups
16:18:40 <nirik> thats not going to be consistent... (the db part anyhow)
16:18:55 <mizdebsk> nirik, what do you mean?
16:19:00 <nirik> I was going to look at disconnecting the replica, backing up and reconnecting.
16:19:41 <nirik> the db may have things it's not yet commited to disk? but I guess it would be consistant...
16:20:02 <mizdebsk> if you take snapshot of filesystem then you can run a different postgresql server on the snapshot and take sql dump out of it
16:20:18 <mizdebsk> due to ACID the data is always in consistent state
16:20:37 <nirik> or just copy off the data right? no need to dump/restore?
16:21:00 <mizdebsk> sure
16:21:36 <nirik> there's some upstream changes that hopefully will help and make backups work ok again... but not sure when they will land.
16:23:13 <nirik> anyhow, if we want to do a stg sync we could I guess do that anytime with a lvm snapshot -> rsync ->
16:23:14 <mboddu> #info we are still facing backup problems, so it might take some time before we sync koji prod to stg
16:23:28 <nirik> but we may want to just wait until the rawhide gating testing is done?
16:24:01 <mboddu> Or collaborate with pingou
16:24:22 <mboddu> nirik: Who is driving epel8 modules task?
16:25:24 <nirik> I think merlinm and sgallagh were working on that, I am not sure if there's others...
16:25:54 <sgallagh> Mostly merlinm at the moment. I've been dealing with other things
16:26:03 <mboddu> Thanks sgallagh
16:26:52 <mboddu> nirik: We sorta promised modules in epel8 when epel8 will be based off of rhel 8.1, so...
16:27:10 <mboddu> I am just worried about that
16:27:49 <nirik> sure. we could do it sooner I guess... just need to check with pingou about impact on them....
16:28:20 <mboddu> Okay
16:28:23 <mboddu> For now:
16:28:57 <mboddu> #info epel8 and epel8-playground platform module definitions are looking fine, mboddu will run the playbook and push it to stg
16:29:03 <sgallagh> Thanks
16:29:14 <mboddu> Moving on
16:29:27 <mboddu> #topic #8949 repo_include_all does not work with external repos
16:29:33 <mboddu> #link https://pagure.io/releng/issue/8949
16:30:06 <mboddu> mizdebsk: ^
16:30:35 <mizdebsk> this one has been blocking me from doing module development for past 4 months, so i'd appreciate if it could be resolved
16:30:56 <mizdebsk> originally i filed an issue for MBS, but it turned out that koji tag configuration needs to be updated
16:31:19 <mizdebsk> i updated it in staging koji and tested, it resolves the issue
16:31:20 <nirik> huh.
16:32:11 <nirik> whats module-f31-build ? I don't see it in prod
16:32:27 <mboddu> nirik: Its a tag
16:32:41 <mboddu> With external repo pointing to https://kojipkgs.fedoraproject.org/repos/f31-build/latest/$arch/
16:32:51 <nirik> ah, misreading it...
16:33:08 <mboddu> And for some reason they named the external repo as f31-build
16:33:21 <mizdebsk> module-f31-build is a koji tag that defines base buildroot for platform:f31 modules
16:33:35 <mizdebsk> it's basically a tag that adds contents of f31-build as external repo
16:33:50 <nirik> does this issue only affect f31?
16:33:58 <mizdebsk> no, it affects all platforms
16:34:06 <nirik> should we fix them all then?
16:34:18 <mboddu> Which is what my next question
16:34:26 <mboddu> May be at least rawhide
16:34:32 <mboddu> But I am dont know about others
16:34:38 <mizdebsk> jkaluza (MBS developer) wanted to only change one platform for now, so that it could be tested more safely
16:34:47 <mizdebsk> but i'm fine with changing all platforms
16:34:58 <mboddu> mizdebsk: Then is it okay if we change with f32(rawhide) ?
16:35:06 <mboddu> Rather than f31
16:35:30 <mizdebsk> the problem with f32 is that it doesn't use bodhi; modules are tagged into f32-modular and shipped to users
16:35:34 <mizdebsk> there is no way i can untag them
16:35:59 <mboddu> Hmmm
16:36:03 <mizdebsk> f31 uses bodhi therefore modules are not automatically shipped to users without packager submitting update
16:36:43 <nirik> ok, so lets do f31 now, test and do the rest next week?
16:36:51 <mizdebsk> +1 from me
16:37:41 <nirik> those commands don't look right tho to me... shouldn't that have a url where module-f31-build is?
16:37:55 <mboddu> I wanted to say, we could try it on f29, since its gonna EOL in 4 weeks, but may be there are more people on f29 compared to f31
16:37:59 <mizdebsk> nirik, no, module-f31-build is a tag, not external repo
16:38:12 <mboddu> mizdebsk: External repo is missing the url
16:38:29 <nirik> yeah, I guess the help info is missing that a tag is ok there too.
16:38:35 <mizdebsk> i used the same commands in stg koji and they worked
16:38:53 <nirik> f31-build is currently: f31-build                 https://kojipkgs.fedoraproject.org/repos/f31-build/latest/$arch/
16:39:10 <mboddu> mizdebsk: Huh, then where is the external repo pointing to?
16:39:15 <mizdebsk> the first command removes external repo from tag, it doesn't remove external repo itself
16:39:31 <mizdebsk> then second command re-adds the external repo to the tag, with different merge mode
16:39:40 <mizdebsk> url of the repo is the same as now, it doesn't change
16:39:42 <mboddu> mizdebsk: Ahhh, right, sorry
16:40:29 <mboddu> So, last question, should we try it on f29 or f31?
16:40:42 <nirik> the koji interface around this isn't great... but ok, yeah...
16:41:16 <nirik> --used helps
16:42:52 <mizdebsk> i'm fine with either of them; i'd suggest f31 because changes are generally first done in later releases then backported to older ones
16:43:11 <mboddu> Okay
16:43:29 <mboddu> #info We will try it on f31 first and if everything is fine, then we will try it on other releases
16:44:01 <mboddu> #topic F31 checksums
16:44:15 <mboddu> So, nirik - https://pagure.io/fedora-websites/pull-request/994
16:44:39 <mboddu> But, dustymabe pointed that checksums for x86_64 are missing from alt.fp.o
16:44:47 <mboddu> But x86_64 is not alt
16:45:05 <nirik> no idea... we can ask relrod when he's around. or in #fedora-websites
16:45:06 <mboddu> But I couldn't find verify.html for getfedora.org
16:45:29 <mboddu> Okay, just wondering if you would know
16:46:01 <mboddu> Okay, moving on
16:46:17 <mboddu> #topic Open Floor
16:46:27 <mboddu> #info f31 is out yesterday, yay
16:46:30 <mboddu> Good job everyone
16:46:39 <nirik> yeah, great work!
16:46:47 <nirik> do we want to talk backlog for a few?
16:46:57 <mboddu> Yes
16:47:16 <nirik> can we close all the f31 change tickets now?
16:47:18 <mboddu> #info mboddu is working on Fedora Minimal Compose and got fedora-messaging working
16:47:28 <mboddu> nirik: Yup, I will go through them
16:47:50 <nirik> yeah, whats the status on minimal composes?
16:48:08 <mboddu> nirik: On fedora minimal compose, what do you think the best way to check if a compose is running or not?
16:48:30 <mboddu> nirik: I haven't worked on it at all last week - spent most of the time on F31 release
16:49:03 <nirik> yeah, thats fair.
16:49:29 <nirik> so whats the current design? just calling pungi? or using odcs? or ?
16:49:38 <mizdebsk> if you used odcs, odcs would know whether compose is running and you could check odcs api
16:49:52 <mboddu> mizdebsk: Ahh cool
16:50:02 <mboddu> nirik: I am thinking of odcs, but I didn't know about ^^
16:50:28 <nirik> it would be nice if we could use it... but I don't know if it gives us what we need...
16:51:20 <mizdebsk> odcs can accept raw pungi config
16:51:38 <nirik> cool. that should be workable then
16:51:57 <mboddu> Yeah, it was one of the things I asked when ODCS was initially launched
16:53:17 <mizdebsk> with odcs the compose will reside on odcs host, not in /mnt/koji; i guess that's not a problem
16:55:03 <mboddu> mizdebsk: Hmmm, may be, or we could add a redirect from /mnt/koji/compose/<minimal-composes> to odcs host?
16:55:43 <nirik> as long as we can downlload the artifacts I don't think it matters
16:55:48 <mizdebsk> or make kojipkgs serve odcs data, since it will be on nfs now
16:57:03 <nirik> sure
16:57:25 <mboddu> Yeah, as what nirik said, if we can access it publicly, thats all I care about
16:57:50 <mboddu> Thats all I got
16:57:58 <mboddu> Anything else?
16:59:14 <mboddu> mizdebsk: I just completed your ticket for f31
16:59:23 <mizdebsk> thanks a lot!
16:59:44 <mizdebsk> i'll give it a try later today and report back in the ticket
16:59:55 <nirik> rawhide seems to be failing on cloud images... :(
17:01:00 <nirik> oh, FYI, I am going to look at moving builders to f31 over the next few weeks...
17:01:33 <mboddu> #info nirik will be looking at moving the builders to f31 in the next few weeks
17:02:00 <dustymabe> i think rawhide was recently building the cloud images, but I notice they don't boot
17:02:00 <mboddu> nirik: We should move them to rawhide, no need of upgrades, just updates :)
17:02:02 * mboddu hides
17:02:52 <nirik> great plan. ;)
17:03:08 <mboddu> I think rawhide failed on cloud just today
17:03:14 <nirik> dustymabe: well, ppc64le/aarch64 was broken until yesterday/today
17:03:59 <dustymabe> if anyone wants to go in and find out why it's not booting that would be great :)
17:04:26 * relrod looks up and sees he was pinged. Sorry, was in an internal meeting.
17:04:41 <mboddu> relrod: no problem
17:04:50 * nirik is doing that now. ;)
17:04:58 <mboddu> Thanks nirik
17:05:18 <mboddu> Let move to #fedora-releng as we exceeded our time
17:05:22 <mboddu> #endmeeting