16:04:10 <mboddu> #startmeeting RELENG (2019-10-31) 16:04:10 <zodbot> Meeting started Wed Oct 30 16:04:10 2019 UTC. 16:04:10 <zodbot> This meeting is logged and archived in a public location. 16:04:10 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:04:10 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:04:10 <zodbot> The meeting name has been set to 'releng_(2019-10-31)' 16:04:10 <mboddu> #meetingname releng 16:04:10 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec 16:04:10 <mboddu> #topic init process 16:04:10 <zodbot> The meeting name has been set to 'releng' 16:04:10 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz 16:05:19 <nirik> morning 16:05:37 <mizdebsk> hello 16:05:47 <mboddu> Hello everyone 16:06:22 <mboddu> #topic #8865 create tags for EPEL-8 module builds in staging 16:06:29 <mboddu> #link https://pagure.io/releng/issue/8865 16:06:31 <mboddu> So... 16:06:55 <mboddu> https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/roles/mbs?id=d2fd1ce6e4a640f1f178f8c34e3259225ad635e6 16:07:08 <mboddu> nirik, mizdebsk : ^ if you can confirm that, I can run the playbook 16:07:28 <nirik> I don't know much about mbs... but that looks ok at a quick glance. 16:07:32 <mboddu> And, while merlinm is getting prepared for modules in epel8 16:07:43 <dustymabe> hello 16:07:52 <mboddu> He filed - https://pagure.io/fedora-infrastructure/issue/8334 16:07:59 * mboddu waves at dustymabe 16:09:55 <nirik> yeah, we need to do a sync, but... 16:10:15 <mizdebsk> the modulemd look good to me 16:10:54 <nirik> not sure how to do that without disrupting people using it now... likely we will have to re-add some tags/etc... and also... we need a good backup to sync over. 16:11:17 <mizdebsk> groups could probably be cleaned up, but that is minor 16:11:54 <mizdebsk> eg. i don't think that fedpkg-minimal should be in build group, only in srpm-build 16:12:36 <mboddu> mizdebsk: We could change it and thanks 16:12:39 <mboddu> for finding it 16:12:51 <mboddu> nirik: When do you think we could be ready for it? 16:13:59 <nirik> well, I need to try and look at the backup situation again.... once thats sorted or made better we could do the sync anytime 16:14:09 <mizdebsk> koji sync takes time, especially when there are db schema changes - sql script needs to be updated and that may require several db syncs 16:14:33 <nirik> yeah 16:14:37 <mizdebsk> in order not to disrupt people testing it would be good to spin a second db (db-koji02.stg) and sync db there 16:14:56 <mizdebsk> once db sync is ready we could quickly re-point koji hub at the newly-synced db 16:15:03 <mizdebsk> that would at least reduce stg koji downtime 16:15:07 <nirik> sure, still needs a good backup tho. 16:15:34 <mizdebsk> and that too, but i thought that replication fixed backups, didn't it? 16:15:37 <mboddu> Yeah, if there is schema change its a headache and backup is still a problem 16:15:59 <nirik> mizdebsk: sadly not... sometimes they finish, sometimes not. 16:18:06 <mizdebsk> we could try to take advantage of lvm snapshots for backups 16:18:40 <nirik> thats not going to be consistent... (the db part anyhow) 16:18:55 <mizdebsk> nirik, what do you mean? 16:19:00 <nirik> I was going to look at disconnecting the replica, backing up and reconnecting. 16:19:41 <nirik> the db may have things it's not yet commited to disk? but I guess it would be consistant... 16:20:02 <mizdebsk> if you take snapshot of filesystem then you can run a different postgresql server on the snapshot and take sql dump out of it 16:20:18 <mizdebsk> due to ACID the data is always in consistent state 16:20:37 <nirik> or just copy off the data right? no need to dump/restore? 16:21:00 <mizdebsk> sure 16:21:36 <nirik> there's some upstream changes that hopefully will help and make backups work ok again... but not sure when they will land. 16:23:13 <nirik> anyhow, if we want to do a stg sync we could I guess do that anytime with a lvm snapshot -> rsync -> 16:23:14 <mboddu> #info we are still facing backup problems, so it might take some time before we sync koji prod to stg 16:23:28 <nirik> but we may want to just wait until the rawhide gating testing is done? 16:24:01 <mboddu> Or collaborate with pingou 16:24:22 <mboddu> nirik: Who is driving epel8 modules task? 16:25:24 <nirik> I think merlinm and sgallagh were working on that, I am not sure if there's others... 16:25:54 <sgallagh> Mostly merlinm at the moment. I've been dealing with other things 16:26:03 <mboddu> Thanks sgallagh 16:26:52 <mboddu> nirik: We sorta promised modules in epel8 when epel8 will be based off of rhel 8.1, so... 16:27:10 <mboddu> I am just worried about that 16:27:49 <nirik> sure. we could do it sooner I guess... just need to check with pingou about impact on them.... 16:28:20 <mboddu> Okay 16:28:23 <mboddu> For now: 16:28:57 <mboddu> #info epel8 and epel8-playground platform module definitions are looking fine, mboddu will run the playbook and push it to stg 16:29:03 <sgallagh> Thanks 16:29:14 <mboddu> Moving on 16:29:27 <mboddu> #topic #8949 repo_include_all does not work with external repos 16:29:33 <mboddu> #link https://pagure.io/releng/issue/8949 16:30:06 <mboddu> mizdebsk: ^ 16:30:35 <mizdebsk> this one has been blocking me from doing module development for past 4 months, so i'd appreciate if it could be resolved 16:30:56 <mizdebsk> originally i filed an issue for MBS, but it turned out that koji tag configuration needs to be updated 16:31:19 <mizdebsk> i updated it in staging koji and tested, it resolves the issue 16:31:20 <nirik> huh. 16:32:11 <nirik> whats module-f31-build ? I don't see it in prod 16:32:27 <mboddu> nirik: Its a tag 16:32:41 <mboddu> With external repo pointing to https://kojipkgs.fedoraproject.org/repos/f31-build/latest/$arch/ 16:32:51 <nirik> ah, misreading it... 16:33:08 <mboddu> And for some reason they named the external repo as f31-build 16:33:21 <mizdebsk> module-f31-build is a koji tag that defines base buildroot for platform:f31 modules 16:33:35 <mizdebsk> it's basically a tag that adds contents of f31-build as external repo 16:33:50 <nirik> does this issue only affect f31? 16:33:58 <mizdebsk> no, it affects all platforms 16:34:06 <nirik> should we fix them all then? 16:34:18 <mboddu> Which is what my next question 16:34:26 <mboddu> May be at least rawhide 16:34:32 <mboddu> But I am dont know about others 16:34:38 <mizdebsk> jkaluza (MBS developer) wanted to only change one platform for now, so that it could be tested more safely 16:34:47 <mizdebsk> but i'm fine with changing all platforms 16:34:58 <mboddu> mizdebsk: Then is it okay if we change with f32(rawhide) ? 16:35:06 <mboddu> Rather than f31 16:35:30 <mizdebsk> the problem with f32 is that it doesn't use bodhi; modules are tagged into f32-modular and shipped to users 16:35:34 <mizdebsk> there is no way i can untag them 16:35:59 <mboddu> Hmmm 16:36:03 <mizdebsk> f31 uses bodhi therefore modules are not automatically shipped to users without packager submitting update 16:36:43 <nirik> ok, so lets do f31 now, test and do the rest next week? 16:36:51 <mizdebsk> +1 from me 16:37:41 <nirik> those commands don't look right tho to me... shouldn't that have a url where module-f31-build is? 16:37:55 <mboddu> I wanted to say, we could try it on f29, since its gonna EOL in 4 weeks, but may be there are more people on f29 compared to f31 16:37:59 <mizdebsk> nirik, no, module-f31-build is a tag, not external repo 16:38:12 <mboddu> mizdebsk: External repo is missing the url 16:38:29 <nirik> yeah, I guess the help info is missing that a tag is ok there too. 16:38:35 <mizdebsk> i used the same commands in stg koji and they worked 16:38:53 <nirik> f31-build is currently: f31-build https://kojipkgs.fedoraproject.org/repos/f31-build/latest/$arch/ 16:39:10 <mboddu> mizdebsk: Huh, then where is the external repo pointing to? 16:39:15 <mizdebsk> the first command removes external repo from tag, it doesn't remove external repo itself 16:39:31 <mizdebsk> then second command re-adds the external repo to the tag, with different merge mode 16:39:40 <mizdebsk> url of the repo is the same as now, it doesn't change 16:39:42 <mboddu> mizdebsk: Ahhh, right, sorry 16:40:29 <mboddu> So, last question, should we try it on f29 or f31? 16:40:42 <nirik> the koji interface around this isn't great... but ok, yeah... 16:41:16 <nirik> --used helps 16:42:52 <mizdebsk> i'm fine with either of them; i'd suggest f31 because changes are generally first done in later releases then backported to older ones 16:43:11 <mboddu> Okay 16:43:29 <mboddu> #info We will try it on f31 first and if everything is fine, then we will try it on other releases 16:44:01 <mboddu> #topic F31 checksums 16:44:15 <mboddu> So, nirik - https://pagure.io/fedora-websites/pull-request/994 16:44:39 <mboddu> But, dustymabe pointed that checksums for x86_64 are missing from alt.fp.o 16:44:47 <mboddu> But x86_64 is not alt 16:45:05 <nirik> no idea... we can ask relrod when he's around. or in #fedora-websites 16:45:06 <mboddu> But I couldn't find verify.html for getfedora.org 16:45:29 <mboddu> Okay, just wondering if you would know 16:46:01 <mboddu> Okay, moving on 16:46:17 <mboddu> #topic Open Floor 16:46:27 <mboddu> #info f31 is out yesterday, yay 16:46:30 <mboddu> Good job everyone 16:46:39 <nirik> yeah, great work! 16:46:47 <nirik> do we want to talk backlog for a few? 16:46:57 <mboddu> Yes 16:47:16 <nirik> can we close all the f31 change tickets now? 16:47:18 <mboddu> #info mboddu is working on Fedora Minimal Compose and got fedora-messaging working 16:47:28 <mboddu> nirik: Yup, I will go through them 16:47:50 <nirik> yeah, whats the status on minimal composes? 16:48:08 <mboddu> nirik: On fedora minimal compose, what do you think the best way to check if a compose is running or not? 16:48:30 <mboddu> nirik: I haven't worked on it at all last week - spent most of the time on F31 release 16:49:03 <nirik> yeah, thats fair. 16:49:29 <nirik> so whats the current design? just calling pungi? or using odcs? or ? 16:49:38 <mizdebsk> if you used odcs, odcs would know whether compose is running and you could check odcs api 16:49:52 <mboddu> mizdebsk: Ahh cool 16:50:02 <mboddu> nirik: I am thinking of odcs, but I didn't know about ^^ 16:50:28 <nirik> it would be nice if we could use it... but I don't know if it gives us what we need... 16:51:20 <mizdebsk> odcs can accept raw pungi config 16:51:38 <nirik> cool. that should be workable then 16:51:57 <mboddu> Yeah, it was one of the things I asked when ODCS was initially launched 16:53:17 <mizdebsk> with odcs the compose will reside on odcs host, not in /mnt/koji; i guess that's not a problem 16:55:03 <mboddu> mizdebsk: Hmmm, may be, or we could add a redirect from /mnt/koji/compose/<minimal-composes> to odcs host? 16:55:43 <nirik> as long as we can downlload the artifacts I don't think it matters 16:55:48 <mizdebsk> or make kojipkgs serve odcs data, since it will be on nfs now 16:57:03 <nirik> sure 16:57:25 <mboddu> Yeah, as what nirik said, if we can access it publicly, thats all I care about 16:57:50 <mboddu> Thats all I got 16:57:58 <mboddu> Anything else? 16:59:14 <mboddu> mizdebsk: I just completed your ticket for f31 16:59:23 <mizdebsk> thanks a lot! 16:59:44 <mizdebsk> i'll give it a try later today and report back in the ticket 16:59:55 <nirik> rawhide seems to be failing on cloud images... :( 17:01:00 <nirik> oh, FYI, I am going to look at moving builders to f31 over the next few weeks... 17:01:33 <mboddu> #info nirik will be looking at moving the builders to f31 in the next few weeks 17:02:00 <dustymabe> i think rawhide was recently building the cloud images, but I notice they don't boot 17:02:00 <mboddu> nirik: We should move them to rawhide, no need of upgrades, just updates :) 17:02:02 * mboddu hides 17:02:52 <nirik> great plan. ;) 17:03:08 <mboddu> I think rawhide failed on cloud just today 17:03:14 <nirik> dustymabe: well, ppc64le/aarch64 was broken until yesterday/today 17:03:59 <dustymabe> if anyone wants to go in and find out why it's not booting that would be great :) 17:04:26 * relrod looks up and sees he was pinged. Sorry, was in an internal meeting. 17:04:41 <mboddu> relrod: no problem 17:04:50 * nirik is doing that now. ;) 17:04:58 <mboddu> Thanks nirik 17:05:18 <mboddu> Let move to #fedora-releng as we exceeded our time 17:05:22 <mboddu> #endmeeting