15:03:42 <mboddu> #startmeeting RELENG (2020-04-14) 15:03:42 <zodbot> Meeting started Tue Apr 14 15:03:42 2020 UTC. 15:03:42 <zodbot> This meeting is logged and archived in a public location. 15:03:42 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:03:42 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:03:42 <zodbot> The meeting name has been set to 'releng_(2020-04-14)' 15:03:43 <mboddu> #meetingname releng 15:03:43 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec 15:03:43 <mboddu> #topic init process 15:03:43 <zodbot> The meeting name has been set to 'releng' 15:03:43 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz 15:03:51 <nirik> morning 15:04:14 <mboddu> Morning 15:04:18 <sharkcz> hello 15:04:25 * mboddu waves at sharkcz 15:04:50 * sharkcz waves back :-) 15:05:07 <mboddu> Let me start with open floor today as I have things to discuss there 15:05:16 <mboddu> Or not 15:05:22 <mboddu> #topic FMC-ODCS 15:05:37 <mboddu> nirik: I need some help here 15:05:50 <nirik> FMC-ODCS? 15:06:34 <sharkcz> I like acronyms ;-) 15:07:03 <mboddu> Fedora Minimal Compose - using ODCS :D 15:07:11 <nirik> ah, ok. 15:07:35 <mboddu> Well, technicall its just FMC, but the problem is with odcs and its permissions 15:07:51 <mboddu> The latest error is https://koji.fedoraproject.org/koji/taskinfo?taskID=43385938 15:08:07 <mboddu> Which means runroot requires access to /srv/odcs/ 15:08:30 <mboddu> It seems it is configured in https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_builder/templates/runroot.conf.j2 15:08:51 <mboddu> I never touched it before, so here is what I am thinking that needs changes 15:09:33 <mboddu> 1. Adding /srv/odcs to default mounts in https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_builder/templates/runroot.conf.j2#n8 15:10:03 <mboddu> 2. Adding a path like in https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_builder/templates/runroot.conf.j2#n39 15:10:29 <mboddu> I dont think we need any path substitutions or safe roots(which I have no idea what they means) 15:10:33 <mboddu> Is my understanding correct? 15:10:54 <mboddu> And, are there any considerations before we can apply this change? 15:10:59 <nirik> yeah, I think it would need to be added... but how did it work before? 15:11:11 <nirik> well, we are in freeze, it would definitely need a freeze break 15:11:54 <nirik> I guess odcs didn't do any images before? 15:11:54 <mboddu> nirik: It never worked, this is the first odcs compose requiring runroot stuff 15:11:59 <mboddu> Yes 15:12:21 <nirik> so for repos/etc it just does them like normal builds... but images it needs runroot, I assume 15:12:52 <mboddu> That is correct 15:13:32 <nirik> ok, then yeah... would need the above, possibly also changing exports on the /srv/odcs to make sure it can be mounted 15:13:57 <mboddu> Okay, I dont know how to set it up 15:15:33 <nirik> I'm not sure the exact syntax there either... would have to look at it more closely. 15:17:27 <mboddu> Okay, do you want me set 15 min aside to look at it later today? 15:17:39 <mboddu> Or it might take more than that? 15:18:58 <nirik> sure, we can look at it outside meeting, etc... shouldn't take long. I wish we could test it in stg. 15:20:33 <mboddu> I wish that too, but provided that we only have few days with odcs releng backend, I didn't start with stg as I didn't want to go through all the issues with stg and waste the little time I had 15:21:11 <nirik> sure 15:21:34 <mboddu> #info nirik and mboddu will meet later today to go over this 15:21:45 <mboddu> #topic F32 RC compose 15:22:35 <mboddu> By the time RC 1.1 failed, I was asleep and luckily (or unluckily for me) I got up at 4am in the morning and was able to get a hold of tkopecek 15:23:08 <nirik> I saw it before I went to bed, but was confused by it. ;) 15:23:08 <mboddu> He is not sure why it failed with that specific error, but he asked me rerun the compose 15:23:32 <mboddu> RC 1.2 seems to be fine and https://koji.fedoraproject.org/koji/buildinfo?buildID=1493304 seems to worked fine 15:23:37 <mboddu> nirik: Yeah, it was odd 15:23:54 <mboddu> I created a koji ticket to look into this issue - https://pagure.io/koji/issue/2145 15:24:06 <nirik> might be some kind of race condition 15:24:12 <nirik> that we normally don't hit 15:24:33 <mboddu> So, he asked me look at the logs for all the builders in livemedia channel and see if any other builder did something wonky 15:25:00 <mboddu> Now my question, is there any easy way to do it or just manually ssh'ing into those boxes and looking for it manually? 15:25:56 <mboddu> And here's the latest news on it - https://pagure.io/releng/issue/9403#comment-642069 :( 15:26:06 <nirik> log01 has all logs from everything... look in /var/log/merged/ 15:27:23 <nirik> fun 15:28:57 <mboddu> Okay, thanks 15:30:10 <mboddu> Hmmm 15:30:11 <mboddu> $ ls /var/log/merged/ 15:30:12 <mboddu> apps.log boot.log cron.log.1.gz kernel.log mail.log.1.gz secure.log 15:30:12 <mboddu> apps.log.1.gz boot.log.1.gz http_error.log kernel.log.1.gz messages.log secure.log.1.gz 15:30:12 <mboddu> audit.log.1.gz cron.log http_error.log.1.gz mail.log messages.log.1.gz 15:30:19 <mboddu> Is it part of apps.log? 15:30:35 * mboddu didn't grep it yet, but I thought I would see kojid.log or something 15:30:36 <nirik> no, messages 15:31:40 <mboddu> Ah cool 15:31:41 <mboddu> Thanks 15:32:07 <mboddu> greping for it in the whole dir would take a lot of time, this helps a lot 15:32:57 <mboddu> #action mboddu will go through the logs and update the koji ticket 15:32:57 <nirik> hopefully it would be there. thats all syslog... 15:33:16 <mboddu> Well, we will do whatever we can :) 15:37:07 <mboddu> #topic Alternate Arch Updates 15:37:14 <mboddu> sharkcz: Any updates? 15:37:39 <sharkcz> no, everything looks good 15:37:57 <sharkcz> I wrote http://fedora.danny.cz/check-compose to check if we have all artifacts 15:38:23 <sharkcz> so I don't have to look "manually" at https://kojipkgs.fedoraproject.org/compose 15:38:52 <mboddu> sharkcz: You could have used https://pagure.io/releng/failed-composes/issues 15:39:15 <sharkcz> the email reports show the changes, not the state 15:39:37 <sharkcz> mboddu: it still needs some digging to see what exactly is missing 15:39:48 <nirik> sharkcz: have you seen any srpm failures on that builder? I haven't looked recently... 15:40:07 <sharkcz> nirik: have seen it myself, nor any reports 15:40:24 <mboddu> sharkcz: Okay, makes sense 15:40:39 <sharkcz> but I think I had a query for koji web so show me the failed tasks ... 15:40:59 <nirik> yeah, but then you have to look at each one to see if it was failed in src.rpm 15:41:10 <sharkcz> like see all failed for buildervm8 15:41:11 <sharkcz> yup 15:42:07 <sharkcz> I don't think there are so many other failures :-) 15:42:15 <sharkcz> let me look at it again 15:44:46 <sharkcz> oh, I even put the URL to the ticket 15:44:49 <mboddu> #info Fedora 32 RC 1.2 is successfully composed 15:46:40 <sharkcz> nirik: so no "srpm build" issue so far 15:47:01 <nirik> yeah. ;( 15:48:23 <sharkcz> we could temporarily disable most of the s390x builders and send some large builds to the queue ... 15:49:03 <sharkcz> or find some other way how to get them to -08 15:49:36 <nirik> well, we could force reassign some... but I would normally only want to do that toscratch builds. 15:50:08 <mboddu> And probably not during freeze 15:50:19 <sharkcz> right :-) 15:50:38 <sharkcz> was just thinking loud 15:50:49 <mboddu> :) 15:51:42 <mboddu> #info We haven't seen any src.rpm build failures lately on s390x machines 15:52:43 <mboddu> Anything else to discuss here? 15:52:54 <sharkcz> not from my side 15:53:15 <sharkcz> just checked rc 1.2 has all the bits I need 15:53:53 <mboddu> Well then, I can give 6 min back to you sharkcz (not nirik :) ) as we are getting pulled into another meeting 15:53:59 <mboddu> Thanks everyone for joining 15:54:10 <sharkcz> ok, thanks guys 15:54:13 <mboddu> sharkcz: That is awesome, happy testing :) 15:54:15 <mboddu> #endmeeting