15:03:42 <mboddu> #startmeeting RELENG (2020-04-14)
15:03:42 <zodbot> Meeting started Tue Apr 14 15:03:42 2020 UTC.
15:03:42 <zodbot> This meeting is logged and archived in a public location.
15:03:42 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:42 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:03:42 <zodbot> The meeting name has been set to 'releng_(2020-04-14)'
15:03:43 <mboddu> #meetingname releng
15:03:43 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec
15:03:43 <mboddu> #topic init process
15:03:43 <zodbot> The meeting name has been set to 'releng'
15:03:43 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
15:03:51 <nirik> morning
15:04:14 <mboddu> Morning
15:04:18 <sharkcz> hello
15:04:25 * mboddu waves at sharkcz
15:04:50 * sharkcz waves back :-)
15:05:07 <mboddu> Let me start with open floor today as I have things to discuss there
15:05:16 <mboddu> Or not
15:05:22 <mboddu> #topic FMC-ODCS
15:05:37 <mboddu> nirik: I need some help here
15:05:50 <nirik> FMC-ODCS?
15:06:34 <sharkcz> I like acronyms ;-)
15:07:03 <mboddu> Fedora Minimal Compose - using ODCS :D
15:07:11 <nirik> ah, ok.
15:07:35 <mboddu> Well, technicall its just FMC, but the problem is with odcs and its permissions
15:07:51 <mboddu> The latest error is https://koji.fedoraproject.org/koji/taskinfo?taskID=43385938
15:08:07 <mboddu> Which means runroot requires access to /srv/odcs/
15:08:30 <mboddu> It seems it is configured in https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_builder/templates/runroot.conf.j2
15:08:51 <mboddu> I never touched it before, so here is what I am thinking that needs changes
15:09:33 <mboddu> 1. Adding /srv/odcs to default mounts in https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_builder/templates/runroot.conf.j2#n8
15:10:03 <mboddu> 2. Adding a path like in https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_builder/templates/runroot.conf.j2#n39
15:10:29 <mboddu> I dont think we need any path substitutions or safe roots(which I have no idea what they means)
15:10:33 <mboddu> Is my understanding correct?
15:10:54 <mboddu> And, are there any considerations before we can apply this change?
15:10:59 <nirik> yeah, I think it would need to be added... but how did it work before?
15:11:11 <nirik> well, we are in freeze, it would definitely need a freeze break
15:11:54 <nirik> I guess odcs didn't do any images before?
15:11:54 <mboddu> nirik: It never worked, this is the first odcs compose requiring runroot stuff
15:11:59 <mboddu> Yes
15:12:21 <nirik> so for repos/etc it just does them like normal builds... but images it needs runroot, I assume
15:12:52 <mboddu> That is correct
15:13:32 <nirik> ok, then yeah... would need the above, possibly also changing exports on the /srv/odcs to make sure it can be mounted
15:13:57 <mboddu> Okay, I dont know how to set it up
15:15:33 <nirik> I'm not sure the exact syntax there either... would have to look at it more closely.
15:17:27 <mboddu> Okay, do you want me set 15 min aside to look at it later today?
15:17:39 <mboddu> Or it might take more than that?
15:18:58 <nirik> sure, we can look at it outside meeting, etc... shouldn't take long. I wish we could test it in stg.
15:20:33 <mboddu> I wish that too, but provided that we only have few days with odcs releng backend, I didn't start with stg as I didn't want to go through all the issues with stg and waste the little time I had
15:21:11 <nirik> sure
15:21:34 <mboddu> #info nirik and mboddu will meet later today to go over this
15:21:45 <mboddu> #topic F32 RC compose
15:22:35 <mboddu> By the time RC 1.1 failed, I was asleep and luckily (or unluckily for me) I got up at 4am in the morning and was able to get a hold of tkopecek
15:23:08 <nirik> I saw it before I went to bed, but was confused by it. ;)
15:23:08 <mboddu> He is not sure why it failed with that specific error, but he asked me rerun the compose
15:23:32 <mboddu> RC 1.2 seems to be fine and https://koji.fedoraproject.org/koji/buildinfo?buildID=1493304 seems to worked fine
15:23:37 <mboddu> nirik: Yeah, it was odd
15:23:54 <mboddu> I created a koji ticket to look into this issue - https://pagure.io/koji/issue/2145
15:24:06 <nirik> might be some kind of race condition
15:24:12 <nirik> that we normally don't hit
15:24:33 <mboddu> So, he asked me look at the logs for all the builders in livemedia channel and see if any other builder did something wonky
15:25:00 <mboddu> Now my question, is there any easy way to do it or just manually ssh'ing into those boxes and looking for it manually?
15:25:56 <mboddu> And here's the latest news on it - https://pagure.io/releng/issue/9403#comment-642069 :(
15:26:06 <nirik> log01 has all logs from everything... look in /var/log/merged/
15:27:23 <nirik> fun
15:28:57 <mboddu> Okay, thanks
15:30:10 <mboddu> Hmmm
15:30:11 <mboddu> $ ls /var/log/merged/
15:30:12 <mboddu> apps.log        boot.log       cron.log.1.gz        kernel.log       mail.log.1.gz      secure.log
15:30:12 <mboddu> apps.log.1.gz   boot.log.1.gz  http_error.log       kernel.log.1.gz  messages.log       secure.log.1.gz
15:30:12 <mboddu> audit.log.1.gz  cron.log       http_error.log.1.gz  mail.log         messages.log.1.gz
15:30:19 <mboddu> Is it part of apps.log?
15:30:35 * mboddu didn't grep it yet, but I thought I would see kojid.log or something
15:30:36 <nirik> no, messages
15:31:40 <mboddu> Ah cool
15:31:41 <mboddu> Thanks
15:32:07 <mboddu> greping for it in the whole dir would take a lot of time, this helps a lot
15:32:57 <mboddu> #action mboddu will go through the logs and update the koji ticket
15:32:57 <nirik> hopefully it would be there. thats all syslog...
15:33:16 <mboddu> Well, we will do whatever we can :)
15:37:07 <mboddu> #topic Alternate Arch Updates
15:37:14 <mboddu> sharkcz: Any updates?
15:37:39 <sharkcz> no, everything looks good
15:37:57 <sharkcz> I wrote http://fedora.danny.cz/check-compose to check if we have all artifacts
15:38:23 <sharkcz> so I don't have to look "manually" at https://kojipkgs.fedoraproject.org/compose
15:38:52 <mboddu> sharkcz: You could have used https://pagure.io/releng/failed-composes/issues
15:39:15 <sharkcz> the email reports show the changes, not the state
15:39:37 <sharkcz> mboddu: it still needs some digging to see what exactly is missing
15:39:48 <nirik> sharkcz: have you seen any srpm failures on that builder? I haven't looked recently...
15:40:07 <sharkcz> nirik: have seen it myself, nor any reports
15:40:24 <mboddu> sharkcz: Okay, makes sense
15:40:39 <sharkcz> but I think I had a query for koji web so show me the failed tasks ...
15:40:59 <nirik> yeah, but then you have to look at each one to see if it was failed in src.rpm
15:41:10 <sharkcz> like see all failed for buildervm8
15:41:11 <sharkcz> yup
15:42:07 <sharkcz> I don't think there are so many other failures :-)
15:42:15 <sharkcz> let me look at it again
15:44:46 <sharkcz> oh, I even put the URL to the ticket
15:44:49 <mboddu> #info Fedora 32 RC 1.2 is successfully composed
15:46:40 <sharkcz> nirik: so no "srpm build" issue so far
15:47:01 <nirik> yeah. ;(
15:48:23 <sharkcz> we could temporarily disable most of the s390x builders and send some large builds to the queue ...
15:49:03 <sharkcz> or find some other way how to get them to -08
15:49:36 <nirik> well, we could force reassign some... but I would normally only want to do that toscratch builds.
15:50:08 <mboddu> And probably not during freeze
15:50:19 <sharkcz> right :-)
15:50:38 <sharkcz> was just thinking loud
15:50:49 <mboddu> :)
15:51:42 <mboddu> #info We haven't seen any src.rpm build failures lately on s390x machines
15:52:43 <mboddu> Anything else to discuss here?
15:52:54 <sharkcz> not from my side
15:53:15 <sharkcz> just checked rc 1.2 has all the bits I need
15:53:53 <mboddu> Well then, I can give 6 min back to you sharkcz (not nirik :) ) as we are getting pulled into another meeting
15:53:59 <mboddu> Thanks everyone for joining
15:54:10 <sharkcz> ok, thanks guys
15:54:13 <mboddu> sharkcz: That is awesome, happy testing :)
15:54:15 <mboddu> #endmeeting