16:00:09 <mboddu> #startmeeting RELENG (2020-02-05)
16:00:09 <zodbot> Meeting started Wed Feb  5 16:00:09 2020 UTC.
16:00:09 <zodbot> This meeting is logged and archived in a public location.
16:00:09 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:09 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:09 <zodbot> The meeting name has been set to 'releng_(2020-02-05)'
16:00:10 <mboddu> #meetingname releng
16:00:10 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec
16:00:10 <mboddu> #topic init process
16:00:10 <zodbot> The meeting name has been set to 'releng'
16:00:10 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
16:00:15 <nirik> morning
16:01:07 <smooge> morning
16:01:24 <mboddu> nirik, smooge : Morning, how goes?
16:01:32 <jrichardson> afternoon
16:01:39 <dustymabe> morning all
16:01:43 <dustymabe> or afternoon/evening
16:02:12 <smooge> mboddu, it goes
16:03:10 <mboddu> Morning everyone
16:04:06 <mboddu> Lets get started
16:05:59 * cverna is kind of around
16:09:37 <mboddu> jrichardson: Do you wanna introduce yourself to the folks here?
16:09:52 <jrichardson> sure
16:09:53 <mboddu> I guess only Dusty might not know about you
16:09:59 <mboddu> But go ahead
16:10:56 <jrichardson> Hi all, my name is James, I'm a mature student at Waterford Institute of Tech and interning with Leigh Griffin on the AAA/Sustaining team here in Waterford
16:11:37 <jrichardson> hoping to be useful soon :) EOM
16:12:12 <cverna> Nice to see you here jrichardson
16:12:43 <jrichardson> cverna, thanks, glad to be
16:13:13 <mboddu> Thanks jrichardson
16:13:21 <mboddu> Very happy that you are here to help :)
16:13:27 <mboddu> So, lets move to the tickets
16:13:45 <nirik> sigh... laptop decided crashing was a good idea.
16:13:50 <mboddu> #topic #9057 Ability and permission to sign applications for download using Fedora Releng keys
16:13:59 <mboddu> #link https://pagure.io/releng/issue/9057
16:14:16 <mboddu> nirik: No problem, we just went through the intro of jrichardson
16:14:19 <mboddu> dustymabe: You are up
16:14:24 <dustymabe> mboddu: I think this is mostly not an issue any longer
16:14:31 <nirik> hi jrichardson!
16:14:34 <dustymabe> we've worked with you guys and have a working relationship on this topic
16:14:44 <dustymabe> but there is still work outstanding to automate more of the process I think
16:14:50 <dustymabe> mboddu: would you agree?
16:15:29 <mboddu> dustymabe: Yes
16:15:30 <jrichardson> nirik, hello!
16:16:55 <dustymabe> mboddu: so we can probably stop discussing this in meetings for now
16:17:20 <mboddu> dustymabe: Okay, can we close the ticket?
16:17:22 <cverna> Who is going to work on the automation ?
16:17:45 <cverna> What needs to be done manually ?
16:18:17 <mboddu> cverna: Its scripted out, but someone with access to the keys has to run it.
16:18:30 <dustymabe> cverna: we are tracking the automation in https://github.com/coreos/fedora-coreos-tracker/issues/335
16:18:45 <cverna> Ok :)
16:18:46 <dustymabe> right now we manually open a ticket and ask releng to run it like: https://pagure.io/releng/issue/9126
16:18:57 <dustymabe> mboddu: we can probably close the ticket
16:18:58 <dustymabe> +1
16:19:11 <mboddu> dustymabe: Do you have everything you need for the automation?
16:19:51 <dustymabe> mboddu: I'm not 100% sure on that
16:20:01 <dustymabe> but will definitely let you know if there are gaps/questions that need to be answered
16:20:33 <mboddu> dustymabe: Okay, please let us know and we would be happy to help
16:21:42 <mboddu> #info Current status is that we have a script to run but someone has to manually run it, dustymabe and team are working on automating it. For now we will close the ticket.
16:22:42 <mboddu> #topic #9153 Use an automated script to control checksums of compose images
16:22:49 <mboddu> #link https://pagure.io/releng/issue/9153
16:23:32 <mboddu> So, generally I have a script that I run before go/no-go for rc composes
16:23:54 <mboddu> But it would be nice to have it for nightly composes and that too automated
16:25:38 <mboddu> nirik: Do we have an bot user that can update wiki?
16:25:58 <nirik> we do not currently no, but one could be made I suppose
16:26:47 <mboddu> nirik: So, I was thinking of updating the nightly.sh script to run the tests and update the wiki, wdyt?
16:28:01 <nirik> which tests?
16:28:44 <mboddu> nirik: checksum tests
16:28:55 <nirik> oh, sorry, missed the topic...
16:29:10 <nirik> can we just add it to the report we email the list?
16:29:58 <mboddu> The whole point is updating the wiki, since they have to manually update the wiki
16:30:24 <nirik> well, I think the question is still open. I like kparal's idea...
16:30:36 <nirik> ie, once we test it remove it from qa matrix
16:31:03 <nirik> so, I'd say lets discuss more in ticket?
16:31:12 * kparal is here
16:31:59 <kparal> mboddu: I'd prefer to just remove it from the wiki and rely on your scripts to do the job
16:32:20 <nirik> hey kparal
16:32:20 <mboddu> Also, now I think about it, when compose is just done, there wont be a wiki pointing to that compose, so we cant just update wiki
16:32:28 * mboddu waves at kparal
16:32:57 <kparal> mboddu: our wiki is just a poor-man's tcms, really :)
16:33:32 <mboddu> haha :)
16:34:19 <mboddu> May be we should just do what nirik said and report it in the email
16:34:33 <kparal> if your scripts blow up when the checksums don't match, we don't need to track the verification result. we'll know that it will be obvious when something goes wrong
16:35:01 <mboddu> ack
16:35:08 <kparal> i.e. the compose will not be where it is supposed to be, or perhaps doomed or something, and everyone will know
16:35:49 <kparal> it's not like it's going to happen frequently
16:36:04 <mboddu> kparal: I remember for one release, the 32 bit images failed that test, but we went ahead and released anyway, so, I am guessing we only test it for release blocking artifacts?
16:36:57 <nirik> we also need to confirm that the images are all in the checksum files.
16:37:05 <kparal> well, I don't remember that one, but we block the release just on release blocking artifacts. So I guess we didn't care about i386 checksums that much
16:38:34 <kparal> I hope that this operation is so simple that if it breaks, it means a releng script broke somewhere (e.g. we added some new composes somewhere but forgot to add them in all places) or there's a serious problem in sha256sum or glibc or similar
16:38:39 <kparal> or a hardware problem
16:39:05 * nirik nods
16:39:17 <kparal> it's like signing packages, if it's broken, we definitely want to stop and look at it
16:39:58 <kparal> nirik: yes we need to check that all images are present in the checksum file and our own testcase doesn't reflect that :)
16:40:20 <mboddu> kparal, nirik : That sounds good to me
16:43:05 <nirik> so, leave ticket/discuss, move on?
16:43:19 <mboddu> #info RelEng will update the nightly script to run "sha256sum -c", "checkisomd5" tests and verify that the images are all in the checksum file and post the results to the compose email and if anyone of them fails, we should not sync the compose to the mirrors
16:44:45 <mboddu> If we want to discuss more, I will post this in the comments and ask for feedback
16:46:30 <nirik> sure.
16:47:13 <mboddu> #undo
16:47:13 <zodbot> Removing item from minutes: INFO by mboddu at 16:43:19 : RelEng will update the nightly script to run "sha256sum -c", "checkisomd5" tests and verify that the images are all in the checksum file and post the results to the compose email and if anyone of them fails, we should not sync the compose to the mirrors
16:47:38 <mboddu> #info RelEng thinks we should update the nightly script to run "sha256sum -c", "checkisomd5" tests and verify that the images are all in the checksum file and post the results to the compose email and if anyone of them fails, we should not sync the compose to the mirrors. We will ask for comments in the ticket.
16:49:55 <mboddu> #topic #9154 F32 Change: new buildroot for CPU baseline update
16:50:02 <mboddu> #link https://pagure.io/releng/issue/9154
16:50:10 <mboddu> Thanks nirik for asking those questions
16:50:19 <mboddu> We haven't got any response from them
16:50:26 <mboddu> I will ping them in the ticket again
16:50:50 <nirik> well, I just added the questions yesterday. ;)
16:50:57 <nirik> I'm sure they will get to it.
16:51:03 <nirik> hopefully my questions made sense
16:51:21 <mboddu> Those are the questions that I had and wanted to discuss
16:51:42 <mboddu> Well, you got more, but I had questions about the buildroot and the signing :)
16:52:29 <mboddu> So, last few min
16:52:33 <mboddu> #topic Open Floor
16:52:46 <nirik> so, I had a few things...
16:52:52 <mboddu> Go ahead
16:53:21 <mboddu> Also, can you let us know whats the issue with the f31 updates composes as well?
16:53:38 <nirik> 1) we have been talking about doing some standups to go over/process tickets and backlog. I think this will be great... will try and keep the list in the loop as to when in case anyone wants to attend.
16:54:33 <nirik> 2) branching is next week. We should try and make sure we are all prepped for it. I'd like to look at signing everything for f33 this weekend so we are ready. Also, note that this time we are doing the 'no new updates in until we have a compose' thing...
16:55:04 <nirik> I think the way we do that is to let packages land in some tag and disable autosign or bodhi processing them until we have a compose...
16:55:21 <nirik> but we should figure the exact change. ;)
16:55:59 <nirik> 3) hopefully rawhide will compose later today with the mass rebuild in it. I guess mboddu you are gonna file the FTBFSes?
16:56:41 <mboddu> On 2. I am guessing we should not run updates composes until we have a branched compose
16:56:46 <mboddu> On 3. yes, I am on it
16:57:09 <nirik> nope... we can't do that because bodhi isn't enabled for another few weeks.
16:57:46 <nirik> also last time we had it like pre-gating rawhide... we should make it like gating rawhide this time.
16:58:20 <mboddu> Okay, I will look into it
16:58:42 <nirik> so basically, branch, then disable probibly autosign so newly built packages go to some tag and sit until we have a branched compose
16:58:51 <mboddu> So, I got 2 things
16:58:55 <nirik> should we open a ticket on this to coordinate?
16:59:02 <nirik> might need to check with rawhide gating folks too.
16:59:03 <mboddu> Sure
16:59:10 <mboddu> ack
17:00:44 <nirik> I think I had something else, but I can't remember now. ;)
17:01:34 <mboddu> nirik: Maybe, we could add fxx-staging tag and make autosign to move the builds to that tag and once we have a compose, we can use mass-tag.py to tag them into fxx
17:03:38 <nirik> I think we don't need to add any new tag, just config autosign to not sign/move things... then when we have a compose we just retag everything and reconfigure autosign to process it again as normal
17:04:06 * nirik has another meeting...
17:05:58 <mboddu> Okay, we can discuss in that ticket
17:06:25 <mboddu> The only reason why I thought of adding another tag is not to lag the signing when we decide to set things again to normal
17:06:43 <mboddu> Thanks for joining everyone
17:07:10 <mboddu> If you have anything else to discuss, lets take it to #fedora-releng
17:07:15 <mboddu> #endmeeting