15:31:10 <mboddu> #startmeeting RELENG (2017-03-06)
15:31:10 <zodbot> Meeting started Mon Mar  6 15:31:10 2017 UTC.  The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:31:10 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:31:10 <zodbot> The meeting name has been set to 'releng_(2017-03-06)'
15:31:10 <mboddu> #meetingname releng
15:31:10 <zodbot> The meeting name has been set to 'releng'
15:31:10 <mboddu> #chair dgilmore nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu Kellin
15:31:10 <zodbot> Current chairs: Kellin dgilmore masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll
15:31:10 <mboddu> #topic init process
15:31:25 <marc84> morning everyone
15:31:43 <bowlofeggs> .hello bowlofeggs
15:31:44 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
15:32:04 <nirik> .hello kevin
15:32:06 <zodbot> nirik: kevin 'Kevin Fenzi' <kevin@scrye.com>
15:34:57 <Kellin> .hello Kellin
15:34:58 <zodbot> Kellin: Sorry, but you don't exist
15:35:07 <Kellin> existential crisis time
15:35:15 <sharkcz> .hello sharkcz
15:35:16 <zodbot> sharkcz: sharkcz 'Dan Horák' <dan@danny.cz>
15:35:32 <Kellin> .hello kellin
15:35:33 <zodbot> Kellin: kellin 'None' <kellin@retromud.org>
15:35:52 <mboddu> Okay, lets start
15:35:54 <ksinny> .hello sinnykumari
15:35:57 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
15:36:56 <mboddu> #topic #6683 fedora-repos-25 still contains wrong Fedora 26 key
15:37:47 <mboddu> I submitted the PR for f25 over the weekend, I will build and create a new update after the meeting
15:38:04 <puiterwijk> mboddu++
15:38:04 <nirik> thanks. we should get that pushed out as soon as we can
15:38:14 <mboddu> nirik: also, can you take a look at https://pagure.io/fedora-repos/pull-requests
15:38:33 <mboddu> and do we need it for f23?
15:39:05 <mboddu> the above PR fixes for f24, and I think we should need it for f23 as well, since its still supported
15:39:09 <puiterwijk> Did it contain the old (invalid) key?
15:39:18 <puiterwijk> f23 isn't supported anymore
15:39:18 <mboddu> puiterwijk: yes
15:39:24 <nirik> f23 is dead already
15:39:38 <mboddu> sorry, my bad, :(
15:40:02 <puiterwijk> It's not supported anymore. But in my mind, that doesn't mean we can't fix this particular thing anymore, since it might make for a very poor user experience if they decide to skip two releases
15:40:03 <nirik> it's hard to keep track. ;)
15:40:20 <nirik> puiterwijk: how?
15:40:28 <puiterwijk> (but then, it is difficult to get the update out and all)
15:40:46 <puiterwijk> nirik: no clue, that's for the releng folks to figure out :D. I'm just offering my opinion :)
15:40:46 <mboddu> nirik: :)
15:40:57 * nirik is strongly against pushing post eol updates
15:41:03 <nirik> it's eol. It's done.
15:41:10 <puiterwijk> Works for me
15:41:11 <mboddu> puiterwijk: I am against pushing updates for EOL releases
15:41:38 <puiterwijk> mboddu: I'm not pushing for one or the other way, I was just giving an opinion
15:41:56 <puiterwijk> But F24 and F25 should be fixed
15:42:03 <nirik> anyhow, lets fix f24 and f25 and drive on
15:42:04 <nirik> yeah
15:42:38 <puiterwijk> And perhaps we should have some way to verify this in the future.
15:42:55 <mboddu> puiterwijk: I can understand your point, but its EOL
15:42:58 <puiterwijk> Like, part of some SOP to check that n-1 and n-2 have the correct key for n
15:43:48 <nirik> perhaps each time we make a key we should update all the versions and confirm it's right.
15:43:51 <mboddu> puiterwijk: We can check add that to branching SOP, but are you thinking of an automated way or just manual verification?
15:44:25 <puiterwijk> mboddu: ideally automated and manual check that the automated test ran, but if nothing else at least manual
15:45:28 <mboddu> puiterwijk: I am not sure how to check it automatically, but we are open to suggestions
15:46:40 <puiterwijk> mboddu: I can write a small script for that.. That's exactly what I did this time, so I can just generalize it
15:47:49 <mboddu> puiterwijk: I mean scripting is fine, but are you planning on running it as a cron somewhere, or just run it when ever we feel like testing keys?
15:48:38 <puiterwijk> mboddu: I would suggest making it a part of some SOPs to run it? branching e.g.
15:48:49 <mboddu> puiterwijk: okay, cool
15:49:34 <mboddu> #info mboddu created PR for f25 and f24 and once they are accepted he will build and create the updates
15:50:09 <mboddu> #info puiterwijk will create a script which will check if all the keys are up to date or not.
15:50:53 <mboddu> moving on to next topic
15:51:31 <mboddu> #topic #6663 Create another new tag for Modularity
15:52:15 <mboddu> Does anyone know what is a blocking model? I dont have any knowledge about it.
15:52:50 <mboddu> I created the tag but dgilmore-afk has more questions about it.
15:53:19 <mboddu> If anyone know what dgilmore-afk is talking about, we can continue and discuss about it or we can move to next topic
15:53:32 <jkaluza> I'm first time on this meeting, but I may know about this topic :)
15:55:02 <jkaluza> Can you send link to that ticket please? I have joined once you told meetbot about this topic
15:55:19 * nirik hasnt followed this one any
15:55:19 <mboddu> jkaluza: https://pagure.io/releng/issue/6663
15:56:03 <jkaluza> Ah, sorry then, no idea what guys meant by blocking model there ;/
15:56:21 <mboddu> jkaluza: no problem :)
15:56:34 <jkaluza> I just know what we want :D
15:57:05 <mboddu> Okay, lets skip it for now, we can check about it once dgilmore-afk is here
15:58:21 <mboddu> #topic #6645 Make sure OSTree repos are sync'ed correctly after the masher completes
15:58:31 <mboddu> #link https://pagure.io/releng/issue/6645
15:59:23 <mboddu> puiterwijk: you are supposed to meet dustymabe and jlebon, anything to share?
15:59:34 <nirik> I could also add info to that ticket.
15:59:53 <puiterwijk> I provided them info, and then they'd schedule a next meeting to talk further as they had to head out
16:00:22 <mboddu> nirik: sure
16:00:51 <puiterwijk> The problem is the two distinct phases touching the repo that might not be alligned
16:00:56 <nirik> the sync script runs every 30min (at :15 and :45) and rsyncs the objects first
16:01:18 <mboddu> nirik: so thats what the 30 mins delay is?
16:01:26 <nirik> we could switch that to just one rsync that does a --delete-after
16:02:07 <nirik> well, no, that doesn't make sense to me.
16:02:22 <nirik> the script runs every 30min, but it should normally sync completely when it's done
16:03:05 <mboddu> nirik: okay
16:04:12 <Kellin> puiterwijk: can you elaborate on 'two distinct phases touching the repo that might not be aligned'?
16:04:37 <mboddu> #info puiterwijk is going to meet with dustymabe and jlebon and discuss about it
16:04:59 <puiterwijk> Kellin: the problem here is that the bodhi push is probably syncing in files while the updates-sync script is pushing them out
16:05:27 <nirik> it's possible yeah.
16:05:44 <nirik> ideally we would kill this cron and replace it with a fedmsg triggered thing.
16:06:09 <nirik> "done with f25 updates" "ok, syncing just that and it's atomic version"
16:06:30 <Kellin> puiterwijk: okay, that makes sense, I took it originally as something between the pushes spaced 30 mins apart, thank you for the clarity
16:06:32 <mboddu> I like nirik suggestion, do it on fedmsg instead of cron
16:06:42 <puiterwijk> So do I
16:07:22 <Kellin> I concur with that - triggered from a message is far safer; does fedmsg allow for an alert if a message is not detected in X time to ensure we know if the queue fails?
16:07:38 <mboddu> puiterwijk: is nirik suggestion a viable solution?
16:07:54 <puiterwijk> mboddu: yes, it is. It just needs someone to do a bit of hacking
16:08:13 <nirik> in fact there's a multiyear infra ticket out on this. ;) Just never enough time to do all the things we would like to.
16:08:15 <mboddu> puiterwijk: nice
16:08:17 <puiterwijk> Kellin: yes, we have nagios monitoring for when specific messages aren't heard of for a long period of time
16:08:30 <puiterwijk> nirik: I think that this would be *perfect* for loopabull actually....
16:08:42 <puiterwijk> This would be a prime candidate for it
16:08:45 <Kellin> puiterwijk: is that a person or a project?
16:08:47 <nirik> could be yeah
16:08:53 <puiterwijk> Kellin: project. It's for releng automation
16:08:56 <puiterwijk> pagure.io/loopabull
16:10:05 * threebean joins late.
16:11:10 <Kellin> puiterwijk: no such project?
16:11:14 <mboddu> Kellin: https://pagure.io/releng-automation
16:12:11 <Kellin> mboddu: thanks
16:14:17 <mboddu> #info We need to look into syncing the objects using a fedmsg instead of a cron job
16:14:55 <mboddu> Okay, next topic
16:14:59 <mboddu> #topic #6600 Mock and sigined rawhides
16:15:06 <mboddu> #link https://pagure.io/releng/issue/6600
16:15:56 <mboddu> nirik: I know you signed everything with f27 key
16:16:13 <nirik> yeah, everything should be signed.
16:16:28 <mboddu> nirik: what about mock?
16:17:05 <nirik> I think they released a new version with f26, but I haven't looked closely
16:18:33 <nirik> I'm not sure what we can really do to make things any easier for them
16:19:00 <mboddu> nirik: yeah, thats true
16:19:42 <nirik> we can get them the key in advance of the switch, but there's going to be a day rawhide switches... and I don't know how we can make sure mock is switched at the same time on all releases.
16:20:26 <nirik> lets wait a bit and see if we hear back from mock folks?
16:20:35 <mboddu> nirik: the latest build they have is before branching, so I am not sure if its resolved in it or not
16:20:49 <mboddu> nirik: yep, lets wait
16:21:05 <mboddu> #info nirik signed everything with f27 key
16:21:46 <mboddu> before we run over time, I want to get the alt arches update
16:21:58 <mboddu> #topic Alternative Architectures updates
16:22:09 <mboddu> sharkcz: any updates?
16:22:22 <mboddu> sharkcz: esp regarding s390 into koji
16:22:43 <pbrobinson> they went out Friday
16:22:43 <sharkcz> we have the new s390 builders in some form and I suppose nirik will follow up with the guys on the details
16:22:56 <pbrobinson> ah, updates went out friday....
16:23:19 <mboddu> pbrobinson, sharkcz : thats awesome news
16:23:25 <mboddu> pbrobinson++
16:23:25 <zodbot> mboddu: Karma for pbrobinson changed to 5 (for the f25 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:23:29 <mboddu> sharkcz++
16:23:29 <zodbot> mboddu: Karma for sharkcz changed to 2 (for the f25 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:23:41 <nirik> yeah, will try and follow up today.
16:23:56 <nirik> it being monday, I have a full pile of issues to work.
16:24:04 <sharkcz> nirik: ok, will try to help if needed
16:24:13 <nirik> sharkcz: thanks.
16:25:00 <mboddu> nirik: may I know what is needed? what to follow up and with who?
16:26:11 <nirik> mboddu: the admins of the machine. They said they made fedora instances... but we need to know: how to actually login to them, what to do if we need to reprovision, can we do that ourselves or do we need them to? ask about a squid cache machine, confim we can login to them and run ansible over them to configure them.
16:26:57 <mboddu> nirik: oh, thats a lot to follow up :)
16:27:14 <nirik> yep. but we will get there. sounds like it's getting close.
16:27:38 <pbrobinson> nirik: I think sharkcz knows some of that as I'm sure he could reboot and reinstall the Fedora ones in the secondary instance
16:28:03 <pbrobinson> and it supports kickstart, so once we get that in place it should be reinstall from ks, then ansible
16:28:06 <sharkcz> pbrobinson: we are getting a new LPAR that won't allow that so easily
16:28:28 <pbrobinson> one step forward... one step back.... :-|
16:29:04 <sharkcz> I think it's a compromise caused by the current location (physical and logical) of the mainframe ...
16:29:29 <mboddu> nirik: yeah, I wished it can be done before the alpha freeze, which is tomorrow(new schedule is not helping us)
16:30:29 <mboddu> #info the updates went out on Friday and nirik is planning to follow up with admins with further questions.
16:31:13 <mboddu> lets move on to next Open Floor, since we went over the scheduled time already
16:31:21 <mboddu> #topic Open Floor
16:31:29 <mboddu> anybody has anything to share?
16:31:37 <pbrobinson> yes
16:31:48 <pbrobinson> the ARMv7 images don't seem to be composing for f26
16:31:49 <pwhalen> looks like the arm disk images failed in the most recent compose, or looking at koji, werent even tried
16:32:00 <pbrobinson> what pwhalen said ;-)
16:32:02 <pwhalen> https://kojipkgs.fedoraproject.org/compose//branched/Fedora-26-20170306.n.0/logs/armhfp/liveimage-Fedora-Xfce-armhfp-26-20170306.n.0.armhfp.log
16:32:05 <pwhalen> that error again
16:32:11 <pwhalen> or warning
16:32:51 <pbrobinson> is the version of pungi on branched host the same as the one on rawhide?
16:32:58 <pwhalen> I dont see them in koji https://koji.fedoraproject.org/koji/tasks?state=all&owner=releng&view=tree&method=appliance&order=-id
16:33:00 <pbrobinson> it might just be pungi needs an updates
16:33:49 <nirik> well, there is a new pungi today...
16:35:59 <mboddu> pbrobinson: I just checked, both rawhide and branched hosts have the same version of pungi
16:38:07 <mboddu> the changelog for the new pungi release has - image-build: Pass arches around as a list (lsedlar)
16:38:08 <mboddu> - image-build: Expand arches for can_fail (lsedlar)
16:38:25 <mboddu> I am not sure, if thats a fix or not
16:39:21 <mboddu> #info ARMv7 images are failing in branched, it needs more digging, probably a pungi issue
16:40:06 <mboddu> anything else before we end the meeting?
16:41:11 <mboddu> Thank you guys for joining.
16:41:15 <ksinny> Is it possible to have atomic host composes enabled for powerpc64le and aarch64?
16:41:34 <marc84> mboddu: thanks
16:41:41 <mboddu> #endmeeting