14:31:01 <mboddu> #startmeeting RELENG (2017-08-07)
14:31:02 <zodbot> Meeting started Mon Aug  7 14:31:01 2017 UTC.  The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:31:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:31:02 <zodbot> The meeting name has been set to 'releng_(2017-08-07)'
14:31:02 <mboddu> #meetingname releng
14:31:02 <zodbot> The meeting name has been set to 'releng'
14:31:02 <mboddu> #chair dgilmore nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu Kellin
14:31:02 <zodbot> Current chairs: Kellin dgilmore masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll
14:31:02 <mboddu> #topic init process
14:31:11 <maxamillion> .hello maxamillion
14:31:12 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
14:31:13 <puiterwijk> .hello puiterwijk
14:31:15 <zodbot> puiterwijk: puiterwijk 'Patrick "マルタインアンドレアス" Uiterwijk' <puiterwijk@redhat.com>
14:31:31 <nirik> morning
14:31:36 <nirik> .hello kevin
14:31:37 <zodbot> nirik: kevin 'Kevin Fenzi' <kevin@scrye.com>
14:31:44 <sharkcz> .hello sharkcz
14:31:45 <zodbot> sharkcz: sharkcz 'Dan Horák' <dan@danny.cz>
14:32:08 <dustymabe> .nextmeeting
14:32:08 <zodbot> dustymabe: (nextmeeting <channel>) -- Return the next meeting scheduled for a particular channel.
14:32:17 <dustymabe> .nextmeeting #fedora-meeting-2
14:32:18 <zodbot> dustymabe: In #fedora-meeting-2 is Fedora ARM & AArch64 status meeting (starting in a day)
14:32:21 <zodbot> dustymabe: In #fedora-meeting-2 is FAmSCo (starting in 2 days)
14:32:24 <zodbot> dustymabe: In #fedora-meeting-2 is Fedora Packaging Committee (starting in 2 days)
14:32:25 * masta is here
14:32:26 <zodbot> dustymabe: - https://apps.fedoraproject.org/calendar/location/fedora-meeting-2%40irc.freenode.net/
14:34:41 <mboddu> Okay, lets start
14:35:06 <mboddu> #topic #6939 Tracking ticket for bodhi->pungi mashing
14:35:19 <mboddu> #link https://pagure.io/releng/issue/6939
14:35:29 <mboddu> dustymabe: you are up :)
14:35:39 <dustymabe> mboddu: :)
14:35:41 <dustymabe> hi everyone
14:35:46 <maxamillion> o/
14:36:07 <dgilmore> hey all
14:36:21 <dustymabe> so basically I have identified changes to the bodhi backend machines that are needed for bodhi + pungi integration work
14:36:43 <dustymabe> I have documented most of these changes in https://pagure.io/fedora-infrastructure/issue/6192
14:37:07 <dgilmore> dustymabe: that should likely be a releng issue
14:37:14 <dgilmore> though it does mix the two
14:37:15 <dustymabe> Please look through that ticket and see if you agree/disagree with anything.
14:37:29 <dustymabe> dgilmore: yeah - it's hard. releng and infra are pretty tightly coupled
14:37:46 <dustymabe> i think the important part is just socializing it across both groups
14:37:51 <dustymabe> which is what I'm doing here :)
14:38:09 <dustymabe> also, there is a 2nd ticket
14:38:13 <dgilmore> dustymabe: there will be more appropriate visibility in releng than infra
14:38:44 <dustymabe> dgilmore: mboddu created a releng tracking ticket and I sent an email to the releng list
14:38:57 <dustymabe> so hopefully we've covered bases
14:39:04 <maxamillion> dustymabe: +1
14:39:10 <dustymabe> the 2nd ticket is this one: https://pagure.io/fedora-infrastructure/issue/6193
14:39:34 <dustymabe> it's related to the folder structure differences between mash and pungi
14:39:48 <mboddu> dustymabe: just make sure that you update the releng ticket with your findings
14:39:50 <dustymabe> and how do we want to handle those differences when we move bodhi to call pungi
14:40:24 <dgilmore> dustymabe: again wrong forum to file the issue
14:40:36 <dgilmore> dustymabe: its a very simple to fix problem
14:40:57 <dustymabe> dgilmore: sure, but what way.. i was just going to create symlinks
14:41:03 <dustymabe> which is a simple fix
14:41:15 <dustymabe> but wanted to ask what everyone thought would be the best fix
14:41:56 <dgilmore> dustymabe: it needs some symlinks
14:42:09 <dgilmore> dustymabe: the process today relies on symlinks already
14:42:22 <dustymabe> ok
14:42:33 <dustymabe> i'll just use symlinks then
14:42:34 <dgilmore> its a matter of tweaking the symlinks
14:42:50 <dustymabe> dgilmore: can you elaborate on exactly what symlinks are set up today?
14:43:08 <dustymabe> I can dig more but I didn't realize there were a lot of symlinks being used
14:43:13 <dustymabe> mostly just one or two
14:43:17 <dgilmore> the way the update push magic works is simple
14:43:32 <dustymabe> magic + simple in the same sentence :)
14:43:48 <dgilmore> bodhi does a mash in /mnt/koji/mash/updates/
14:44:04 <dgilmore> say to f25-updates-170807.1350
14:44:24 <dgilmore> when all the mashes are done and bodhi is ready to make the content available to the mirrors
14:44:29 <dgilmore> it updates some symlinks
14:44:54 <dgilmore> ls -lah /mnt/koji/mash/updates/f25-updates
14:44:55 <dgilmore> lrwxrwxrwx. 1 apache apache 58 Aug  7 02:07 /mnt/koji/mash/updates/f25-updates -> /mnt/koji/mash/updates/f25-updates-170806.1901/f25-updates
14:45:32 <dustymabe> that it?
14:46:12 <dgilmore> the cronjob that runs every 15 minuts that syncs the content to the master mirror just does a rsync from  /mnt/koji/mash/updates/f25-updates/ to /pub/fedora/linux/updates/25/ and /pub/fedora-secondary/updates/25/
14:46:18 <dgilmore> dustymabe: thats it
14:46:33 <dustymabe> ok, I understood all of that
14:46:38 <dgilmore> bodhi then watches the master mirror for the metadata to update and says we are done
14:46:41 <dustymabe> this ticket is actually somewhat more complicated, though
14:46:48 <dustymabe> at least I think it is
14:46:49 <dgilmore> dustymabe: no its not
14:47:27 <dustymabe> ok so today you have
14:47:32 <dustymabe> $topdir/f26-updates-170804.1348/f26-updates/aarch64/
14:47:37 <dustymabe> and
14:47:39 <dustymabe> $topdir/f26-updates-170804.1348/f26-updates/x86_64/
14:47:42 <dgilmore> dustymabe: no
14:48:04 <dgilmore> dustymabe: $topdir/f26-updates-170804.1348/f26-updates/ to /mnt/koji/mash/updates/f26-updates
14:48:15 <dustymabe> yeah I get that
14:48:22 <dgilmore> then let the existing processes take over
14:48:23 <dustymabe> i'm talking about what is underneath that directory
14:48:36 <dgilmore> dustymabe: I think you are over complicating it
14:48:54 <dustymabe> ok, all i'm pointing out is that what is underneath that directory will look different
14:49:02 <dgilmore> dustymabe: so?
14:49:07 <dustymabe> pungi puts files out in a different hierarchy than mash did
14:49:12 <dgilmore> we accept that its different and roll with it
14:49:18 <dgilmore> lets not complicate it at all
14:49:31 <dustymabe> dgilmore: that's fine - i'm just trying to not make a change without telling anyone
14:49:39 <dustymabe> i imagine some other work would need to be done
14:49:45 <dgilmore> dustymabe: we know its going to change
14:49:48 <dgilmore> its accepted
14:49:57 <dustymabe> and no work needs to be done by anyone else?
14:50:02 <dgilmore> mirrormanager will need tweaking
14:50:13 <dgilmore> but that will be all
14:50:22 <dustymabe> ok, that's fine. I just really wanted to make sure
14:50:37 <dustymabe> you never know who built what process that depended on that hierarchy
14:50:47 <maxamillion> +1 - I didn't know the full impact of such a change either
14:51:10 <dustymabe> also, one tricky part here is that we are adding our changes to bodhi such that they can be activated by configuration
14:51:20 <dustymabe> so if it ends of causing issues we can easily revert the change
14:51:36 <dustymabe> if our file hierarcies are going to be different between the two, then that will be less easy to do
14:51:42 <dustymabe> which is ok
14:51:46 <dustymabe> we just need to be prepared
14:52:05 <dustymabe> who do I need to engage on doing mirrormanager work
14:52:49 <puiterwijk> dustymabe: create a mirrormanager ticket
14:52:59 <dustymabe> puiterwijk: ok
14:53:07 <puiterwijk> github.com/fedora-infra/mirrormanager2
14:53:13 <dustymabe> that's it from me folks, dgilmore thanks for the discussion
14:53:43 <dgilmore> dustymabe: when the location changes we will have to update the internal mappings for where the repos are
14:54:00 <dustymabe> 'internal mappings' of mirrormanager
14:54:02 <dustymabe> ?
14:54:17 <dustymabe> dgilmore: nothing will need to change client side?
14:54:24 <dustymabe> i.e. in the fedora-repos package?
14:54:28 <puiterwijk> dustymabe: no
14:54:38 <dustymabe> what if people use baseurl= ?
14:54:39 <puiterwijk> mirrormanager tells clients basically the exact url
14:54:42 <dgilmore> dustymabe: we should update the baseurl location, but no
14:54:47 <dustymabe> dgilmore: +1
14:54:49 <puiterwijk> Then it breaks. And that's their fault.
14:54:58 <dgilmore> whats puiterwijk said
14:54:59 <dustymabe> ok
14:55:01 <dustymabe> good to know
14:55:06 * puiterwijk always tells people to never use baseurl. It's a terrible idea, and insecure
14:55:10 <dustymabe> knowledge is power
14:55:11 <maxamillion> will the baseurl location change on the mirrors though?
14:55:19 <dgilmore> maxamillion: yes
14:55:21 <puiterwijk> maxamillion: yep.
14:55:26 <puiterwijk> But mirrormanager can account for that
14:55:26 <dgilmore> but tahts transparent to users
14:55:33 <dgilmore> its all handled in mm
14:55:34 <puiterwijk> What Dennis said
14:55:34 <maxamillion> alright
14:55:43 <dustymabe> cool, thanks everyone
14:55:51 <dustymabe> mboddu: the meeting is yours :)
14:56:01 * mboddu also learned a lot from this discussion :)
14:56:09 <mboddu> dustymabe: :)
14:57:12 <dustymabe> mboddu: I ask the questions everyone thought they already knew the answers to :)
14:57:35 <dustymabe> i annoy dgilmore, but it's for the good of the planet :)
14:57:55 <maxamillion> :)
14:59:20 <mboddu> #info This needs a change in MirrorManager but symlinks can be handled easily
14:59:52 <mboddu> Okay, next topic
15:00:37 <mboddu> I want to go over alternate arches and open floor first before going to discuss about other topics
15:00:56 <mboddu> #topic Alternative Architectures updates
15:01:04 <mboddu> sharkcz: any updates?
15:01:20 <sharkcz> mboddu: nope
15:02:00 <sharkcz> mean all looks good
15:02:22 <mboddu> sharkcz: :) thanks
15:02:34 <dustymabe> mboddu: I don't know what all was supposed to be under this topic
15:02:41 <dustymabe> but ksinny has done some work recently
15:02:49 <dustymabe> investigating enabling more architectures for atomic host
15:03:15 <dustymabe> i think there weren't any real blockers - and most of the stuff can be worked out pretty easily
15:03:34 <mboddu> #info Everything is fine with alternate architectures.
15:03:36 <dustymabe> https://pagure.io/atomic-wg/issue/299
15:03:53 <mboddu> dustymabe: oh, awesome
15:04:12 <mboddu> #topic Open Floor
15:04:39 <mboddu> I have one update here
15:04:45 <maxamillion> I just wanted to mention that the stage container builds are offline still, I'm still working on the new architecture for OSBS
15:04:45 <puiterwijk> So, maybe we should (quickly?) discuss PAgure on Dist-Git?
15:04:48 <mboddu> #info Tomorrow, Aug 8th is Fedora 24 EOL
15:04:58 * nirik has 2 things.
15:05:10 <mboddu> puiterwijk: yes, thats why I want to do Open floor first
15:05:52 <mboddu> #info The stage container builds are offline and Adam is still working on the new architecture for OSBS
15:06:05 <puiterwijk> So, quick thing from me: this morning, I've decided to update the koji config and restart all builders (killing all active builds) because it was possible to build things from forks
15:06:10 <mboddu> puiterwijk: you wanna go first?
15:06:25 <dgilmore> puiterwijk: :(
15:06:47 <puiterwijk> Also, I am looking at fixing some other possible consistency dangers
15:07:06 <puiterwijk> Will let people know what I fixed afterwards
15:07:15 <dgilmore> puiterwijk: cheers
15:07:23 <maxamillion> puiterwijk++
15:07:25 <maxamillion> puiterwijk: thanks!
15:07:27 <mboddu> puiterwijk++
15:07:27 <zodbot> mboddu: Karma for puiterwijk changed to 20 (for the f26 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:07:28 <puiterwijk> That's it from me for this particular moment.
15:07:47 <puiterwijk> (and yes, that's vague. I will be more concrete in the next day or two)
15:08:19 <mboddu> #info Patrick updated the koji config today morning and restarted all builders (killing all active builds) because it was possible to build things from forks
15:08:53 <mboddu> #info Patrick is also working on fixing some other possible issues
15:08:58 <mboddu> nirik: your turn
15:09:26 <nirik> first: FYI, we are planning a mass update/reboot cycle this week/tomorrow and wed. (announcements should go out this morning). So there will be some downtime tomorrow, and we should likely avoid pushing updates tomorrow or make sure they are done in time.
15:09:48 * dgilmore has one thing also
15:09:49 <mboddu> masta: ^
15:10:00 <maxamillion> +1
15:10:10 <dustymabe> nirik: does that include stage machines?
15:10:17 * masta takes note
15:10:18 <dustymabe> i.e. bodhi-backend01 in stage?
15:10:22 <nirik> likely 21UTC... not sure when things finish these days.
15:10:34 <nirik> dustymabe: no. stage I was planning to do today.
15:10:49 <nirik> I can try and work around any testing you are doing tho
15:11:38 <nirik> just let me know. and I will ping you before I do anything.
15:11:43 <dustymabe> nirik: just send me a message before you do
15:11:45 <dustymabe> thanks
15:12:06 <mboddu> #info Infra is planning a mass update/reboot cycle this week/tomorrow and wed. So there will be some downtime.
15:13:09 <nirik> Second item: Is there any chance we can move the time of this meeting? I know everyones schedules are pretty full. When I was in MDT an 8:30am monday meeting was anoying, now that I am in PDT a 7:30am meeting monday is more anoying. ;)
15:13:28 <maxamillion> oof
15:13:36 <maxamillion> I'm open to reschedule if others are
15:13:41 <nirik> if folks want to keep the time thats fine, but note that I might not attend all the time.
15:13:54 <mboddu> dgilmore: ^^
15:14:14 <mboddu> I am okay with rescheduling it
15:14:16 <dustymabe> nirik: in PDT? did you move?
15:14:20 <nirik> yes.
15:14:25 <dustymabe> cool
15:14:36 <dgilmore> mboddu: you run it and own it
15:14:48 <nirik> denver -> salem, or (currently)
15:15:25 <mboddu> dgilmore: yeah, but I wanted to make sure that main participants of this meeting are okay with it.
15:16:08 <puiterwijk> I just have one request: could we get the time of the meeting set in UTC? So as not to move relative to UTC with DST?
15:16:41 <dgilmore> puiterwijk: we had that and people had issues with DST changes
15:16:44 <puiterwijk> (I hate DST, and intercontinental DST is... even more fun)
15:16:55 <mboddu> nirik: if no has issues, I will reschedule this meeting to a later time(probably not in the afternoon since we have people attending from EU)
15:17:15 <nirik> mboddu: we could also do a whenisgood or better opensource one.
15:17:19 <puiterwijk> dgilmore: I think it's easier to just say "do your own DST calculations" than the DST switches intercontinentally...
15:18:15 <maxamillion> which ever the group prefers, I'm good with ... we can do a whenisgood or whatever
15:19:15 <dgilmore> puiterwijk: it was conflicts with meetings etc
15:19:53 <dgilmore> puiterwijk: as we just followed DST and did not move
15:20:05 <dgilmore> but then others had meetings that moved
15:20:39 <puiterwijk> dgilmore: Right. I get that. Just realize that if we make it not set in UTC, I will most likely be about 2 hours off for a few weeks per year...
15:21:02 <dgilmore> sure
15:21:15 <dgilmore> its worth looking at again
15:21:28 <dgilmore> I will leave it to mboddu to figure out what is best
15:21:51 <puiterwijk> :). My note was just a request
15:22:14 <mboddu> dgilmore, puiterwijk : as nirik suggested we can do whenisgood and we will see from there
15:22:35 <puiterwijk> +1
15:23:02 <mboddu> dgilmore: your turn
15:23:33 <mboddu> dgilmore: you said you have something to talk
15:24:37 <dgilmore> mboddu: yep. currently running "time rm -rf *-1704* *-1705* *-1706* *-1707*" in /mnt/koji/mash/updates/ as we have every mash going back to the start of april
15:24:50 <dgilmore> I thought bowlofeggs had enabled the auto cleanup
15:24:55 <dgilmore> we need to get that done
15:25:23 <dgilmore> allong with the EOL once its all done I will run the script to clean up the f24 signed rpms in /mnt/koji/packages
15:25:29 <mboddu> Even I thought that auto clean up is enabled
15:25:32 <dgilmore> or get mboddu or Kellin to do it
15:25:47 <mboddu> dgilmore: sure, I can do it
15:25:59 <Kellin> is it already written and just needs automation or?
15:27:22 <dgilmore> Kellin: just needs running
15:27:53 <dgilmore> https://pagure.io/releng/blob/master/f/scripts/koji-cleanup-signed.py
15:28:00 <dgilmore> it even has the f24 keys listed
15:29:15 <dgilmore> on that I have to run
15:30:11 <mboddu> #info We need to look into bodhi auto cleanup to clean the mash lock files under /mnt/koji/mash/updates/
15:30:40 <mboddu> Okay, I think we are right on time, good we discussed open floor items
15:31:18 <mboddu> anybody got anything else?
15:31:57 <mboddu> Okay, thanks for joining guys
15:32:03 <mboddu> #endmeeting