14:31:01 <mboddu> #startmeeting RELENG (2017-08-07) 14:31:02 <zodbot> Meeting started Mon Aug 7 14:31:01 2017 UTC. The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:31:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:31:02 <zodbot> The meeting name has been set to 'releng_(2017-08-07)' 14:31:02 <mboddu> #meetingname releng 14:31:02 <zodbot> The meeting name has been set to 'releng' 14:31:02 <mboddu> #chair dgilmore nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu Kellin 14:31:02 <zodbot> Current chairs: Kellin dgilmore masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll 14:31:02 <mboddu> #topic init process 14:31:11 <maxamillion> .hello maxamillion 14:31:12 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com> 14:31:13 <puiterwijk> .hello puiterwijk 14:31:15 <zodbot> puiterwijk: puiterwijk 'Patrick "マルタインアンドレアス" Uiterwijk' <puiterwijk@redhat.com> 14:31:31 <nirik> morning 14:31:36 <nirik> .hello kevin 14:31:37 <zodbot> nirik: kevin 'Kevin Fenzi' <kevin@scrye.com> 14:31:44 <sharkcz> .hello sharkcz 14:31:45 <zodbot> sharkcz: sharkcz 'Dan Horák' <dan@danny.cz> 14:32:08 <dustymabe> .nextmeeting 14:32:08 <zodbot> dustymabe: (nextmeeting <channel>) -- Return the next meeting scheduled for a particular channel. 14:32:17 <dustymabe> .nextmeeting #fedora-meeting-2 14:32:18 <zodbot> dustymabe: In #fedora-meeting-2 is Fedora ARM & AArch64 status meeting (starting in a day) 14:32:21 <zodbot> dustymabe: In #fedora-meeting-2 is FAmSCo (starting in 2 days) 14:32:24 <zodbot> dustymabe: In #fedora-meeting-2 is Fedora Packaging Committee (starting in 2 days) 14:32:25 * masta is here 14:32:26 <zodbot> dustymabe: - https://apps.fedoraproject.org/calendar/location/fedora-meeting-2%40irc.freenode.net/ 14:34:41 <mboddu> Okay, lets start 14:35:06 <mboddu> #topic #6939 Tracking ticket for bodhi->pungi mashing 14:35:19 <mboddu> #link https://pagure.io/releng/issue/6939 14:35:29 <mboddu> dustymabe: you are up :) 14:35:39 <dustymabe> mboddu: :) 14:35:41 <dustymabe> hi everyone 14:35:46 <maxamillion> o/ 14:36:07 <dgilmore> hey all 14:36:21 <dustymabe> so basically I have identified changes to the bodhi backend machines that are needed for bodhi + pungi integration work 14:36:43 <dustymabe> I have documented most of these changes in https://pagure.io/fedora-infrastructure/issue/6192 14:37:07 <dgilmore> dustymabe: that should likely be a releng issue 14:37:14 <dgilmore> though it does mix the two 14:37:15 <dustymabe> Please look through that ticket and see if you agree/disagree with anything. 14:37:29 <dustymabe> dgilmore: yeah - it's hard. releng and infra are pretty tightly coupled 14:37:46 <dustymabe> i think the important part is just socializing it across both groups 14:37:51 <dustymabe> which is what I'm doing here :) 14:38:09 <dustymabe> also, there is a 2nd ticket 14:38:13 <dgilmore> dustymabe: there will be more appropriate visibility in releng than infra 14:38:44 <dustymabe> dgilmore: mboddu created a releng tracking ticket and I sent an email to the releng list 14:38:57 <dustymabe> so hopefully we've covered bases 14:39:04 <maxamillion> dustymabe: +1 14:39:10 <dustymabe> the 2nd ticket is this one: https://pagure.io/fedora-infrastructure/issue/6193 14:39:34 <dustymabe> it's related to the folder structure differences between mash and pungi 14:39:48 <mboddu> dustymabe: just make sure that you update the releng ticket with your findings 14:39:50 <dustymabe> and how do we want to handle those differences when we move bodhi to call pungi 14:40:24 <dgilmore> dustymabe: again wrong forum to file the issue 14:40:36 <dgilmore> dustymabe: its a very simple to fix problem 14:40:57 <dustymabe> dgilmore: sure, but what way.. i was just going to create symlinks 14:41:03 <dustymabe> which is a simple fix 14:41:15 <dustymabe> but wanted to ask what everyone thought would be the best fix 14:41:56 <dgilmore> dustymabe: it needs some symlinks 14:42:09 <dgilmore> dustymabe: the process today relies on symlinks already 14:42:22 <dustymabe> ok 14:42:33 <dustymabe> i'll just use symlinks then 14:42:34 <dgilmore> its a matter of tweaking the symlinks 14:42:50 <dustymabe> dgilmore: can you elaborate on exactly what symlinks are set up today? 14:43:08 <dustymabe> I can dig more but I didn't realize there were a lot of symlinks being used 14:43:13 <dustymabe> mostly just one or two 14:43:17 <dgilmore> the way the update push magic works is simple 14:43:32 <dustymabe> magic + simple in the same sentence :) 14:43:48 <dgilmore> bodhi does a mash in /mnt/koji/mash/updates/ 14:44:04 <dgilmore> say to f25-updates-170807.1350 14:44:24 <dgilmore> when all the mashes are done and bodhi is ready to make the content available to the mirrors 14:44:29 <dgilmore> it updates some symlinks 14:44:54 <dgilmore> ls -lah /mnt/koji/mash/updates/f25-updates 14:44:55 <dgilmore> lrwxrwxrwx. 1 apache apache 58 Aug 7 02:07 /mnt/koji/mash/updates/f25-updates -> /mnt/koji/mash/updates/f25-updates-170806.1901/f25-updates 14:45:32 <dustymabe> that it? 14:46:12 <dgilmore> the cronjob that runs every 15 minuts that syncs the content to the master mirror just does a rsync from /mnt/koji/mash/updates/f25-updates/ to /pub/fedora/linux/updates/25/ and /pub/fedora-secondary/updates/25/ 14:46:18 <dgilmore> dustymabe: thats it 14:46:33 <dustymabe> ok, I understood all of that 14:46:38 <dgilmore> bodhi then watches the master mirror for the metadata to update and says we are done 14:46:41 <dustymabe> this ticket is actually somewhat more complicated, though 14:46:48 <dustymabe> at least I think it is 14:46:49 <dgilmore> dustymabe: no its not 14:47:27 <dustymabe> ok so today you have 14:47:32 <dustymabe> $topdir/f26-updates-170804.1348/f26-updates/aarch64/ 14:47:37 <dustymabe> and 14:47:39 <dustymabe> $topdir/f26-updates-170804.1348/f26-updates/x86_64/ 14:47:42 <dgilmore> dustymabe: no 14:48:04 <dgilmore> dustymabe: $topdir/f26-updates-170804.1348/f26-updates/ to /mnt/koji/mash/updates/f26-updates 14:48:15 <dustymabe> yeah I get that 14:48:22 <dgilmore> then let the existing processes take over 14:48:23 <dustymabe> i'm talking about what is underneath that directory 14:48:36 <dgilmore> dustymabe: I think you are over complicating it 14:48:54 <dustymabe> ok, all i'm pointing out is that what is underneath that directory will look different 14:49:02 <dgilmore> dustymabe: so? 14:49:07 <dustymabe> pungi puts files out in a different hierarchy than mash did 14:49:12 <dgilmore> we accept that its different and roll with it 14:49:18 <dgilmore> lets not complicate it at all 14:49:31 <dustymabe> dgilmore: that's fine - i'm just trying to not make a change without telling anyone 14:49:39 <dustymabe> i imagine some other work would need to be done 14:49:45 <dgilmore> dustymabe: we know its going to change 14:49:48 <dgilmore> its accepted 14:49:57 <dustymabe> and no work needs to be done by anyone else? 14:50:02 <dgilmore> mirrormanager will need tweaking 14:50:13 <dgilmore> but that will be all 14:50:22 <dustymabe> ok, that's fine. I just really wanted to make sure 14:50:37 <dustymabe> you never know who built what process that depended on that hierarchy 14:50:47 <maxamillion> +1 - I didn't know the full impact of such a change either 14:51:10 <dustymabe> also, one tricky part here is that we are adding our changes to bodhi such that they can be activated by configuration 14:51:20 <dustymabe> so if it ends of causing issues we can easily revert the change 14:51:36 <dustymabe> if our file hierarcies are going to be different between the two, then that will be less easy to do 14:51:42 <dustymabe> which is ok 14:51:46 <dustymabe> we just need to be prepared 14:52:05 <dustymabe> who do I need to engage on doing mirrormanager work 14:52:49 <puiterwijk> dustymabe: create a mirrormanager ticket 14:52:59 <dustymabe> puiterwijk: ok 14:53:07 <puiterwijk> github.com/fedora-infra/mirrormanager2 14:53:13 <dustymabe> that's it from me folks, dgilmore thanks for the discussion 14:53:43 <dgilmore> dustymabe: when the location changes we will have to update the internal mappings for where the repos are 14:54:00 <dustymabe> 'internal mappings' of mirrormanager 14:54:02 <dustymabe> ? 14:54:17 <dustymabe> dgilmore: nothing will need to change client side? 14:54:24 <dustymabe> i.e. in the fedora-repos package? 14:54:28 <puiterwijk> dustymabe: no 14:54:38 <dustymabe> what if people use baseurl= ? 14:54:39 <puiterwijk> mirrormanager tells clients basically the exact url 14:54:42 <dgilmore> dustymabe: we should update the baseurl location, but no 14:54:47 <dustymabe> dgilmore: +1 14:54:49 <puiterwijk> Then it breaks. And that's their fault. 14:54:58 <dgilmore> whats puiterwijk said 14:54:59 <dustymabe> ok 14:55:01 <dustymabe> good to know 14:55:06 * puiterwijk always tells people to never use baseurl. It's a terrible idea, and insecure 14:55:10 <dustymabe> knowledge is power 14:55:11 <maxamillion> will the baseurl location change on the mirrors though? 14:55:19 <dgilmore> maxamillion: yes 14:55:21 <puiterwijk> maxamillion: yep. 14:55:26 <puiterwijk> But mirrormanager can account for that 14:55:26 <dgilmore> but tahts transparent to users 14:55:33 <dgilmore> its all handled in mm 14:55:34 <puiterwijk> What Dennis said 14:55:34 <maxamillion> alright 14:55:43 <dustymabe> cool, thanks everyone 14:55:51 <dustymabe> mboddu: the meeting is yours :) 14:56:01 * mboddu also learned a lot from this discussion :) 14:56:09 <mboddu> dustymabe: :) 14:57:12 <dustymabe> mboddu: I ask the questions everyone thought they already knew the answers to :) 14:57:35 <dustymabe> i annoy dgilmore, but it's for the good of the planet :) 14:57:55 <maxamillion> :) 14:59:20 <mboddu> #info This needs a change in MirrorManager but symlinks can be handled easily 14:59:52 <mboddu> Okay, next topic 15:00:37 <mboddu> I want to go over alternate arches and open floor first before going to discuss about other topics 15:00:56 <mboddu> #topic Alternative Architectures updates 15:01:04 <mboddu> sharkcz: any updates? 15:01:20 <sharkcz> mboddu: nope 15:02:00 <sharkcz> mean all looks good 15:02:22 <mboddu> sharkcz: :) thanks 15:02:34 <dustymabe> mboddu: I don't know what all was supposed to be under this topic 15:02:41 <dustymabe> but ksinny has done some work recently 15:02:49 <dustymabe> investigating enabling more architectures for atomic host 15:03:15 <dustymabe> i think there weren't any real blockers - and most of the stuff can be worked out pretty easily 15:03:34 <mboddu> #info Everything is fine with alternate architectures. 15:03:36 <dustymabe> https://pagure.io/atomic-wg/issue/299 15:03:53 <mboddu> dustymabe: oh, awesome 15:04:12 <mboddu> #topic Open Floor 15:04:39 <mboddu> I have one update here 15:04:45 <maxamillion> I just wanted to mention that the stage container builds are offline still, I'm still working on the new architecture for OSBS 15:04:45 <puiterwijk> So, maybe we should (quickly?) discuss PAgure on Dist-Git? 15:04:48 <mboddu> #info Tomorrow, Aug 8th is Fedora 24 EOL 15:04:58 * nirik has 2 things. 15:05:10 <mboddu> puiterwijk: yes, thats why I want to do Open floor first 15:05:52 <mboddu> #info The stage container builds are offline and Adam is still working on the new architecture for OSBS 15:06:05 <puiterwijk> So, quick thing from me: this morning, I've decided to update the koji config and restart all builders (killing all active builds) because it was possible to build things from forks 15:06:10 <mboddu> puiterwijk: you wanna go first? 15:06:25 <dgilmore> puiterwijk: :( 15:06:47 <puiterwijk> Also, I am looking at fixing some other possible consistency dangers 15:07:06 <puiterwijk> Will let people know what I fixed afterwards 15:07:15 <dgilmore> puiterwijk: cheers 15:07:23 <maxamillion> puiterwijk++ 15:07:25 <maxamillion> puiterwijk: thanks! 15:07:27 <mboddu> puiterwijk++ 15:07:27 <zodbot> mboddu: Karma for puiterwijk changed to 20 (for the f26 release cycle): https://badges.fedoraproject.org/tags/cookie/any 15:07:28 <puiterwijk> That's it from me for this particular moment. 15:07:47 <puiterwijk> (and yes, that's vague. I will be more concrete in the next day or two) 15:08:19 <mboddu> #info Patrick updated the koji config today morning and restarted all builders (killing all active builds) because it was possible to build things from forks 15:08:53 <mboddu> #info Patrick is also working on fixing some other possible issues 15:08:58 <mboddu> nirik: your turn 15:09:26 <nirik> first: FYI, we are planning a mass update/reboot cycle this week/tomorrow and wed. (announcements should go out this morning). So there will be some downtime tomorrow, and we should likely avoid pushing updates tomorrow or make sure they are done in time. 15:09:48 * dgilmore has one thing also 15:09:49 <mboddu> masta: ^ 15:10:00 <maxamillion> +1 15:10:10 <dustymabe> nirik: does that include stage machines? 15:10:17 * masta takes note 15:10:18 <dustymabe> i.e. bodhi-backend01 in stage? 15:10:22 <nirik> likely 21UTC... not sure when things finish these days. 15:10:34 <nirik> dustymabe: no. stage I was planning to do today. 15:10:49 <nirik> I can try and work around any testing you are doing tho 15:11:38 <nirik> just let me know. and I will ping you before I do anything. 15:11:43 <dustymabe> nirik: just send me a message before you do 15:11:45 <dustymabe> thanks 15:12:06 <mboddu> #info Infra is planning a mass update/reboot cycle this week/tomorrow and wed. So there will be some downtime. 15:13:09 <nirik> Second item: Is there any chance we can move the time of this meeting? I know everyones schedules are pretty full. When I was in MDT an 8:30am monday meeting was anoying, now that I am in PDT a 7:30am meeting monday is more anoying. ;) 15:13:28 <maxamillion> oof 15:13:36 <maxamillion> I'm open to reschedule if others are 15:13:41 <nirik> if folks want to keep the time thats fine, but note that I might not attend all the time. 15:13:54 <mboddu> dgilmore: ^^ 15:14:14 <mboddu> I am okay with rescheduling it 15:14:16 <dustymabe> nirik: in PDT? did you move? 15:14:20 <nirik> yes. 15:14:25 <dustymabe> cool 15:14:36 <dgilmore> mboddu: you run it and own it 15:14:48 <nirik> denver -> salem, or (currently) 15:15:25 <mboddu> dgilmore: yeah, but I wanted to make sure that main participants of this meeting are okay with it. 15:16:08 <puiterwijk> I just have one request: could we get the time of the meeting set in UTC? So as not to move relative to UTC with DST? 15:16:41 <dgilmore> puiterwijk: we had that and people had issues with DST changes 15:16:44 <puiterwijk> (I hate DST, and intercontinental DST is... even more fun) 15:16:55 <mboddu> nirik: if no has issues, I will reschedule this meeting to a later time(probably not in the afternoon since we have people attending from EU) 15:17:15 <nirik> mboddu: we could also do a whenisgood or better opensource one. 15:17:19 <puiterwijk> dgilmore: I think it's easier to just say "do your own DST calculations" than the DST switches intercontinentally... 15:18:15 <maxamillion> which ever the group prefers, I'm good with ... we can do a whenisgood or whatever 15:19:15 <dgilmore> puiterwijk: it was conflicts with meetings etc 15:19:53 <dgilmore> puiterwijk: as we just followed DST and did not move 15:20:05 <dgilmore> but then others had meetings that moved 15:20:39 <puiterwijk> dgilmore: Right. I get that. Just realize that if we make it not set in UTC, I will most likely be about 2 hours off for a few weeks per year... 15:21:02 <dgilmore> sure 15:21:15 <dgilmore> its worth looking at again 15:21:28 <dgilmore> I will leave it to mboddu to figure out what is best 15:21:51 <puiterwijk> :). My note was just a request 15:22:14 <mboddu> dgilmore, puiterwijk : as nirik suggested we can do whenisgood and we will see from there 15:22:35 <puiterwijk> +1 15:23:02 <mboddu> dgilmore: your turn 15:23:33 <mboddu> dgilmore: you said you have something to talk 15:24:37 <dgilmore> mboddu: yep. currently running "time rm -rf *-1704* *-1705* *-1706* *-1707*" in /mnt/koji/mash/updates/ as we have every mash going back to the start of april 15:24:50 <dgilmore> I thought bowlofeggs had enabled the auto cleanup 15:24:55 <dgilmore> we need to get that done 15:25:23 <dgilmore> allong with the EOL once its all done I will run the script to clean up the f24 signed rpms in /mnt/koji/packages 15:25:29 <mboddu> Even I thought that auto clean up is enabled 15:25:32 <dgilmore> or get mboddu or Kellin to do it 15:25:47 <mboddu> dgilmore: sure, I can do it 15:25:59 <Kellin> is it already written and just needs automation or? 15:27:22 <dgilmore> Kellin: just needs running 15:27:53 <dgilmore> https://pagure.io/releng/blob/master/f/scripts/koji-cleanup-signed.py 15:28:00 <dgilmore> it even has the f24 keys listed 15:29:15 <dgilmore> on that I have to run 15:30:11 <mboddu> #info We need to look into bodhi auto cleanup to clean the mash lock files under /mnt/koji/mash/updates/ 15:30:40 <mboddu> Okay, I think we are right on time, good we discussed open floor items 15:31:18 <mboddu> anybody got anything else? 15:31:57 <mboddu> Okay, thanks for joining guys 15:32:03 <mboddu> #endmeeting