14:51:53 <mboddu> #startmeeting RELENG (2017-03-13)
14:51:53 <zodbot> Meeting started Mon Mar 13 14:51:53 2017 UTC.  The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:51:53 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:51:53 <zodbot> The meeting name has been set to 'releng_(2017-03-13)'
14:51:54 <mboddu> #meetingname releng
14:51:54 <zodbot> The meeting name has been set to 'releng'
14:51:54 <mboddu> #chair dgilmore nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu Kellin
14:51:54 <zodbot> Current chairs: Kellin dgilmore masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll
14:51:54 <mboddu> #topic init process
14:52:22 <maxamillion> .hello maxamillion
14:52:23 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
14:52:38 * masta waves hello
14:53:07 <maxamillion> masta: o/
14:53:28 <dgilmore> hi
14:53:35 * nirik is sort of here, and sort of busy dealing with email
14:53:54 <maxamillion> masta: hey, I met a dude from the DFWUUG at SCaLE 15x ... he said to say hello :)
14:53:58 <mboddu> Sorry for starting the meeting late, but lets start
14:54:03 <mboddu> #topic #6663 Create another new tag for Modularity
14:54:12 <mboddu> #link https://pagure.io/releng/issue/6663
14:54:59 <dgilmore> my issues with the proposed model for dealing with adding packages to the tags is that blocking them becomes super difficult
14:56:15 <dgilmore> threebean: are you here?
14:58:45 <dgilmore> I will give some info
14:59:00 * mboddu sent an irc message on #fedora-releng to threebean
14:59:07 <dgilmore> adding all the packages to a single tag makes running the current sync process easy
14:59:56 <dgilmore> but when packages get removed froma  module they will need to be blocked in the module, and all subsequent modules. afaik each module will be created from scratch and not have inhertance
15:00:09 <dgilmore> inheritance adds cost to the modules
15:00:24 <dgilmore> mboddu: he is in here pinging him is sufficuent
15:01:46 <dgilmore> there is also potentially a lot of issues with other pieces of the processes
15:02:01 <dgilmore> things like koji-gc on the modules
15:02:13 <dgilmore> and signing of packages etc
15:02:56 <dgilmore> but the immediate concerns with setting up things by just adding everything to a base tag is how we deal with blocking, and then unblocking if necessary
15:03:22 <dgilmore> threebean: I think that we need to have a presentation on what modules are and how they should work
15:03:49 <dgilmore> because we can not go off and play cowboy in implementing them
15:04:10 <dgilmore> anyway thats the quick outline of my immediate concerns
15:04:31 <dgilmore> without threebean here I do not know we can do or say too much
15:05:05 <mboddu> dgilmore: okay, I can schedule a meeting with threebean, what do you say?
15:05:08 <dgilmore> #info main concern over the current model is how we deal with blocking and unblocking packages as needed
15:05:26 <dgilmore> mboddu: please don't let me talk to him
15:05:40 <dgilmore> #info there are many concerns over modularity
15:06:04 <dgilmore> #action dgilmore to talk to threebean and make sure he is here next week or will setup another meeting
15:06:21 <dgilmore> lets move ob
15:06:23 <mboddu> dgilmore: okay
15:06:53 <mboddu> #topic #6645 Make sure OSTree repos are sync'ed correctly after the masher completes
15:07:05 <mboddu> #link https://pagure.io/releng/issue/6645
15:07:21 * sharkcz is back from another meeting
15:08:59 <mboddu> puiterwijk: what do you suggest, using fedmsg by replacing cron job seems like a better option?
15:09:12 <masta> mboddu, there is a special order to rsync an ostree
15:09:40 <mboddu> masta: special order?
15:10:03 <masta> mboddu, yeah so the ostree is synched atomicly
15:10:07 <dgilmore> there is another issue open to replace that script
15:10:18 <masta> but that looks to not be the issue here
15:10:53 <nirik> well, I don't think we are using any particular order currently
15:11:02 <dgilmore> https://pagure.io/releng/issue/6692 is the issue for rewriting how we sync out updated content
15:11:54 <nirik> we also have updates and updates-testing in the same tree don't do? or do we?
15:12:07 <dgilmore> we do
15:12:23 <puiterwijk> So, the problem is that we sync objects before refs. But if the new refs get synced in by bodhi compose while the updates-sync is running, it shows exactly what this ticket is about
15:12:30 <nirik> so an updates could finish, sync and then a updates-testing finishes, syncs and things update poorly
15:12:30 <dgilmore> longer term we will have multiple arches and likely multiple releases in teh same tree also
15:12:59 <masta> https://paste.fedoraproject.org/paste/utfvVbjzq0lYFy2xGdr96F5M1UNdIGYhyRLivL9gydE=  <-- that is how I sync the ostree, fyi
15:13:30 <nirik> that would help, but we would also need to block and only update one stream at a time.
15:13:55 <dgilmore> https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/bodhi2/backend/files/fedora-updates-push#n144
15:14:02 <dgilmore> that is how we currently do it
15:14:21 <dgilmore> I honestly think the best way to work with things is to make a new ostree everytime
15:14:34 <dgilmore> then push the new commit into the main repo using ostree itself
15:14:54 <puiterwijk> dgilmore: ostree doesn't have a push mode last time I asked
15:14:55 <masta> sounds good to me
15:15:16 <puiterwijk> (I think at least)
15:15:52 <puiterwijk> Also, that would mean we reset the version number to 1 every time.
15:16:00 <dgilmore> puiterwijk: I was told it did
15:16:24 <dgilmore> puiterwijk: we probably should work with dustymabe and get some RFE's in
15:16:35 <puiterwijk> dgilmore: okay. So let's say we trust your memory better than mine (safe assumption), I still don't think an entirely new tree everytime is going to cause upgrade/version problems
15:16:55 <puiterwijk> s/don't/do/
15:16:57 <dgilmore> #action dgilmore and puiterwijk to work with dustymabe to file RFE's for ostree enhancement as needed to improve things
15:17:13 <dgilmore> puiterwijk: I am sure there will be issues
15:17:19 <nirik> this is definitely something to keep in mind when trying to revamp the syncing...
15:17:22 <Kellin> dgilmore: should I be involved on that too since I'm assigned/working on the updates for alt-arches which might be similarish?
15:17:49 <dgilmore> Kellin: maybe, there is no alt arch supoort in any of our ostree work today
15:18:08 <dgilmore> atomic-host today is x86_64 only
15:18:28 <puiterwijk> dgilmore: do note: if there's an ostree push mode, we could just run an ostree push from /mnt/koji/mash to /mnt/koji, and that should be it
15:18:42 <dgilmore> puiterwijk: sure
15:18:42 <puiterwijk> Aka, then we shouldn't even need to do new trees everytime.
15:18:48 <Kellin> dgilmore: I mean in the larger context of working on 6692 and infra 3983 ( nirik and I briefly touched on this earlier this morning, is this in scope of those?)
15:19:09 <dgilmore> puiterwijk: well I think in order to do what we need we should do a new tree everytime
15:19:22 <puiterwijk> Let's ask Dusty
15:19:27 <dgilmore> Kno idea what those issues are without looking
15:19:59 <dgilmore> Oh i see
15:20:05 <dgilmore> gahh can not type
15:20:09 * pbrobinson is here now
15:20:24 <Kellin> dgilmore: it's Monday, typing difficult is +5 today
15:20:43 <dgilmore> Kellin: sure. I do not remeber ticket numbers. the syscing ticket would be in scope of changes to how we sysnc ostrees
15:21:24 * Kellin nods.  puiterwijk had pinged me this morning with regard to some issues that fell into these two , which is how I found them today
15:22:06 <mboddu> dgilmore: can you include me as well in the discussion with puiterwijk and Dusty?
15:22:18 * nirik notes all the issues are just not solveable unless we reorg where we put things.
15:23:22 <dgilmore> mboddu: it will be all open
15:23:57 <mboddu> dgilmore: okay, then I can follow the discussion on #fedora-releng
15:24:16 <mboddu> lets move on
15:24:17 <dgilmore> lets move on for now?
15:24:35 <mboddu> #topic Alternative Architectures updates
15:24:45 <mboddu> sharkcz: any updates?
15:25:34 <sharkcz> nirik got a reply to his questions about the new fedora lpar, but seems the setup from engops hasn't been finalized yet
15:25:50 <nirik> There's a bit of news on s390 builders... the admins there want to setup a meeting to talk about any provisioning questions and...
15:25:58 <nirik> they have run into some network issues yeah
15:26:27 <mboddu> nirik: fun times ;)
15:26:55 <mboddu> nirik: when do you think everything will be done(guesstimate)?
15:27:33 <nirik> no idea.
15:27:47 <mboddu> nirik: okay
15:28:06 <nirik> I've been hoping for a while now, but it's impossible to estimate fixing network problems when we don't know what the solution is. ;)
15:28:55 <mboddu> dgilmore: I think s390 wont be in primary koji for 26 release, or is it still possible?
15:30:16 <mboddu> #info nirik planned to talk to admins regarding s390 builders and they ran into network issues.
15:31:15 <mboddu> okay, lets move on
15:31:30 <mboddu> #topic Open Floor
15:31:44 <dgilmore> mboddu: it is too late now
15:32:15 <mboddu> dgilmore: okay
15:32:47 <dgilmore> if we do not have a alpha compose by tomorrow then f26 will have to slip
15:33:03 <dgilmore> but at this point it is too late to slip s390 infor f26
15:33:20 <mboddu> sorry, we were not able to cover most of the topics since I started it late, but has anybody has anything to share?
15:33:33 <mboddu> dgilmore: okay, thats what I thought
15:34:26 <dgilmore> #info f26 Alpha compose needs to come in by tomorrow if we want to avoid a slip
15:34:47 <dgilmore> mboddu: how is the cleanup for mass rebuild coming along?
15:35:02 <dgilmore> mboddu: did you go through the logs to see what failed?
15:35:29 <dgilmore> in order to see if there was things we did that could be done better to avoid issues
15:37:02 <mboddu> dgilmore: you mean mass branching?
15:37:14 <dgilmore> mboddu: no
15:37:17 <dgilmore> mass rebuild
15:37:34 <dgilmore> have you been checking on thew rebuild status?
15:37:50 <dgilmore> packages that are marked as needs-rebuild and thatfailed
15:37:59 <dgilmore> have the numbers been going down?
15:38:28 <mboddu> dgilmore: okay, yeah they went down, but still there are some remaining
15:38:41 <Kellin> dgilmore: mboddu is there a dashboard for that?
15:39:07 <mboddu> Kellin: http://dl.fedoraproject.org/pub/alt/mass-rebuild/f26-need-rebuild.html http://dl.fedoraproject.org/pub/alt/mass-rebuild/f26-failures.html
15:39:46 <dgilmore> Kellin: not really a dashboard
15:40:03 <dgilmore> but there is some scripts we run to update the current status
15:40:32 <mboddu> Kellin: as dgilmore said, its not a dashboard but we can see what we need
15:41:17 <mboddu> dgilmore: can we move this discussion to #fedora-releng, as the time is up?
15:41:19 <dgilmore> we could probably do with having something in the neds-rebuild list but not failed
15:41:34 <dgilmore> it will mean that the package failed to submit to koji for some reason
15:42:08 <dgilmore> we could also do with something to check the bugs filed for packages and if the bug is still open but a build was completed
15:42:14 <dgilmore> sure
15:42:32 <dgilmore> mboddu: I was wanting to check where its all at
15:42:42 <dgilmore> and its an appropriate open floor topic
15:42:43 <mboddu> dgilmore: okay
15:42:53 <dgilmore> but we can continue it elsewhere
15:43:16 <mboddu> dgilmore: sure
15:43:20 <dgilmore> 753 failed builds:
15:43:29 <dgilmore> 1024 packages need rebuilding:
15:45:00 <mboddu> #info From mass rebuild as per 2017-03-13 15:27:16 - 753 failed builds and 1024 packages need rebuilding
15:45:08 <mboddu> thank you all for joining
15:45:14 <mboddu> #endmeeting