15:34:28 <dgilmore> #startmeeting RELENG (2015-07-27)
15:34:28 <zodbot> Meeting started Mon Jul 27 15:34:28 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:34:28 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:34:35 <dgilmore> #meetingname releng
15:34:35 <zodbot> The meeting name has been set to 'releng'
15:34:35 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
15:34:36 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
15:34:38 <dgilmore> #topic init process
15:34:59 * sharkcz is here
15:35:28 <nirik> morning
15:35:54 <dgilmore> not sure how many are here today
15:36:19 <dgilmore> lmacken: are you here?
15:37:04 <dgilmore> http://paste.fedoraproject.org/248620/11405143/ \0/
15:37:32 <nirik> woah, worked? how is that possible! :)
15:37:37 <dgilmore> even though repoclosure failed the last pungi compose ran to completion
15:37:58 <nirik> is it intended to use this for alpha?
15:38:01 <dgilmore> nirik: I synced the package sets to be installed to the kickstarts
15:38:07 * masta is here
15:38:09 <dgilmore> nirik: not at all
15:38:19 <nirik> ok. whew. already tons of moving parts. ;)
15:38:42 <tyll> Hi there
15:38:44 <dgilmore> https://kojipkgs.fedoraproject.org/compose/23/Fedora-23-20150727.n.0/
15:38:47 <dgilmore> is the compose
15:38:52 <masta> dgilmore: congrats on the working compose link above =)
15:39:03 <dgilmore> nirik: it will be Beta at the earliest
15:39:26 <masta> dgilmore: the repoclosure fails internally, it's probably a trivial fix.
15:39:28 <dgilmore> now we can look at what is missing and what needs changed
15:40:20 <masta> we really do want to have those repoclosures, and fix them if there are any findings.
15:40:40 <dgilmore> masta: looking at the logs its broken deps
15:41:04 <dgilmore> package: pdns-3.4.5-3.fc23.i686 from repoclosure-Cloud.i386
15:41:04 <dgilmore> unresolved deps:
15:41:05 <dgilmore> libmbedtls.so.9
15:41:08 <dgilmore> for instance
15:41:41 <masta> yes, I seem to recall that now. Ok, so we can work towards solving that this week.
15:41:54 <masta> dgilmore: again, congrats man!
15:43:32 <dgilmore> not sure we have much to discuss today
15:43:48 <dgilmore> we have one ticket and lmacken seems not here
15:44:01 <dgilmore> we have secondary arch status and it seems they are not here
15:44:15 <dgilmore> So I will give a quick update
15:44:15 <nirik> I really really want to discuss bodhi2.
15:44:21 <dgilmore> #topic pungi update
15:44:27 <dgilmore> nirik: me too
15:44:38 <dgilmore> pungi finally now runs
15:45:09 <dgilmore> need to sit down and look at adding livecds
15:45:22 <dgilmore> and figuring out all the changes we will need to make
15:45:41 <masta> dgilmore: is that the livemedia-creator stuff ?
15:45:41 <dgilmore> If I can get rawhide running I will look at changing rawhide over
15:45:49 <dgilmore> masta: thats part of it
15:46:25 * nirik notes he fixed rawhide compose. Was broken by new rpm landing on friday night.
15:46:55 <dgilmore> nirik: fun, what was the fix?
15:47:05 <dgilmore> I did build a new deltarpm over the weekend
15:48:26 <nirik> createrepo_c, drpm, and a few other packages in the base compose chroot had broken deps on old librpm
15:48:31 <nirik> so I rebuilt those.
15:48:46 <dgilmore> okay
15:50:17 <dgilmore> potential for big changes in rawhide this week
15:50:22 <nirik> oh?
15:50:31 <dgilmore> getting in pungi etc
15:50:39 <nirik> ah yeah.
15:50:51 <nirik> that should mostly only affect the install path tho right?
15:50:53 <dgilmore> #info potential for big changes in rawhide this week
15:50:54 <nirik> not existing users?
15:51:09 <dgilmore> nirik: it may break mirrormanager
15:51:29 <dgilmore> but yeah should not effect users otherwise
15:51:33 <nirik> how so? directory tree changes?
15:51:50 <masta> nirik: probably
15:51:55 <dgilmore> nirik: yeah. the tree will look something like a release
15:52:01 <dgilmore> thinsg will move around
15:52:16 <dgilmore> nirik: so will have to get with adrian and see what may happen
15:52:28 <nirik> yeah, ok.
15:52:35 <masta> Is there a git for mirror manager? I've never bothered to read it over.
15:52:41 <nirik> yep
15:53:04 <masta> pagure.io ?
15:53:08 <nirik> https://github.com/fedora-infra/mirrormanager2
15:53:15 <nirik> but it could move to pagure, sure.
15:53:23 * masta goes off looking in the usual palaces
15:53:30 <masta> oh cool, ty for the link nirik
15:53:33 <dgilmore> palaces?
15:53:38 <dgilmore> wow
15:53:40 <dgilmore> :P
15:53:47 <masta> dgilmore: hehe sry my bad, places.
15:54:05 <dgilmore> anyway thats the brief update fo pungi
15:54:11 <dgilmore> #topic bodhi2
15:54:19 <dgilmore> nirik: so bodhi2
15:54:26 <dgilmore> can we get it deployed today?
15:54:36 <nirik> dgilmore: in theory yeah. ;)
15:54:53 <nirik> but...
15:55:08 <nirik> I'm not sure the status of api users (there's some things using the old bodhi1 api)
15:55:19 <nirik> and we really need to finish the current bodhi1 updates push
15:55:50 <nirik> If we do not do it today, next window I think is after flock.
15:56:00 <dgilmore> since we are supposed to enable bodhi for f23 tomorrow
15:56:05 <nirik> right.
15:56:09 <masta> yikes!
15:56:27 <nirik> ideally, I would like to get the current stuck push to finish and do another bodhi1 one.
15:56:36 <nirik> so we are mostly caught up, then switch to bodhi2.
15:56:40 <nirik> but dunno
15:57:04 <nirik> I have the bodhi2 prod instances already created.
15:57:17 <nirik> we just need to install some things, deal with the database migration and adjust proxies.
15:57:18 <masta> nirik: yeah, I guess any pending push need to finish.
15:57:33 <dgilmore> having the in process push finished seems a must
15:57:44 <dgilmore> unless we can finish it in bodhi2
15:58:12 <nirik> I doubt that...
15:58:28 <dgilmore> I doubt it also
15:58:45 <masta> yeah, I 3rd that sentiment
15:58:48 <nirik> so, I guess lets wait and discuss with lmacken when he's in and decide then out of meeting?
15:58:59 <dgilmore> nirik: sounds like a plan
15:59:03 <nirik> do we need to run things by fesco or qa or the like?
15:59:13 <dgilmore> would be nice to know what apps may break with the move
15:59:19 <dgilmore> nirik: maybe qa
15:59:26 <nirik> I am pretty sure fedora-easy-karma will.
15:59:30 <masta> yeah, qa might be a consumer of api stuff
15:59:30 <dgilmore> but I do not think we need to run it by fesco
15:59:42 <nirik> bodhi2 should be ready for talking with taskotron.
16:00:23 <nirik> oh, another possible hickup
16:00:25 * nirik checks something
16:00:43 <nirik> yeah, I think we are doomed.
16:00:54 <dgilmore> nirik: ?
16:01:00 <nirik> bodhi2 isn't yet in main fedora and thus maintainers won't have bodhi2-client...
16:01:08 <nirik> unless bodhi1 client can talk to bodhi2 api
16:01:47 <masta> uh!
16:01:53 <dgilmore> nirik: that would be bad
16:02:07 <dgilmore> fedpkg probably will need updated also
16:02:27 <nirik> yeah, I think we are punt until after flock.
16:03:01 <masta> this seems like a set of things that would expand/lengthen the task beyond what is doable today =(
16:04:29 <nirik> yeah, lets punt until after flock and pick a real actual hard date... and organize it more.
16:04:41 <dgilmore> sounds good
16:04:46 <dgilmore> sucks but is wiser
16:04:58 * nirik nods.
16:05:04 <dgilmore> #info set a hard date for migration after flock
16:05:13 <masta> are those bohdi2-client things and fedpkg things... are those things that need to have a flag day?
16:05:32 <nirik> something like aug 19th
16:06:13 <dgilmore> masta: likely, so we probably want to try do the ssl certs etc
16:06:26 <dgilmore> and the lookaside migration
16:06:49 <masta> dgilmore: yes, that is why I wonder if the work can be rolled into the ssl cert stuff ;-)
16:07:01 * lmacken rolls in from
16:07:05 <lmacken> an appointment
16:07:07 <nirik> hey lmacken. ;)
16:07:17 <dgilmore> but the ssl side we need to figure out the long term ca plan
16:07:21 <lmacken> so yeah, the API users and consumers of python-fedora still need work.
16:07:34 <lmacken> I wanted to get that done last week, but 2 big outages and 2 fires hindered that
16:07:45 <nirik> yeah. ;(
16:07:51 <dgilmore> lmacken: :(
16:07:53 <lmacken> it can talk to taskotron, but I don't think taskotron can talk to it at the moment, until python-fedora gets ported
16:08:18 <nirik> lmacken: so lets plan for mid aug (and pick a date soon) and then work toward that.
16:08:31 <lmacken> but I successfully submitted an update and got the masher to spit out repos, emails, etc...
16:08:35 <lmacken> nirik: alright
16:09:02 <nirik> that will give time to get bodhi2 in fedora, look at python-fedora changes, schedule outage, tell everyone it's switching, etc.
16:09:05 <lmacken> that's a much safer bet
16:10:34 * nirik nods.
16:15:53 <nirik> anything else today?
16:16:02 <nirik> oh, dgilmore: did you get the fedora-24 keys finally done?
16:17:01 <dgilmore> nirik: I did
16:17:17 <dgilmore> I need to give people access and push out some updated fedora-repos builds
16:17:18 <nirik> if you get me access I can start rawhide signing. ;)
16:17:37 <nirik> related there was a request to include other releases keys in fedora-repos.
16:17:43 <nirik> ie, all currently supported one.
16:18:03 <nirik> https://bugzilla.redhat.com/show_bug.cgi?id=1246701
16:18:04 <dgilmore> nirik: I saw that and I kinda think it is a bad idea
16:18:11 <dgilmore> the use cases seem odd
16:18:13 <nirik> yeah, I am not sure either...
16:18:26 <nirik> I don't think it does any harm to have the non eol ones.
16:18:31 <nirik> but yeah.
16:18:51 <tyll> the keys are already in the mock package to be used in mock chroots
16:18:56 <dgilmore> keys we do provide in the bug either
16:19:06 <dgilmore> as i will be pushing f24 keys to f22
16:19:07 <nirik> tyll: yeah. which seems a weird place for them.
16:19:25 <tyll> yes, but it is the same usecase - being able to securely create chroots
16:19:25 <dgilmore> nirik: I was pretty shocked when I saw they did that
16:19:59 <tyll> I was happy that they finally got there and gpgcheck is enabled by default in current mock ;-)
16:20:47 <nirik> anyhow, I don't care about adding the non eol ones, seems ok to me.
16:20:48 <tyll> nevertheless IMHO it is a good idea to provide trusted GPG keys for all releases - maybe in a generic keys package so they are not installed/used by accident
16:20:54 <masta> huh... was not aware mock did that.
16:21:11 <nirik> tyll: well, providing the eol one could encourage people to use eol releases...
16:21:50 <nirik> anyhow, did not mean to derail the meeting. ;)
16:22:01 <dgilmore> I think changing would add to package churn a bit
16:22:27 <dgilmore> the only way to avoid it would be to have more keys sitting there for new releases before they come out
16:22:45 <dgilmore> but then you still need updates to remove eol keys
16:25:07 <nirik> perhaps there's a way they could use the mock ones in their use case...
16:25:14 <nirik> (but that would need repo changes I guess.
16:25:42 <dgilmore> nirik: maybe
16:25:55 <dgilmore> not sure it is overly harmful
16:26:06 <dgilmore> just seems like some extra something
16:26:18 <dgilmore> does anyone have anything else?
16:26:21 <tyll> afaik nearly nobody removes gpg keys from the RPM database, therefore I guess it does not make a big change if the eol gpg key is removed from the filesystem
16:27:16 <tyll> I was wondering what I should do with the FTBFS package that are depenecies for a lot of packages
16:27:25 <tyll> retiring the all seems to be a
16:27:28 <tyll> bit harsh
16:27:42 <nirik> tyll: which ones?
16:28:15 <nirik> these are in rawhide only? or 23 + rawhide?
16:28:31 <tyll> e.g. typesafe-config
16:28:35 <tyll> f23 + rawhide
16:29:12 <tyll> and the packages depending on prelink
16:29:28 <tyll> but for prelink it seems people are now working on  a solution
16:29:41 <dgilmore> tyll: we probably need to make the packages depending on prelink not depend on it
16:29:45 <nirik> yeah, the execstack thing
16:29:54 <nirik> dgilmore: they use execstack at build time
16:29:56 <dgilmore> and typesafe-config probably needs to be evaluated
16:30:03 <dgilmore> nirik: right
16:30:11 <dgilmore> but do they have to?
16:30:23 <nirik> the currently thought is to get binutils or something to carry execstack
16:30:24 <tyll> dgilmore: I started with aide, which was easy
16:30:28 <nirik> some of them might not
16:31:42 <dgilmore> I think libreoffice is one of them
16:32:31 <tyll> also there is dogtag-pki that depends on FTBFS package esc - is dogtag-pki the pki pkg we plan to use in the future?
16:32:50 <nirik> we looked at it, but frankly I thought it was over complex for our needs.
16:32:54 <nirik> but that could just be me
16:34:41 <dgilmore> tyll: we have no real idea
16:34:46 <dgilmore> we did look at it
16:35:03 <dgilmore> tyll: freeipa depends on dogtag-pki
16:35:15 <dgilmore> ot uses it for the backaned
16:35:32 <dgilmore> and if we based fas3 on freeip then I think we should really look at it
16:36:53 <dgilmore> but that is a bit out of scope for here
16:37:24 <dgilmore> do we need to re-evaluate how we retire orphans and FTBFS?
16:37:45 <dgilmore> or do we need to engage in a different way?
16:38:08 <tyll> it works quite good for most packages imho
16:38:33 <tyll> but sometimes it seems that nobody cares about important packages a lot of pkgs depend on it
16:38:57 <masta> yeah, it seems the impact of dependencies is glossed over.
16:39:01 <nirik> I could try and look at typesafe-config, but I have a bunch of things to do today already. ;(
16:39:09 <tyll> maybe more  FTBFS nag mails might help, e.g. once a week or so
16:39:47 <tyll> and for orphans it seems that a lot of people just orphan pkgs when they should actually be retired because the original maintainer already knows that it is obsolete
16:40:15 <dgilmore> tyll: I honestly think that is the absolute worst thing to do
16:40:19 <masta> tyll: the paradox is if more nags are sent, more likely folks ignore the nags.
16:40:25 <dgilmore> tyll: it will get seen and noise and spam
16:40:28 <dgilmore> and ignored
16:41:17 <masta> tyll: it's a signal: noise thing.
16:41:31 <tyll> the other possiblity would be to retire pkgs more often
16:41:54 <tyll> then it is not a big amount of pkgs at a certain time
16:42:25 <masta> tyll: how often would you propose ?
16:42:37 <dgilmore> tyll: again I think people will just ignore it
16:42:46 <dgilmore> there needs to be a balance
16:43:22 <dgilmore> between cleaning things up and keeping the noise down
16:43:36 <dgilmore> people get emails daily for broken deps
16:43:42 <dgilmore> and they ignore them
16:44:59 <tyll> masta: I do not yet have an actual idea, the problem with FTBFS pkgs is that they are only identified with mass rebuilds and not regularly
16:45:16 <dgilmore> tyll: not at all true
16:45:29 <dgilmore> we have koeshi that does thousands of builds a week
16:45:29 <nirik> koschei can identify some that are tracked... like the php and perl stacks
16:46:07 <dgilmore> we need to do something
16:46:13 <dgilmore> I just do not know what it is
16:46:25 <dgilmore> it seems that as things are they do not work really well
16:47:21 <tyll> koshei is only a recent addition afaik - nevertheless, using it as a help for faster action on FTBFS packages sounds good
16:48:02 <nirik> yeah, perhaps we can discuss at flock and brainstorm some ideas.
16:48:05 <nirik> tyll: you will be at flock?
16:48:12 <tyll> nirik: yes
16:48:16 <nirik> excellent.
16:48:24 <tyll> discussing it there sounds good to me as well
16:48:41 <masta> I keep wondering if there is a way to have builds marked as important, maybe tagged somehow, and if any things FTBS that have impact of an important build... it signals an alert.
16:49:01 <masta> kochei seems a good fit for that kind of thing
16:49:05 <dgilmore> masta: that would come down to implementing the rings proposal
16:49:19 <nirik> well, we do have 'critical path'
16:49:20 <dgilmore> and raising alarms when things in ring 0 fail
16:49:22 <tyll> are there still the critical paht pkgs?
16:49:31 <dgilmore> there are
16:49:38 <dgilmore> but I hardly ever update it
16:49:46 <dgilmore> I think its not setup for f23
16:51:49 <nirik> perhaps that work could be leveraged for rings
16:51:50 <masta> critical path sounds like a great idea. Maybe have 'important path' too, or whatever notion of rings...
16:52:08 <dgilmore> #info discussions at flock are needed on dealing with orphans and FTBFS better
16:52:17 <masta> dgilmore: yeah, the bummer of tags and stuff  like that is maintaining the metadata
16:52:25 <masta> curating
16:53:28 <masta> tyll: thanks for bringing this up, I guess we need to think about it more.
16:53:56 <dgilmore> anything more to discuss here today on this?
16:54:00 <dgilmore> or anything else?
16:54:36 * tyll has nothing
16:54:42 * nirik has nothing off hand.
16:54:43 <dgilmore> #endmeeting