15:36:13 <dgilmore> #startmeeting RELENG (2014-08-25) 15:36:13 <zodbot> Meeting started Mon Aug 25 15:36:13 2014 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:36:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:36:26 <dgilmore> #meetingname releng 15:36:26 <zodbot> The meeting name has been set to 'releng' 15:36:30 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson 15:36:30 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson sharkcz tyll 15:36:41 <dgilmore> #topic init process 15:36:56 * masta is here 15:37:00 * bochecha is here 15:37:07 <tyll> Hi all 15:37:07 * nirik is here 15:37:08 * sharkcz is here 15:37:47 <dgilmore> #topic #5931 [Proposal] Move new branch and unretire requests to pkgdb2 15:37:57 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5931 15:38:04 <dgilmore> pingu ping 15:38:28 <pingou> hi 15:38:52 <pingou> so I think on the pkgdb2 side it's ready 15:38:56 <dgilmore> okay 15:39:17 <pingou> I have been working on a fedmsg-pkgdb2branch that would take pkgdb2 fedmsg messages and call pkgdb2branch on pkgs01 15:39:22 <pingou> for the packages concerned 15:39:53 <pingou> the idea is then to provide a CLI to interact with pkgdb2's AdminAction, and close bugzilla tickets 15:40:11 <pingou> the git management would then be taken care of via fedmsg 15:41:00 <pingou> I guess that's it for me :) 15:41:16 <nirik> cool. 15:41:41 <tyll> #info < pingou> I have been working on a fedmsg-pkgdb2branch that would take pkgdb2 fedmsg messages and call pkgdb2branch on pkgs01 15:41:51 <tyll> #info < pingou> the idea is then to provide a CLI to interact with pkgdb2's AdminAction, and close bugzilla tickets 15:41:58 <pingou> nirik: does that answer your question about the fedmsg-pkgdb2branch project you had earlier? 15:42:33 <nirik> just one further one... 15:42:35 <tyll> #info < pingou> so I think on the pkgdb2 side it's ready 15:43:05 <pingou> the part I'm not sure about is: how to deal with fedmsg message missed? 15:43:07 <nirik> with other things we fedmsg enabled we have a catchall thing (ie, with fasClient we have a daily cron to make sure it's synced if it missed a fedmsg). Do we have any way to do that with branch requests? 15:43:15 <nirik> exactly. ;) 15:43:27 <pingou> so daily run of pkgdb2branch on all packages? 15:43:36 <nirik> not sure how costly that is... 15:43:39 <pingou> or bi-daily? 15:44:08 <nirik> there's a lot of packages. 15:44:13 <pingou> true there 15:44:14 <nirik> I guess we could see how long that takes... 15:44:41 <pingou> maybe not pkgdb2branch then but a dedicated one, sending a report 15:44:57 <pingou> (and/or fixing) 15:45:32 <nirik> yeah, could work. 15:45:38 <nirik> we can investigate that out of meeting? 15:45:41 <pingou> sure 15:45:50 * pingou has both scripts on his todo 15:45:55 <nirik> cool. 15:46:16 <nirik> I'm hoping to work on ansible migration for pkgs01.stg this week. 15:46:21 <nirik> so we could have a good place to test. 15:46:23 <pingou> cool :) 15:46:35 <pingou> let me know if I can help w/ that 15:46:42 <tyll> #info add daily cron script as fallback for missed fedmsg messages 15:46:47 <pingou> (although bochecha is likely in a better position for this) 15:47:02 <tyll> #info < nirik> I'm hoping to work on ansible migration for pkgs01.stg this week. 15:48:20 <bochecha> pingou, I can't help on this, it's my patches that nirik will review ;) 15:48:21 <dgilmore> anything else here? 15:50:45 <dgilmore> #topic #5959 Enable keep-alive connections for koji (primary and secondaries) 15:50:57 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5959 15:51:18 <dgilmore> pbrobinson is in the UK and they have a public holiday 15:51:30 <dgilmore> not sure if this was implemented and the results 15:51:46 <tyll> #info deferred due to public holidy in the UK 15:51:46 <dgilmore> #topic #5870 rawhide signing 15:51:54 <dgilmore> #undo 15:51:54 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0xb6eacd0> 15:51:55 <dgilmore> #undo 15:51:55 <zodbot> Removing item from minutes: INFO by tyll at 15:51:46 : deferred due to public holidy in the UK 15:52:05 <dgilmore> #info deferred due to public holidy in the UK 15:52:10 <dgilmore> #topic #5870 rawhide signing 15:52:17 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5870 15:52:25 <dgilmore> okay so this is kinda working 15:52:33 <dgilmore> other than sigul keeps falling over 15:52:59 <dgilmore> in particular the bridge is locking up frequently 15:53:06 <tyll> yes, therefore training someone from Europ (e.g. pingou ) to restart it would be nice 15:53:31 <dgilmore> I will talk with nirik to find someone extra 15:53:44 <tyll> also maybe some automatic monitoring would be nice 15:53:53 <dgilmore> the passphrase to unlock the nss database is shared and not per user 15:53:55 <pingou> we're two -main in Europe, anyone of us could do :) 15:54:02 <tyll> e.g. ngios alerts when the bridges are broken 15:54:03 <pingou> (if that helps of course) 15:54:15 <nirik> we could possibly setup a nagios alert yeah... 15:54:36 <nirik> it seems like secondary falls over more than primary? or about the same? 15:55:06 <nirik> tyll: how close are we to all signed 21/22? 15:55:25 <tyll> secondary is used more, so I notice it easier 15:55:40 <dgilmore> secondary is signing 3 tiems the rpms 15:55:48 <dgilmore> but i think its about equal 15:56:27 <dgilmore> its the bridge locking up, we should try get some of mitr's time and see if we can't fix it 15:56:36 <nirik> yeah. that would sure be nice. ;) 15:57:00 <tyll> nirik: Currently the script is running non stop, but I did not check how many RPMs are still unsigned - if anything is currently unsigned, there is a bug (at least in primary, not sure if someone signed everything from the mass rebuilds in the secondaries) 15:57:36 <tyll> #info sigul breaks regularly, dgilmore and nirik will find additional admins that can restart it when they are not available 15:57:37 <nirik> ok, entering freeze for 21 we need to make sure everything is signed. 15:57:46 <masta> I tried last night on ppc, but something was wrong with sigul 15:57:57 <nirik> and then we can change the autosigner to concentrate on rawhide I guess. 15:58:24 <dgilmore> tyll: I know some things are unsigned. 15:58:48 <dgilmore> I had to add --nogpgcheck when installing software onto the new laptop this weekend 15:59:11 <nirik> http://koji.fedoraproject.org/mash/branched-20140825/logs/mash.log 15:59:21 <tyll> dgilmore: since there is no gating, there might also be timing issues, especially when sigul is down 15:59:22 <nirik> ( that lists the things without the right key) 15:59:47 <dgilmore> tyll: right 16:00:37 <tyll> The "None" epoch in mash#s log makes it not nice to grep in the autosigner logs 16:02:10 <dgilmore> yeah its kinda ugly 16:02:26 <dgilmore> that we expose epochs in some places and not others is ugly 16:02:36 * nirik nods. 16:03:19 <dgilmore> nirik: i need to flush the squid caches again 16:03:32 <dgilmore> probably change pungify 16:03:38 <tyll> I will take a look to check why the packages from the mash log are not signed 16:03:39 <dgilmore> thats a differnt topic though 16:03:59 <nirik> yeah, lets get f21 frozen, then we can concentrate the autosigner on rawhide again. 16:04:03 <nirik> and gating, etc 16:05:31 <dgilmore> yep 16:05:39 <dgilmore> we get auto gating with bodhi 16:05:58 <nirik> right, f21 becomes not this tickets issue once we freeze. ;) 16:06:30 <tyll> btw. Do we now have enough space on the secondary kojis for the signed RPMs? 16:07:15 <dgilmore> not 100% sure 16:07:48 <dgilmore> vtap-fedora-nfs01.storage.phx2.redhat.com:/vol/fedora_arm/data 5.0T 4.2T 761G 85% /mnt/koji 16:07:58 <sharkcz> ppc has 1.9T free 16:08:14 <nirik> so yeah, I have been working with netapp folks to get more space free on primary... 16:08:20 <dgilmore> vtap-fedora-nfs01.storage.phx2.redhat.com:/vol/fedora_s390/data 6.5T 5.8T 666G 90% /mnt/koji 16:08:30 <nirik> I removed old mashes and reduced the number of snapshots we keep. 16:08:31 <sharkcz> s390 has 600G free, but it gets expanded +0.5T when free space is below 200G 16:08:48 <nirik> sharkcz: did we have any snapshots on it? 16:09:04 <sharkcz> nirik: on s390? nope, it's disabled 16:09:25 <nirik> yeah, I see that now, ok. 16:09:26 <sharkcz> ppc has snapshots enabled 16:09:38 <sharkcz> if I see correctly 16:09:51 <nirik> we may get migrated to a new cmode filer before too long. I am talking with storage folks about that. 16:09:58 <nirik> any outage should be hopefully pretty short. 16:13:08 <dgilmore> anything else here? 16:13:33 <tyll> not from me 16:13:41 <dgilmore> #topic #5914 Move fedmsg based blocking service to Fedora Infrastructure 16:13:46 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5914 16:13:56 <tyll> There is nothing new here due to lack of time 16:14:00 <dgilmore> tyll: I need to get you a user cert right 16:14:20 <tyll> dgilmore: yes, eventually 16:14:38 <dgilmore> okay, let us know what you need 16:14:45 <dgilmore> sadly i can not make time 16:15:17 <dgilmore> #topic Secondary Architectures updates 16:15:18 <dgilmore> #topic Secondary Architectures update - ppc 16:15:29 <dgilmore> sharkcz: so i know your having issues with pungi 16:15:35 <dgilmore> anything else going on? 16:15:42 <dgilmore> you're 16:16:03 <sharkcz> seems they were cause by the mirrorlist (or outdated mirrors), looks OK with pointers to koji/mash 16:16:19 <dgilmore> sharkcz: okay, maybe some latent yum bugs 16:16:36 <sharkcz> well, almost ok, just grub2 is not included by lorax 16:16:41 <masta> I was having some problems with more recent versions of the sigulsing_unsigned.py on ppc recently, had to checkout an older version. Hoping to bisect the issue later today 16:16:57 <dgilmore> sharkcz: that would be a lorax template update then? 16:17:22 <sharkcz> don't thiok so, as the standalone lorax run did it correctly 16:17:27 <dgilmore> masta: that seems odd. im using the current code on both arm and primary 16:17:45 <dgilmore> sharkcz: maybe a pungi bug then 16:17:56 <masta> dgilmore: the error text complained about kojihelper 16:18:07 <sharkcz> dgilmore: have a log from pungi, so we can look 16:18:15 <dgilmore> we should get it fixed asap so we do not need to try push in an updated pungi 16:18:28 <sharkcz> yes 16:19:16 <dgilmore> sharkcz: there was a self introduction on the ppc list last week 16:19:20 <dgilmore> no one has responded 16:19:49 <sharkcz> ah, right, I forgot and no one else probably saw it :-) 16:19:58 <sharkcz> :-( 16:20:23 <masta> oh yes, I think somebody nominated me to be a package sponsor... I was kinda amused by that. 16:20:24 <sharkcz> it's the guy who is/was trying to revive ppc 16:20:44 <masta> dgilmore: is this the yaboot email? 16:21:01 <dgilmore> masta: well thats one of the things he wants to work on 16:21:06 <dgilmore> but we really want it dead 16:21:19 <sharkcz> ack 16:22:05 <dgilmore> at this point 32 bit ppc would be massive massive amounts of work to revive 16:22:06 <masta> yeah I was wanting some guidance on that, firstly I don't know how I got pulled into sponsoring yaboot, or if we even want yaboot.... 16:22:30 <dgilmore> yaboot is the least of the issues 16:22:42 * nirik personally thinks a $35 arm board is more functional than a 32bit ppc from ages past, but to each their own. 16:22:43 <sharkcz> dgilmore: yes, maybe as terciary arch 16:22:43 <masta> yaboot runs on a system I have here, but I was planning to upgrade that to grub2 16:22:57 <dgilmore> sharkcz: I guess so 16:23:05 <sharkcz> but not our fight ... 16:23:28 <dgilmore> sharkcz: I think we should go to FESCo and get them to say how they think it should work 16:24:17 <sharkcz> dgilmore: would make sense probably 16:25:40 <dgilmore> I do not want to add it back to the ppc hub 16:25:46 <dgilmore> it will need to be seperate 16:25:58 <dgilmore> but how we handle it I think FESCo should say 16:26:00 * sharkcz agrees 16:26:37 <dgilmore> #action dgilmore to open ticket with FESCo on how we should handle 32 bit ppc 16:27:01 <sharkcz> I don't think one man hobby project can manage it, it's more a fulltime job 16:27:18 <sharkcz> as a full "secondary" arch 16:27:20 <dgilmore> right 16:27:50 <masta> yes 16:29:52 <sharkcz> otherwise no major issues on ppc afaik, except mariadb and its tests-suite 16:29:59 <dgilmore> okay 16:30:01 <dgilmore> :( 16:30:10 <dgilmore> #topic Secondary Architectures update - s390 16:30:17 <dgilmore> sharkcz: you're up agian 16:30:20 <dgilmore> again 16:31:10 <sharkcz> yes :-) the glibc ABI is 60+% done, will have to concentrate on the perl rebootstrap, should be easier now as I can use the old rpms (pre glibc 2.19) 16:31:21 <sharkcz> glibc ABI rebuild 16:31:59 <sharkcz> it also means that currently the f21 is broken for users 16:32:47 <sharkcz> I guess it should be fixed/rebuilt next week 16:33:12 <sharkcz> I've learned the hard way 6 month ago :-) 16:34:13 <sharkcz> also no composes yet, so we don't know is the tools still work, but hopefully the ydo 16:34:24 <sharkcz> s/is/if/ 16:35:04 <sharkcz> EOM 16:36:04 <dgilmore> okay thanks 16:36:12 <dgilmore> #topic Secondary Architectures update - arm 16:36:27 <dgilmore> with pbrobinson being away there is no update from him 16:37:13 <dgilmore> Planning to setup a box in phx early this week so we can get composes moving. I have done a TC compose from an external box and installed it. not expecting any big surprisses 16:37:29 <dgilmore> afaik the boot.iso and install dvd will not be bootable 16:37:47 <dgilmore> there is no support in mkisofs afaik to make bootable aarch64 media 16:37:58 <dgilmore> #topic Open Floor 16:38:06 <dgilmore> anyone have anything they want to discuss? 16:38:27 <masta> I want to learn about koji-shadow, could somebody show me? 16:38:49 <masta> either ppc64 or aarch64 ? 16:40:14 <dgilmore> masta: sure, you will need to work with the folks currently running it, which should be pbrobinson all both ppc and aarch64 16:40:36 <sharkcz> masta: see http://ppc.koji.fedoraproject.org/shadow/ - it has logs, config and script how it is run 16:40:36 <masta> okay 16:41:29 <dgilmore> #info fesco ticket for 32 bit ppc https://fedorahosted.org/fesco/ticket/1336 16:41:39 <sharkcz> masta: also http://sharkcz.livejournal.com/11410.html 16:41:44 <masta> thanks, koji-shadow is a gap in my knowledge of how things work on secondaries. 16:42:30 <sharkcz> masta: and read the source code :-) 16:43:37 <masta> sharkcz: yes indeed! 16:44:00 <masta> who is the maintainer of ks? 16:44:05 <dgilmore> me 16:44:18 <dgilmore> with many patches from the regular users 16:44:27 <sharkcz> like /me :-) 16:45:46 <sharkcz> dgilmore: did you see https://fedoraproject.org/wiki/Architectures/KojiShadowNG (I sent the link to releng list ~week ago) 16:45:53 <dgilmore> sharkcz: I did 16:46:02 <dgilmore> not had a chance to follow up on it yet 16:46:07 <sharkcz> ok, it's just a start 16:46:13 <dgilmore> last week was all about trying to get a TC done 16:46:19 <nirik> is that going to look at using fedmsg? 16:46:52 <sharkcz> like integrating koji-stalk (written by dwa)? probably yes 16:47:18 <nirik> it would be nicer for our fedmsg bus is builds on secondaries just happened around the time they did on primary. 16:47:21 <dgilmore> nirik: likely we will integrate koji-stalk and add a daemon mode 16:47:39 <nirik> right now there's a giant pile of them that happen at once when someone runs koji shadow 16:47:40 <sharkcz> nirik: problem is the something gets broken 16:47:49 <dgilmore> nirik: it would be nicer for many things, practically its not always possibe 16:47:58 <nirik> sure, agreed. would be nice to try. ;) 16:48:02 <sharkcz> and it happens all the time :-( 16:48:31 <dgilmore> I do want to have secondary arches more integrated into everything and make it more visable 16:48:40 <dgilmore> so breakages are more noticed 16:48:44 * sharkcz agrees 16:50:41 <dgilmore> that is a longer term roadmap thing 16:50:47 <dgilmore> but something to keep in mind 16:51:53 <sharkcz> same as allow multiple people to work on a arch, etc 16:52:38 <dgilmore> yeah 16:52:47 <dgilmore> needs to be easier 16:52:53 <dgilmore> and have full composes all teh time 16:53:24 <sharkcz> yeah, would be nice :-) good to know we have still plenty of work ahead :-) 16:53:41 <dgilmore> pretty sure we have years of work ahead 16:53:48 <nirik> yep 16:53:51 <dgilmore> for multiple people 16:54:32 <dgilmore> bochecha: ping, want to give an update of the composedb? 16:55:01 <bochecha> dgilmore, ah, I talked about it with dmach today 16:55:32 <bochecha> one thing he started working on is a module to parse/make composeinfo files 16:56:05 <bochecha> would be nice to look into using that (in releng/scripts/build_composeinfo for example), so that we share the same format as rhel 16:56:15 <bochecha> and then these composeinfo files can be used as the input for composedb 16:56:27 <dgilmore> bochecha: sure 16:56:54 <bochecha> only beaker uses composeinfo files currently, right? 16:57:19 <bochecha> (because dmach is changing the format, so we'll need to make sure users are ported over) 16:58:00 * sharkcz needs to leave ... 16:58:16 <dgilmore> sharkcz: cheers 16:58:28 <dgilmore> bochecha: he shouldn't change the format 16:58:41 <dgilmore> at least not without talking to the beaker devs 16:58:49 <dgilmore> as they are the only consumers i know of 16:58:58 <bochecha> I do think he's talking with the beaker devs 16:59:23 <dgilmore> I really do not like when internal releng folks change things and then try to push it back into Fedora 16:59:34 <dgilmore> we really should develop the changes in Fedora 16:59:37 <bochecha> but writing a module means that beaker can be ported to using it, so that everybody is using the same format 16:59:44 <bochecha> agreed 17:00:00 <bochecha> we're probably going to be using it before them, though ;) 17:00:27 <bochecha> and I do think it makes sense, the current composeinfo format is a pretty horrible hack to shoehorn the data into an ini file 17:00:44 <dgilmore> sure. not syaing its perfect 17:00:55 <dgilmore> but we have a good platform to effect change 17:00:58 <dgilmore> lets do it right 17:01:10 <bochecha> (in fact, I wanted to talk to about doing that, before dmach even told me about it today) 17:02:02 <dgilmore> okay 17:02:03 <bochecha> so, doing it right means 1. opensourcing this module, 2. getting it into fedora, and 3. porting build_composeinfo to it, right? 17:02:11 <dgilmore> yep 17:02:18 <bochecha> well, I'll be doing that this week then :) 17:02:19 <dgilmore> making sure beaker devs are on board 17:02:23 <bochecha> of course 17:02:31 <dgilmore> they do import fedora into internal beaker 17:02:43 <dgilmore> its the only reason we make the .composeinfo file in fedora 17:02:55 <bochecha> well, I'm hoping to use it as the input for composedb 17:03:06 <bochecha> as it contains most of the stuff that I'd need in it 17:03:06 <dgilmore> :) that is fine 17:03:47 <dgilmore> 5 17:04:34 <dgilmore> bochecha: we can add whats missing 17:04:44 <bochecha> yes 17:04:56 <bochecha> and that's pretty much it for me, I'm still wondering if/how we could reuse internal work on this, and at the same time hacking on a quick poc for a composedb, in case we decide not to reuse it 17:05:22 * bochecha isn't sure how much he is allowed to say at this point, given that the internal work is not FOSS yet :x 17:05:27 <dgilmore> bochecha: okay, depends how modular thinsg are I guess 17:05:38 <bochecha> yeah 17:05:57 <dgilmore> so that things can be extended/swapped out as need be for different use cases 17:06:28 <dgilmore> I am hoping that we can get the composedb in place for F22 17:07:01 * bochecha gets cranking 17:07:55 <dgilmore> bochecha: having frontend/backend totally seperate is fine also 17:08:09 <bochecha> sure 17:09:30 <dgilmore> I have mentioned a few times i think having pkgdb/bodhi/koji/composedb as one integrated user interface would be awesome 17:09:43 <dgilmore> and get rid of the seperate versions 17:10:23 <bochecha> fedora community? :P 17:10:32 <dgilmore> ive said it before 17:10:50 <dgilmore> have it completely replace one thing 17:10:54 <dgilmore> then add the rest 17:11:44 * bochecha has to go soon 17:11:53 <bochecha> (and it seems we lost everyone else ^_^) 17:13:01 <dgilmore> yeah lets wrap up 17:13:16 <dgilmore> I just wante to get some more visability on the composedb 17:13:34 <dgilmore> will close in a minute unless someone has something else 17:13:42 <bochecha> right, forgot the link: moz-action:switchtab,https://fedoraproject.org/wiki/Compose_Database 17:13:55 <bochecha> huh? what's that moz-action stuff... 17:14:02 <bochecha> #link https://fedoraproject.org/wiki/Compose_Database 17:14:18 <dgilmore> no clue 17:14:24 <dgilmore> #endmeeting