17:04:52 <dgilmore> #startmeeting RELENG (2015-02-16)
17:04:52 <zodbot> Meeting started Mon Feb 16 17:04:52 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:04:52 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:05:02 <dgilmore> #meetingname releng
17:05:02 <zodbot> The meeting name has been set to 'releng'
17:05:02 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson
17:05:02 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson sharkcz tyll
17:05:03 <dgilmore> #topic init process
17:05:13 <dgilmore> sorry for the late start
17:05:45 <tyll> No problem
17:06:58 <dgilmore> eye doctor took way too long
17:07:19 <nirik> morning
17:08:33 <Corey84> .fas corey84
17:08:34 <zodbot> Corey84: corey84 'Corey84' <sheldon.corey@gmail.com>
17:08:57 <dgilmore> hi Corey84
17:09:04 <dgilmore> #topic #5931 [Proposal] Move new branch and unretire requests to pkgdb2
17:09:14 <dgilmore> pingou: where are we now?
17:09:18 * Corey84 attempting to learn this stuff so kinda hanging out in back row
17:09:43 <dgilmore> Corey84: :) people are always welcome. feel free to ask questions
17:09:55 <Corey84> im not shy :)
17:09:58 <pingou> dgilmore: FESCo decided that it'd be better if the request for Fedora branch also had the 7 days period that we have for EPEL branch requests
17:10:04 <pingou> this is open for review
17:10:21 <dgilmore> pingou: okay. I missed last weeks FESCo meeting :(
17:10:22 <pingou> (and it is not applicable if the user requesting the branch has approveacls on the package)
17:10:41 <pingou> once this is merged, I'll do a new release in stg and call for more testing :)
17:10:47 <dgilmore> cool
17:12:10 <dgilmore> any other update?
17:13:09 <dgilmore> #topic #6016 Use fedpkg-minimal in Fedora buildroots
17:13:10 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6016
17:13:30 <dgilmore> fedpkg-minimal is in fedora
17:13:44 <nirik> cool. when can we turn it on? ;)
17:13:48 <dgilmore> just need to update the groups in koji
17:14:02 <dgilmore> and make sure that the package is available to the buildroot
17:14:43 <nirik> perhaps rawhide first, then move to others?
17:14:52 <nirik> since it needs an override or in stable?
17:15:00 <dgilmore> #action dgilmore update the srpm creation buildroot to use fedpkg-minimal
17:15:14 <dgilmore> ill do rawhide and branched
17:15:15 <tyll> epel 5 would be nice to be able to retire some broken pkgs
17:15:21 <nirik> hum.
17:15:30 <nirik> .branches fedpkg-minimal
17:15:31 <zodbot> nirik: f20 f21 f22 master
17:15:45 <dgilmore> tyll: right we can get rid of the ancient fedpkg/fedora-packager in epel5
17:16:06 <dgilmore> we will have to get epel branches and builds done
17:16:09 <nirik> it was only built for rawhide so far.
17:16:26 <dgilmore> that will need fixing also
17:16:34 <nirik> well, rawhide and branched.
17:16:40 <nirik> the one build is tagged with both. :(
17:16:51 <nirik> https://koji.fedoraproject.org/koji/buildinfo?buildID=609961
17:17:10 <dgilmore> nirik: that is expected
17:17:15 <nirik> ok
17:17:21 <dgilmore> I believe it was built pre branch
17:17:39 <dgilmore> and when we isolate the tag we do so by tagging all the builds in f22 into f23
17:19:05 * pbrobinson is here
17:19:41 <dgilmore> #topic #5963 Orphaned vulnerable packages in EPEL
17:19:49 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5963
17:20:05 <dgilmore> tyll: where does this stand?
17:20:38 <tyll> I am currently waiting on fedpkg-minimal for EPEL so I can retire the packages with broken deps
17:21:52 <tyll> iirc there are some dependencies for fedpkg or fedora-packager that currently have broken deps
17:21:54 <dgilmore> okay
17:22:17 <tyll> it is easier to wait till I can cleanup everything completely than trying to pick the pkgs that won't break anything
17:22:41 <dgilmore> indeed
17:23:05 <dgilmore> #topic #6027 secondary arch old mash trees cleanup
17:23:13 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6027
17:23:20 <dgilmore> pbrobinson: any update here?
17:23:34 <nirik> we should add something to do this to the rawhide/branched scripts.
17:23:51 <pbrobinson> dgilmore: on my list for this week, was travelling/DevConf/Team stuff last week
17:24:01 <pbrobinson> nirik: yes, that's the plan
17:25:06 <dgilmore> cool
17:25:07 <dgilmore> #topic Secondary Architectures updates
17:25:07 <dgilmore> #topic Secondary Architectures update - ppc
17:25:15 <dgilmore> pbrobinson: how is ppc?
17:26:40 <Daredel> hi everyone
17:26:46 <dgilmore> hi Daredel
17:27:01 <pbrobinson> dgilmore: on my list to deal with this week too as discussed last week
17:27:09 <dgilmore> pbrobinson: okay
17:27:16 <tyll> Hi Daredel
17:27:16 <dgilmore> #topic Secondary Architectures update - s390
17:27:34 <dgilmore> no sharkcz this week. not sure if he is back at work yet
17:27:43 <pbrobinson> I think tomorrow
17:27:44 <nirik> I do have a bit there...
17:27:50 <stereotype> hello (i'm new around here...)
17:28:14 <dgilmore> hi stereotype
17:28:15 <nirik> we found, racked and got the new s390 hub machine setup. It still needs installed and then I'd like to ansiblize it.
17:28:27 <tyll> Daredel: stereotype: You can find the topics we discussed before you joined here: http://meetbot.fedoraproject.org/fedora-meeting-2/2015-02-16/releng.2015-02-16-17.04.log.txt
17:28:29 <dgilmore> nirik: that would be good
17:28:31 <nirik> we also got all the newer ppc machines moved to a new rack.
17:28:36 <nirik> and power8 stuff racked up.
17:28:43 <stereotype> great thanks
17:28:43 <dgilmore> cool
17:28:47 <pbrobinson> nirik: any reason it can't be installed using standard pxe/ansible methods from the outset?
17:29:08 <nirik> pbrobinson: none from my side. I wanted to make sure sharkcz was ok with it...
17:29:22 <dgilmore> nirik: we should enable fas on them all
17:29:23 <nirik> we do need to still configure the raid
17:29:37 <dgilmore> nirik: perhaps using a sysadmin-seconfary group in fas
17:29:38 <pbrobinson> nirik: just go that route as that's the way I want all secondary hubs to be maintained.
17:29:47 <nirik> sounds good to me. ;)
17:30:03 <dgilmore> thats sysadmin-secondary
17:30:04 <nirik> I was going to set it up as a virthost.
17:30:15 <nirik> then run a db and hub as virtuals on it, like we do elsewhere
17:30:23 <pbrobinson> nirik: it doesn't make sense for other stuff and we can all get up to speed on the anisble for Fedora infra as we go. It gives us things like FAS auth etc
17:30:28 <dgilmore> nirik: sounds like a good plan
17:30:33 <nirik> yep.
17:30:57 <pbrobinson> nirik: please do and let me know if you need anything, as dgilmore said a ysadmin-secondary would be good there too
17:31:33 <nirik> sure. does that already exist?
17:31:40 <dgilmore> not that I know of
17:31:51 <dgilmore> I do not think we need to do one per arch
17:31:53 <nirik> ok, we can sort it out
17:32:55 <dgilmore> sound sgood
17:33:00 <dgilmore> #topic Secondary Architectures update - arm
17:33:10 <dgilmore> pbrobinson: any arm specific things to bring up
17:33:11 <dgilmore> ?
17:33:31 <nirik> We also moved the aarch64 stuff we had in phx2 and got serial and power working on it.
17:33:55 <dgilmore> awesome
17:34:07 <nirik> and we have a nice pile of space in that rack for expanding. ;)
17:34:07 <dgilmore> need to kickstart them with f21
17:34:10 <pbrobinson> no ARM specific stuff at the moment, just awaiting to get access to the boxes nirik moved the other day so they can be reinstalled with F-21 for the next set of release stuff
17:34:17 <dgilmore> and run ansible
17:34:29 * nirik nods.
17:34:31 <pbrobinson> dgilmore: yes, discussing that with nirik as we speak
17:34:58 <nirik> I need to config the power there... but it's working.
17:35:06 <dgilmore> cool
17:35:27 <dgilmore> #topic Open Floor
17:35:28 <nirik> and the serial is so much nicer than the old stuff we have.
17:35:34 <dgilmore> nice
17:35:49 <dgilmore> I have one open floor item
17:36:09 <dgilmore> we have a f22 TC request but waiting o n the goahead from qa
17:36:16 <nirik> I have one or two too.
17:36:29 <dgilmore> there is some changes that need to be made to the compsoe process before we start it
17:36:57 <dgilmore> related. we need to setup a compose process for making updated docker and cloud images
17:37:08 <dgilmore> and we should see if we should make updates for lives
17:37:14 <dgilmore> at least the desktop ones
17:37:22 <dgilmore> that is all from me
17:37:43 <nirik> so two things I guess:
17:38:16 <nirik> 1) I just made a pkgs02 production instance... we are going to sync pkgs data over to it and test it a bit, but we would like to move it in place of pkgs01 before freeze if possible
17:38:26 <nirik> the staging one has been fine for a while now.
17:38:36 <dgilmore> okay
17:38:51 <pingou> and this week we're all available if needed
17:38:59 <nirik> 2) there's a random squid issue I haven't been able to isolate. Sometimes it doesn't want to download right to builders and builds fail.
17:39:09 <pingou> (even 24h availabitility as bochecha is in Beijin)
17:39:19 <nirik> I'll try and gather more info, but if you see it thats likely whats happening, and any ideas on why welcome. ;)
17:39:27 <dgilmore> nirik: and number2 is new with the new kojipkgs box?
17:39:41 <pbrobinson> nirik: is the squid config in the public ansible?
17:39:46 <nirik> yeah.
17:39:48 <nirik> pbrobinson: yep
17:40:03 <nirik> it seems to fail downloading gcc every time for some reason.
17:40:15 <nirik> but only some builds and after some time.
17:40:23 <nirik> at least all I can think is that it's squid somehow
17:40:52 <pbrobinson> nirik: something to do with the object size configuration in squid?
17:41:04 <nirik> http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/kojipkgs/files/squid.conf
17:41:15 <nirik> I made everything gigantic... since we have lots of memory
17:41:50 <nirik> and there's no errors on the squid side and I can download it fine manually.
17:41:58 <nirik> but in the builds it says "too slow, giving up"
17:42:12 <nirik> so it could be some mock issue too I guess.
17:43:02 <dgilmore> nirik: could be mock
17:43:18 <dgilmore> nirik: does wget work fine inside of phx2?
17:44:10 <nirik> yep.
17:44:15 <dgilmore> hrrm
17:44:17 <nirik> http://koji.fedoraproject.org/koji/taskinfo?taskID=8923589 is an example failure
17:44:18 <dgilmore> could be mock
17:44:38 <nirik> http://kojipkgs.fedoraproject.org//repos/f23-build/454711/x86_64/toplink/packages/gcc/5.0.0/0.12.fc23/x86_64/gcc-5.0.0-0.12.fc23.x86_64.rpm: [Errno 14] curl#18 - "transfer closed with 11593597 bytes remaining to read"
17:44:57 <nirik> or something with builders, dunno. anyhow, just wanted to make folks aware of the issue.
17:45:18 <pbrobinson> nirik: is the base OS RHEL-7?
17:45:23 <dgilmore> something that needs looked at
17:45:29 <nirik> pbrobinson: on the squid box? yes.
17:45:38 <pbrobinson> and what was the previous OS prior to the rebuild?
17:45:42 <dgilmore> rhel6
17:46:29 <pbrobinson> OK, I seem to remember something similar to this on a customer site. I'll see if I can dig out some details from my notes but I think there might be some tuning we can possibly do on the IP/tcp stack
17:46:51 <pbrobinson> nirik: on a KVM VM? or phys HW?
17:47:11 <nirik> it's a kvm vm
17:47:16 <nirik> cool. ;)
17:48:20 <pbrobinson> nirik: yes, there's something with el7 and large/bulk transfers like you'd get from a squid box that rings a bell (although it might just be the drugs/fever getting to my brain) I'll try and dig details out later this evening
17:48:38 <nirik> that would be awesome. :) Thanks.
17:50:26 <dgilmore> anything else?
17:50:27 <tyll> stereotype: Daredel: Do you want to introduce yourselves?
17:50:57 <Daredel> yeah :D
17:51:53 <Daredel> I'm from Colombia, I really like programing, I hope I can help with the releng team :)
17:53:02 <dgilmore> Daredel: welcome, what languages do you program in?
17:53:32 <Daredel> also, I have been using open source software for a year, and have experience with RHEL administration an troubleshooting
17:53:55 <dgilmore> in releng we generally use python
17:54:16 <Daredel> bash - Java - Python, also know a couple more but haven't practice much.
17:55:18 <dgilmore> Daredel: cool. I will be writing up some documents on changes we need in our tooling and processes. there is lots of opportunity for writing in python
17:55:56 <tyll> I have also something: Do we now have a F23 signing key? Does signing with koji work again?
17:56:08 <tyll> signing rawhide/branched in koji
17:56:17 <dgilmore> tyll: we do not yet have the f23 keys made
17:56:24 <dgilmore> that is going to be done today
17:56:56 <tyll> ok
17:57:53 <stereotype> hi im from new zealand
17:58:27 <dgilmore> hi stereotype, north or south island?
17:58:37 <stereotype> south, dunedin
17:58:39 <Daredel> dgilmore: nice, I'll be looking forward help with that
17:58:43 <dgilmore> col :)
17:58:43 <nirik> tyll / dgilmore: I have been signing branched... splitting things into chunks of 1000
17:59:29 <stereotype> i work as a sys admin and use bash mostly but very keen to learn python
17:59:45 <nirik> it would be much nicer if that bug was fixed. ;)
17:59:50 <tyll> nirik: does sigulsign support this directly or did you do it manually?
18:00:30 <nirik> tyll: manually. koji list-tagged f22 --latest --quiet >& f22-list; split -l 1000 f22-list; then sigul_sign xaa, xab, etc.
18:00:52 <tyll> nirik: can you maybe try to manually patch the logging not to fail to see if it is just the logging or also some other issue?
18:01:22 <nirik> tyll: I guess, but that would require hotpatching the hub... kinda of disruptive...
18:01:38 <tyll> nirik: I could also take a look but I believe I do not have the root permissions on the system (but I also do not know which system it would be, probably the hub)
18:02:22 <tyll> nirik: the required change is minimal
18:03:25 <tyll> I am not sure how fast we can get a real RPM with the logging issue fixed
18:03:38 <tyll> I can patch it in dist-git, prepare a build etc
18:03:39 <nirik> tyll: sure. can you create me a patch? I can try and apply it.
18:05:39 <tyll> nirik: which host is the hub?
18:05:49 <nirik> koji01
18:06:01 <nirik> (and koji02, but koji01 is primary right now)
18:06:15 <tyll> nirik: I can login there - I will provide the patch
18:07:07 <dgilmore> koji02 would probably be a good place to patch and test it
18:08:30 <dgilmore> stereotype: anything in particular you are interested in helping with?
18:09:56 <stereotype> not sure? i'm not too fussed...
18:10:44 <dgilmore> stereotype: okay. to be happiest and most effective you should contribute in an area that interests you
18:11:02 <stereotype> cool... i like scripting
18:12:33 <dgilmore> cool
18:12:48 <dgilmore> well if no one has anything else i will end the meeting
18:13:08 <dgilmore> stereotype and Daredel be sure to hang out in #fedora-releng
18:13:15 <stereotype> cool
18:13:19 <dgilmore> feel free to ask questions
18:13:34 <Daredel> sure, i'll be idel all the afternoons
18:13:36 <tyll> nirik: here is the patch: http://paste.fedoraproject.org/186168/10402142
18:13:39 <Daredel> idle*
18:13:59 <nirik> tyll: cool. will try and apply and test it in a bit.
18:14:25 <tyll> nirik: maybe it is also enough to just increase a memory limit so the warning is not triggered
18:14:54 <tyll> nirik: I really do not understand from the code when context.method might be defined and when not
18:15:10 <dgilmore> tyll: likely. we are likely using more ram now that the hubs are 64 bit
18:16:14 <tyll> I see
18:17:13 <dgilmore> #endmeeting