14:00:58 <tflink> #startmeeting fedora-qadevel
14:00:58 <zodbot> Meeting started Mon Mar 20 14:00:58 2017 UTC.  The chair is tflink. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:58 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:00:58 <zodbot> The meeting name has been set to 'fedora-qadevel'
14:00:58 <tflink> #topic Roll Call
14:01:04 * kparal is here
14:01:05 * mkrizek 
14:01:13 <tflink> #chair mkrizek kparal lbrabec
14:01:13 <zodbot> Current chairs: kparal lbrabec mkrizek tflink
14:01:17 * roshi is here
14:01:20 * lbrabec is here
14:02:11 * jskladan is here
14:02:16 <tflink> #chair jskladan robyduck
14:02:16 <zodbot> Current chairs: jskladan kparal lbrabec mkrizek robyduck tflink
14:02:18 <tflink> oops
14:02:21 <tflink> #undo
14:02:21 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x4e0f37d0>
14:02:27 <tflink> #topic Roll Call
14:02:30 <tflink> #unchair robyduck
14:02:30 <zodbot> Current chairs: jskladan kparal lbrabec mkrizek tflink
14:02:35 <tflink> #chair roshi
14:02:35 <zodbot> Current chairs: jskladan kparal lbrabec mkrizek roshi tflink
14:03:24 <tflink> ok, lets get this party started
14:03:25 <tflink> #topic Announcements and Information
14:03:56 <tflink> #info Dist-Git Task Storage Deployed to and Enabled in Production- tflink
14:04:03 <tflink> other stuff to add?
14:04:33 <roshi> #info Still investigating issues with nested virt for Cloud/Atomic compose tests - roshi
14:04:50 <kparal> #info base-images are building again
14:05:12 <kparal> #info bot indexing disabled for resultsdb
14:05:40 <kparal> #info taskotron landing page now has a fancy instance switcher -- lbrabec
14:06:56 <tflink> any comments/questions?
14:08:09 * tflink assumes not, moves on
14:08:17 <tflink> #topic Dist-Git Task Storage
14:08:30 <tflink> this is in production now but hasn't really been announced yet
14:08:48 <tflink> roshi: were you able to get the docs you were working on pushed somewhere?
14:09:19 <roshi> I have a draft pushed to the wiki
14:09:26 <kparal> I thought we were pushing it just into dev
14:09:36 <tflink> nope, prod
14:09:37 <roshi> then got sidelined looking into dev to get it running
14:09:41 <kparal> ok
14:10:01 <kparal> will we be deploying the new libtaskotron build or is it not needed for that?
14:10:10 <tflink> not needed AFAIK
14:10:10 <roshi> #info draft Taskotron Overview documentation on wiki: https://fedoraproject.org/wiki/User:Roshi/QA/Taskotron_Overview - roshi
14:10:32 <tflink> we can deploy it since IIRC, it would reduce the number of failed jobs we're seeing
14:10:43 <kparal> it would
14:10:49 <kparal> but we're now in freeze, so at least to dev
14:11:01 <kparal> +stg
14:11:56 <tflink> I'm pretty sure I did a release/build on Friday
14:12:15 <tflink> yeah, I did
14:12:30 <tflink> https://koji.fedoraproject.org/koji/buildinfo?buildID=870047
14:13:20 <tflink> I suspect that we're going to start getting more questions about how to write tasks in the near future
14:13:20 <kparal> if you need help adjusting yumrepoinfo in ansible, I can help
14:13:32 <tflink> kparal: is it just a config change?
14:13:38 * tflink thought there was a code change as well
14:13:43 <kparal> yes, some new fields
14:13:53 <kparal> code change as well
14:14:13 <kparal> ideally should be updated together
14:14:20 <kparal> even though I tried to make it backwards compatible
14:15:14 <tflink> would applying a patch to yumrepoinfo even help?
14:15:42 <kparal> not without the code change
14:15:46 <kparal> but it should break stuff
14:15:49 <kparal> *shouldn't
14:16:04 <tflink> i don't see a point in deploying the config change by itself, then
14:16:16 <tflink> let's wait to see if we end up slipping again this week
14:17:10 <tflink> I'm not sure how much else there is here other than the usual "lets improve docs" :)
14:17:37 <tflink> and giving folks a heads up that questions are likely to be incoming
14:17:54 <tflink> anything else on this?
14:18:33 * tflink assumes not, moves on
14:18:39 <tflink> #topic ResultsDB Performance
14:19:13 <tflink> for some reason, resultsdb started showing issues with load last week. not sure why it didn't happen before or if it just wasn't noticed
14:19:32 * jskladan was not able to reproduce the "hangs" after the load drops
14:19:41 <tflink> after some botched config changes on Friday, things seem to be ok-ish for the moment
14:19:45 <kparal> Josef claims bodhi is not the culprit here, but either way, we were discussing changes how to optimize how it queries resultsdb. I'll post some notes into qa-devel or resultsdb-users
14:20:19 <jskladan> kparal: bodhi is the least likely problem, since we were not getting these hangs in the past months
14:20:44 <kparal> the 5xx errors were present in resultsdb in the past, but it occurred rarely (few times per day at most, I think). now it basically died under load
14:21:07 <tflink> it sounds like the most likely way forward will involve the following things: move production resultsdb to its own db host, get multiple resultsdb httpd hosts and load-balance
14:21:22 <tflink> jskladan: did I get that about right?
14:21:31 <kparal> and optimize the consumers
14:21:42 <kparal> jskladan++ for sending patches to pingou
14:21:51 <jskladan> tflink: yup
14:22:02 <jskladan> that's what my gut feeling says is a good path forward
14:22:20 <tflink> unless things get a lot worse, I don't think that much of that'll happen until after freeze lifts
14:22:39 <tflink> AFAIK, the performance is OK enough for the moment
14:23:05 <kparal> seems useable
14:23:51 <jskladan> might be worth getting some actual sysadmin (no disrespect to Tim implied) to have a look at our apache/db settings
14:24:14 <tflink> no offense taken
14:25:14 <tflink> if we can find someone to look at what we have, that would be great
14:25:41 <tflink> anything else on this?
14:25:54 <kparal> nope
14:26:08 <tflink> other than the lesson of don't just blindly increase threads/processes for mod_wsgi?
14:26:40 <jskladan> :D
14:27:02 * jskladan read some stuff on that during weekend, and 5 processes + 100 threads seems to be frowned upon :D
14:27:45 <tflink> yeah, that's what I gathered from the commit notes and emails around that change :-D
14:28:25 * jskladan has nothing to add, though
14:28:31 <tflink> anything else on this topic?
14:30:31 * tflink assumes not
14:30:37 <tflink> #topic Moar Ansible?
14:30:48 <tflink> I think that this came up earlier this year but I wanted to mention it again
14:32:22 <tflink> I'd like to start moving our jobs to using more ansible
14:32:46 <tflink> the way that the zuul v3 folks are doing things looks sane to me but I haven't looked into things very far
14:32:50 * tflink is trying to find a link
14:33:11 <tflink> https://review.openstack.org/#/c/442180/
14:33:31 <tflink> https://review.openstack.org/#/c/438281/
14:34:00 <tflink> actually, that first one was abandoned - there was a choice between the two
14:34:41 <tflink> I don't recall there being any huge objections
14:34:54 <tflink> but comments/thoughts/objections?
14:36:21 * mkrizek suddenly appreciates phab more when looking at gerrit
14:36:40 <jskladan> mkrizek++
14:36:40 <zodbot> jskladan: Karma for mkrizek changed to 2 (for the f25 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
14:38:05 <tflink> from what I've heard, the big advantages to gerrit have to do with workflow more than web interface
14:38:18 <tflink> AFAIK, there is some pretty big redesign work going on around the web ui
14:38:23 <tflink> but we digress :)
14:38:53 <mkrizek> yeah, no objections from me
14:39:37 <tflink> ok, moving on
14:39:41 <tflink> #topic Tasking
14:39:49 <tflink> is anyone in need of tasks?
14:40:19 * jskladan is good
14:40:25 <kparal> nope
14:40:27 <roshi> if I could get assigned some that were already done, that'd be great
14:40:36 <roshi> bolster the "tasks finished" numbers :p
14:40:39 <tflink> roshi: just get the atomic stuff done
14:40:46 <tflink> then you'll have completed tasks
14:40:48 * mkrizek will probably ask during the week
14:40:51 <tflink> sounds good
14:41:02 <roshi> we've also release validation to do this week
14:41:16 <tflink> very true
14:41:38 * tflink moves on to ...
14:41:43 <tflink> #topic Open Floor
14:41:49 <tflink> any thing else that folks want to bring up?
14:42:01 <tflink> The dashboard that pingou made looks nice: http://ambre.pingoured.fr/brtm/
14:42:26 <kparal> I sent an email about supported arches into qa-devel
14:42:33 <kparal> responses welcome
14:43:11 <kparal> so far I heard in person that we should make --arch a single string (instead of a list) and that we should not bother with i386 minions for the moment
14:43:45 <tflink> it sounds like we should change up depcheck or move over to rpm-deplint as well
14:44:13 <tflink> i forget, was an analysis done on whether there were any important discrepancies between the two tools?
14:45:33 <kparal> I found no discrepancies
14:45:39 <kparal> some weeks back
14:45:56 <kparal> also I'm working on making rpmdeplint arch-specific
14:46:08 <kparal> but that's not that important
14:46:17 * tflink is reminded that there seemed to be some interest from secarch folks to start running automated tests, needs to follow up on that
14:46:37 <tflink> anything else or shall we end 14 minutes early?
14:46:44 <kparal> nothing more here
14:47:55 <tflink> ok, thanks for coming, everyone
14:48:00 * tflink will send out minutes shortly
14:48:07 <tflink> #endmeeting