15:03:22 <tflink> #startmeeting fedora-qadevel
15:03:22 <zodbot> Meeting started Tue Feb 23 15:03:22 2016 UTC.  The chair is tflink. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:22 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:03:22 <zodbot> The meeting name has been set to 'fedora-qadevel'
15:03:22 <tflink> #topic Roll Call
15:03:35 <tflink> #chair kparal mkrizek lbrabec
15:03:35 <zodbot> Current chairs: kparal lbrabec mkrizek tflink
15:03:46 * mkrizek 
15:03:52 <kparal> lbrabec is a lie
15:03:57 * kparal is here
15:04:02 * jskladan is present
15:04:39 <tflink> no jan, either?
15:04:56 <jskladan> tflink: nope
15:04:57 <kparal> nope
15:04:58 <jskladan> just left
15:05:04 <jskladan> kparal: ninja'd
15:05:33 <jskladan> tflink: he was mumbling something running out of tissues and toilet paper...
15:06:00 <tflink> that does sound like a problem ...
15:06:22 <tflink> anyhow, I think we have all the folks we're going to get, so let's get started
15:06:32 <tflink> #topic Announcements and Information
15:06:41 <tflink> #info cli options are now forwarded to minions (D743) - kparal
15:06:42 <tflink> #info stdout+stderr are now printed chronologically (D749) - kparal
15:06:42 <tflink> #info Image building/grabbing is working fine on qa11 - jskladan
15:06:42 <tflink> #info changes in resultsdb required for namespacing are in git - jskladan
15:06:42 <tflink> #info proof-of-concept for openqa's needle organization in subdirs done - jskladan, jsedlak
15:06:43 <tflink> #info dist git style checks - discussions about tasks pulling - mkrizek, tflink, kparal
15:06:45 <tflink> #info various infra fixes and maintanence - mkrizek
15:06:54 <tflink> any question/comments on that or additional items?
15:07:36 <mkrizek> not here
15:07:41 <jskladan> nope
15:07:42 <kparal> no
15:08:11 <tflink> ok, moving on
15:08:17 <tflink> #topic Image Building
15:08:33 <tflink> jskladan: how has the building been going in dev?
15:08:45 <tflink> building/distribution
15:09:05 <jskladan> tflink: as far as I can tell, it's working fine
15:09:25 <jskladan> I'd be tempted to push it up-the-chain to either stg or prod
15:09:29 <jskladan> whatever you feel is OK
15:09:48 <tflink> that's what I was wondering about
15:10:03 <tflink> let's get it into stg and if all goes well, prod by the end of the week
15:10:06 <jskladan> I'd go STG + make DEV download it from there
15:10:16 <jskladan> and push it to PROD if stuff does not explode
15:10:18 <jskladan> :)
15:10:31 <mkrizek> are we going to have more building system deployments? or just one for dev/stg/prod?
15:10:43 <jskladan> mkrizek: just one I hope
15:10:50 <tflink> where do we want the image building to happen?
15:11:00 <tflink> if it's in prod, let's just wait to move it until then
15:11:16 <mkrizek> but where do we test fixes?
15:11:54 <tflink> do we need to check if the produced images are the same enough?
15:12:22 <jskladan> tflink: mkrizek: there is some merit to that, especially if we change the source kickstart
15:12:33 <kparal> we'll probably want to run the latest images on dev/stg first and later on production
15:12:39 <jskladan> but it could IMHO be just tested by over-riding the source image in configuration
15:12:44 <jskladan> for the testing's sake
15:13:57 <jskladan> and then moving the KS to prod
15:13:57 <jskladan> since we don't really touch imagefactory's code
15:13:57 <jskladan> I don't feel it's necesary to have multiple image-building services
15:13:57 <jskladan> we could also have some other flavor
15:13:57 <jskladan> like "taskotron_cloud_dev"
15:13:59 <jskladan> that we could play around with
15:14:00 <tflink> i think that there will be less use for a dev instance of the image builder long term but for the medium term, we still need to figure out building more than f23 images etc.
15:14:04 <jskladan> not disrupting the prod system
15:14:29 <kparal> it's not just about the kickstart, any system updates can break our images
15:14:51 <tflink> but that still doesn't require multiple builders in the long run, I don't think
15:15:00 <kparal> I don't say we need multiple build services, but I think there should be delay between using the latest images on dev and prod
15:15:01 <jskladan> tflink: I agree
15:15:10 <tflink> like jskladan was saying, i think that having a "dev" flavor would accomplish pretty much the same thing
15:15:23 <jskladan> kparal: as it makes sense, it IMHO is quite impractical, we are rebuilding once a day
15:15:24 <tflink> and if stuff blows up, we can just use older images until it's fixed
15:15:37 <jskladan> and keeping 5 days of history
15:16:00 <tflink> do we have any monitoring on the image building, especially if something fails?
15:16:29 <tflink> when is the switchover supposed to happen? isn't it like 03:00 UTC?
15:16:37 <jskladan> tflink: nope, especially since there is no way to get the build logs (i.e. "why the hell it failed") from the failed build)
15:16:50 <jskladan> tflink: switchover to what?
15:17:03 <tflink> when do the new images get downloaded to the client hosts?
15:17:18 <tflink> jskladan: not even notificaitons of "this failed, you might want to take a look at it"?
15:17:41 <jskladan> at the moment, the cron-job is set to check for new builds once per hour
15:17:46 <jskladan> could be changed, ofc
15:18:27 <tflink> I'm just wondering if it would be wise to have that set to when one of us will be around
15:18:37 <jskladan> tflink: nope. But listing the failed builds is easy. If there's a sane way to send email to someone, saying "this build failed", it can be done quite easily
15:19:05 <tflink> jskladan: the local postfix relay should be set up on those machines
15:19:24 <tflink> otherwise, routing through bastion works
15:19:27 <jskladan> but I did not bother with it, as the imagefactory is less than usefull for determining "what" went wrong
15:19:31 <tflink> and is less likely to get stuck in queue
15:19:59 <tflink> yeah but I think it'd be good to have some notice if/when things don't build successfully
15:20:06 <jskladan> so, technically, checking for failed builds is easy (and implemented), I'm just removing them just after the build fails now
15:20:36 <tflink> how long do you think it'd take to implement email-on-failure?
15:21:30 <jskladan> as I said, I don't really know what to set up, and how, to be able to just email somebody@redhat.com, but if we figure this out, then the code for emailing itself is doable in an hour max
15:21:48 <tflink> k, we can sync up on that outside the meeting
15:21:52 <jskladan> k
15:22:07 <tflink> #action tflink to file ticket about base_image notifications, add required email info
15:22:16 <tflink> anything else on this topic?
15:22:41 * tflink assumes not, moves on to
15:22:44 <tflink> #topic Tasking and Roadmap
15:23:10 <tflink> I've been getting more tickets into phabricator
15:23:26 <tflink> there are more to put in and I'm waiting on results from some tickets before getting much farther, though
15:23:28 <tflink> https://phab.qadevel.cloud.fedoraproject.org/tag/libtaskotron/
15:23:42 <tflink> the workboard for libtaskotron, at least is more organized
15:23:58 <kparal> what's groomed?
15:23:59 <tflink> the "groomed" and "on deck" columns are sorted in order of priority
15:24:11 <tflink> kparal: enough information to actually work on
15:24:12 <kparal> the same question for 'on deck'
15:24:37 <tflink> "on deck" is the batch of tasks that should be chosen from next - the next priorities
15:24:46 <kparal> ok
15:25:38 <tflink> does this process make sense? are folks ok with giving it a try?
15:25:49 <kparal> handy workboard, I'll try to use it more
15:25:58 <mkrizek> makes sense to me
15:26:07 <tflink> I'm still trying to figure out how to handle multiple project tickets, though
15:26:13 <tflink> resultsdb isn't organized yet
15:26:21 <tflink> and those tickets don't show up on the libtaskotron board
15:26:47 * tflink is planning to mess around with stg to find a decent way before making a mess and spamming everyone to death in prod :)
15:27:46 <tflink> does everyone have enough to keep them busy for the rest of the week?
15:27:58 <jskladan> yup
15:28:13 * kparal set his notification for workboard adjustments to phab notification only, no email
15:28:25 <kparal> yes
15:28:27 <tflink> didn't know you could do that
15:28:30 <mkrizek> yes
15:28:41 <tflink> cool
15:28:46 <tflink> that brings us to
15:28:47 <kparal> https://phab.qadevel.cloud.fedoraproject.org/settings/panel/emailpreferences/
15:28:51 <tflink> #topic Open Floor
15:29:15 * tflink will be spending some time with those preferences later
15:29:22 <tflink> any other topics that folks want to bring up?
15:29:23 <kparal> we should probably mention today's hot topic of openqa integration
15:29:31 <tflink> ah, good point
15:29:32 <kparal> not sure if you've read all of that, tflink
15:29:41 <tflink> all of what?
15:29:48 <kparal> the discussions we had
15:29:55 <tflink> where were they?
15:30:10 <kparal> on irc mostly I think. tldr coming up
15:30:14 <kparal> pungi4 is supposed to be deployed tomorrow
15:30:24 <tflink> yeah, that's what adam said
15:30:33 <kparal> openqa no longer pulls, needs to respond to fedmsgs
15:30:52 <kparal> they wanted to use taskotron. but we probably can't finish the patches so soon
15:31:17 <kparal> and also there's one small issue - our minions would get blocked while openqa is crunching
15:31:19 <kparal> https://phab.qadevel.cloud.fedoraproject.org/T726
15:31:20 <tflink> I forget what all they needed, to be honest
15:31:27 <tflink> eh, I'm not so worried about that
15:31:31 <tflink> we have 30 minions in prod
15:31:44 <kparal> yes, for the moment, not such a problem
15:31:52 <kparal> that ticket is for future consideration
15:32:00 <tflink> point taken - not so worried about that for the moment
15:32:20 <kparal> anyway, the end result is that they'll write a small fedmsg listener to fire off their openqa trigger
15:32:29 <kparal> that should be really simple
15:32:36 <tflink> what all did they need from us?
15:32:54 <kparal> they originally wanted to use taskotron-trigger
15:32:56 <tflink> would have been nice to have a little more specific warning
15:33:00 <kparal> and report to resultsdb
15:33:06 <tflink> yeah, I want to have that happen as well
15:33:29 <kparal> but the only immediate concern is the triggering, and for that they can write a few lines of code as their custom fedmsg listener
15:33:32 <tflink> "oh, btw, we're pushing a huge change to production tomorrow, have fun with that"
15:33:41 <kparal> so in the end, we don't need to integrate it into taskotron _today_
15:34:17 <tflink> oh well, we're qa. it's normal to not be told about upcoming changes :)
15:34:23 <kparal> about reporting to resultsdb, that can wait. and we'll decide whether we want to move them to taskotron, or just allow them to report to resultsdb from their openqa trigger machine
15:34:50 <kparal> so that's the tldr version, just to get you into the picture
15:35:03 <tflink> k, thanks
15:35:15 * tflink will read the IRC stuff after the meeting
15:35:26 <kparal> and also anyone else reading this :)
15:36:01 <kparal> that's it from me, no further topic
15:36:03 <tflink> is there anything else on this topic?
15:36:39 <mkrizek> no
15:36:48 <tflink> any other topics to bring up?
15:37:22 * tflink takes that as a no
15:37:34 <tflink> thanks for coming, everyone
15:37:39 * tflink will send out minutes shortly
15:37:44 <tflink> #endmeeting