14:01:18 <tflink> #startmeeting qadevel
14:01:18 <zodbot> Meeting started Mon Sep 22 14:01:18 2014 UTC.  The chair is tflink. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:18 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:01:24 <tflink> #meetingname qadevel
14:01:24 <zodbot> The meeting name has been set to 'qadevel'
14:01:31 <tflink> #topic Roll Call
14:01:46 <tflink> who's around for the qadevel meeting?
14:01:51 * roshi is here
14:01:55 * satellit_e listening
14:01:56 * handsome_pirate is actually awake for once
14:02:30 * mkrizek is here
14:02:33 * kparal here
14:02:45 <tflink> #chair kparal mkrizek
14:02:45 <zodbot> Current chairs: kparal mkrizek tflink
14:03:14 <tflink> no josef today or did he head out already?
14:03:30 <kparal> he had to drive his friend to the hospital
14:03:44 <tflink> oof, that sounds like a good reason not to be here
14:04:25 <kparal> something about a broken leg and an ironing board. don't ask me how
14:04:45 * threebean lurks
14:05:09 <tflink> :-/
14:05:11 <handsome_pirate> kparal:  That sounds like one of those stories for beer a few years from now
14:05:40 <tflink> anyhow, let's get this party started
14:05:47 <tflink> #topic status updates
14:06:16 <tflink> I'll pester josef about execdb later but I suspect he's been working more on testing as of late
14:06:31 <tflink> mkrizek: how's the ephemeral buildslave work going?
14:07:18 <mkrizek> tflink: unfortunately not good, I have been distracted with sickness and weird trigger issues (I have fix most of them though)
14:07:48 <tflink> k, the trigger issues are more important at the moment anyways
14:08:06 <mkrizek> yeah, I figured
14:08:30 <tflink> I've not made a whole lot of progress on the remote logging setup, either outside of being pretty sure that rsyslog is going to be the way to go for now
14:09:25 <handsome_pirate> rsyslog isn't too difficult to set up
14:09:41 <tflink> sure, but that's only part of the puzzle
14:10:35 <tflink> kparal: I assume that you've not made much progress on package installation from formulas due to testing work?
14:11:11 <kparal> none, I have to say. I've been testing mostly and reading up on what happened
14:11:29 <kparal> sorry
14:11:50 <tflink> no worries, getting F21 out the door is a good reason :)
14:12:47 <tflink> unless there are status updates I've forgotten, let's move on
14:13:31 <tflink> #topic Moving Taskotron to Production
14:14:06 <tflink> Unless there are showstoppers that I don't know about, the current plan is to get taskotron production up and running before beta freeze
14:14:36 <tflink> https://phab.qadevel.cloud.fedoraproject.org/T323
14:15:01 * tflink is working on the docs for infra, should be done today or tomorrow
14:15:23 <kparal> when should AutoQA get shut down, also before Beta freeze?
14:15:49 <tflink> I'd like to keep it around (maybe in hibernation) for a bit in case something goes horribly wrong
14:15:57 <tflink> I don't expect problems but you never know
14:16:15 <tflink> and rebuilding autoqa once its destroyed would be a non-trivial effort
14:17:09 <tflink> the clients are easy to rebuild - it's the master that would take some time
14:17:23 <kparal> the problem I see at the moment is depcheck output
14:17:30 <kparal> it does not really say why it failed
14:17:42 <kparal> you can go to the debug log, but you need to know how
14:18:01 <kparal> e.g. this failed: http://taskotron-dev.fedoraproject.org/resultsdb/results/164578
14:18:12 <kparal> but you won't know why if you receive this link: http://taskotron-dev.fedoraproject.org/taskmaster//builders/x86_64/builds/31563/steps/runtask/logs/stdio
14:18:28 <tflink> just to be clear - your concern is more about how the results is presented in resultsdb/bodhi more than the output of depcheck itself
14:18:49 <kparal> unless you go here: http://taskotron-dev.fedoraproject.org/taskmaster//builders/x86_64/builds/31563/steps/cat_log/logs/stdio
14:19:12 <kparal> well, we can either fix depcheck output, or resultsdb
14:19:21 <kparal> I think fixing depcheck would be much easier
14:19:41 <tflink> or both
14:19:41 <kparal> the same information it currently prints to TAP should also be printed to stdout
14:19:45 <handsome_pirate> kparal:  Okay, what should I do?
14:19:50 <handsome_pirate> ah
14:20:08 <handsome_pirate> kparal:  do you have a phabricator ticket?
14:20:16 <kparal> handsome_pirate: improve it in such a way that a maintainer who receives FAILED result can easily see what was wrong
14:20:16 <tflink> I thought that we changed that already
14:20:33 <kparal> tflink: that's today's run
14:20:47 <kparal> handsome_pirate: I don't have a ticket, should I create one for you?
14:20:58 <tflink> kparal: I'm not saying that it was changed already, just wondering if something was missed
14:21:06 <handsome_pirate> kparal:  If you don't mind, and assign it to me
14:21:11 <kparal> handsome_pirate: ok
14:21:17 <handsome_pirate> kparal:  Thanks
14:21:37 <kparal> at least in my opinion it's quite important to improve this before going into production
14:21:58 <kparal> people won't know how to see the debug log
14:22:30 <tflink> the way that logging statements and stdout are interleaved like that doesn't help
14:22:45 <tflink> interleaved in non-chronological order
14:23:06 <kparal> the problem is stdout/stderr interleaving, that doesn't keep order
14:23:13 <kparal> and that's quite unfortunate, yeah :/
14:23:44 <kparal> we could redirect all to stdout, then it would work better
14:24:09 <kparal> otherwise it would be a complex buildbot hack, if we wanted to keep it separated, I guess
14:24:11 * tflink wonders if depcheck should use logging instead of stdout, as well
14:24:40 <handsome_pirate> As in, dump stderr output to log rather than mix with stdout?
14:24:53 <kparal> upgradepath uses stdout (print) for some things as well
14:25:02 <tflink> handsome_pirate: that could be one option but I'd rather not do that
14:25:47 <handsome_pirate> For the time being, I'll poke at outputting everything to stdout
14:26:12 <handsome_pirate> That would keep everything together and should reduce chronological confusion
14:27:20 <tflink> we have roughly 3 weeks before beta freeze
14:27:20 <kparal> it seems that depcheck currently _does_ print all to stdout
14:27:36 <tflink> it's the logging statements that are going to stderr
14:27:42 <kparal> so no change needed I guess. if we want to do some fixes, it will be probably somewhere lower
14:28:24 <handsome_pirate> In libtaskotron, then?
14:28:36 <tflink> handsome_pirate: maybe
14:28:44 <handsome_pirate> tflink:  Should I dump the logging statements to stdout?
14:28:58 <handsome_pirate> tflink:  Or leave as-is and look in libtaskotron?
14:29:03 <tflink> handsome_pirate: or just start using logging in depcheck - that way the statements would have timestamps
14:29:43 <handsome_pirate> tflink:  Roger, I'll do that
14:30:44 <threebean> a question - will there be a new release of resultsdb before things go to production?
14:30:44 <tflink> are there any other issues that could block moving to production?
14:31:01 <tflink> threebean: there can be, are we missing your last patch?
14:31:08 <threebean> it is low priority/non-essential, but yeah.
14:31:20 * handsome_pirate has to take off, will work on depcheck logging this afternoon
14:31:49 <tflink> I'd like to have new builds of just about everything to verify the update procedure
14:32:18 <tflink> ie - write the SOP, test it in stg to make sure nothing's missing before moving to prod
14:32:31 * threebean nods
14:32:33 <threebean> sounds good :)
14:33:25 <tflink> which reminds me that I need to file a ticket to use the backlog feature of fedmsg-hub
14:33:55 <kparal> upgradepath uses both, so if you want to have an inspiration, check it out
14:33:55 <kparal> I think the primary objective is now to make the output helpful, and interleaving it right is a secondary goal
14:34:15 <tflink> agreed
14:34:29 <kparal> I just had a 1-minute lag
14:34:35 <tflink> I thought that we were supposed to be printing tap to stdout post-run anyways
14:34:37 <kparal> 3-minute probably
14:36:02 <kparal> handsome_pirate: if you haven't received the message, you can check out upgradepath for inspiration if you want to use logging inside depcheck
14:38:15 <tflink> any other potentially production-blocking issues?
14:38:51 <kparal> can't think of anything else at the moment
14:39:54 <tflink> moving on, then
14:39:58 <tflink> #topic Iteration 6
14:40:16 <tflink> with a long delay, the start of iteration 6 :)
14:40:59 <tflink> T325 and T331 are probably the highest priority, unassigned tickets right now
14:41:45 * mkrizek takes T331
14:41:57 <tflink> https://phab.qadevel.cloud.fedoraproject.org/T325
14:42:15 <tflink> anyone willing/able to come up with monitoring requirements and methods for taskotron?
14:42:41 <tflink> mkrizek: it's not quite so easy as changing the config - it'll require rebuilding some clients as well
14:43:09 <tflink> which is a fun process that I need to learn at some point
14:43:18 <mkrizek> er :/
14:43:56 <tflink> but it'll also be a good time to start moving vms off the old, about to be retired qa0* machines
14:44:26 <tflink> hopefully we'll be able to get more of the new machines up and running before the warranty on the old boxes expires
14:46:39 <tflink> alternatively, if someone wants to coordinate the backup of taskotron, I can work on the monitoring
14:47:47 * kparal doesn't feel like working on admin stuff, but if there's no one else, he can try
14:48:54 <tflink> kparal: even another set of eyes looking at what we need to monitor would be helpful
14:50:54 <kparal> ok
14:51:13 <tflink> I'll update the ticket with the things that I've thought of already
14:53:30 <tflink> kparal: otherwise, are you planning to work on taskotron output or pkg installation from formulas
14:53:30 <tflink> ?
14:54:10 <kparal> by taskotron output you mean interleaving stdout/stderr or something else?
14:54:22 <tflink> your concern about output being useful
14:54:27 <tflink> unless that was limited to depcheck
14:54:36 <kparal> mainly it was
14:55:10 <kparal> adding a link to buildbot job page would help a lot as well, but that's going to be a bit more difficult I guess
14:55:51 <tflink> aren't we using the resultsdb link in bodhi comments?
14:56:57 <kparal> and from resultsdb you can go only to the main log, it seems
14:57:08 <tflink> which other log would you link to?
14:57:21 * tflink notes that we're almost out of time
14:57:30 <kparal> the debug log is sometimes helpful
14:58:03 <kparal> I don't say it has to be too visible, but somehow making it possible to navigate to it would be nice
14:58:14 <tflink> debug log?
14:58:19 <kparal> /var/log/taskotron.log
14:58:22 <kparal> the last step in a job
14:58:35 <kparal> with DEBUG output enabled
14:58:52 <tflink> that's not going to happen in 3 weeks
14:58:56 <kparal> yes
14:58:59 <tflink> nvm
14:59:02 <tflink> it's monday
14:59:28 <tflink> at least, we could update the "reading results" docs to mention the taskotron logs
14:59:43 <kparal> I can do that
15:00:02 <tflink> it may be worth discussing whether it's worth the effort to add a second log link to resultsdb
15:00:31 <tflink> but that should wait for josef
15:01:03 <tflink> we're out of time, though
15:01:13 <tflink> continue the discussion as needed in #fedora-qa?
15:01:34 * kparal nods
15:01:38 <mkrizek> sounds good
15:01:39 <tflink> #topic open floor
15:01:49 <tflink> anything else that should be mentioned?
15:02:11 <tflink> I have one request - please assign priorities when you file tickets. the default "needs triage" is not very helpful
15:02:23 <tflink> unless "needs triage" is appropriate
15:02:48 <kparal> usually I try to do that
15:03:05 <tflink> the last several taskotron tickets that were filed did not have priorities set
15:03:30 <tflink> anyhow, if there's nothing else, I'll close out the meeting and we can move over to the regular qa meeting
15:03:48 * tflink will send out minutes shortly
15:03:54 <tflink> thanks for coming, everyone!
15:03:59 <tflink> #endmeeting