16:00:35 <sgallagh> #startmeeting Server SIG Weekly Meeting (2016-01-26)
16:00:35 <zodbot> Meeting started Tue Jan 26 16:00:35 2016 UTC.  The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:35 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:35 <zodbot> The meeting name has been set to 'server_sig_weekly_meeting_(2016-01-26)'
16:00:36 <sgallagh> #meetingname ServerSIG
16:00:36 <zodbot> The meeting name has been set to 'serversig'
16:00:36 <sgallagh> #chair sgallagh mizmo nirik stefw adamw simo danofsatx mhayden jds2001
16:00:36 <sgallagh> #topic roll call
16:00:36 <zodbot> Current chairs: adamw danofsatx jds2001 mhayden mizmo nirik sgallagh simo stefw
16:00:45 <mhayden> .hello mhayden
16:00:46 <zodbot> mhayden: mhayden 'Major Hayden' <major@mhtx.net>
16:00:50 <sgallagh> .hello sgallagh
16:00:50 <nirik> .hello kevin
16:00:51 <Dodji> Hello
16:00:53 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
16:00:57 <zodbot> nirik: kevin 'Kevin Fenzi' <kevin@scrye.com>
16:00:59 <sgallagh> Dodji: Thanks for joining us today.
16:01:14 <Dodji> thank you for inviting me
16:01:19 <simo> .hello simo
16:01:20 <zodbot> simo: simo 'Simo Sorce' <ssorce@redhat.com>
16:01:20 <danofsatx> .hello dmossor
16:01:26 <zodbot> danofsatx: dmossor 'Dan Mossor' <danofsatx@gmail.com>
16:02:18 <Dodji> .hello Dodji
16:02:19 <zodbot> Dodji: Sorry, but you don't exist
16:02:23 <Dodji> :)
16:02:38 <danofsatx> Dodji: it's FAS account
16:02:47 <Dodji> .hello dodji
16:02:48 <zodbot> Dodji: dodji 'Dodji Seketeli' <dodji@seketeli.org>
16:02:51 <Dodji> better
16:03:08 <Dodji> danofsatx: thanks
16:03:13 <danofsatx> yw
16:04:23 <sgallagh> We technically have quorum, but we're missing at least one person that I really want to be part of this conversation...
16:04:26 <sgallagh> adamw: Are you around?
16:06:21 <sgallagh> OK, well, I guess we may as well get started.
16:06:24 <sgallagh> #topic Agenda
16:06:40 <sgallagh> #info Agenda Topic: ABI/API Compatibility
16:06:55 <sgallagh> Other topics for this week?
16:09:07 <sgallagh> OK, guess not
16:09:15 <sgallagh> #topic ABI/API Compatibility.
16:10:15 <sgallagh> Dodji: Would you be willing to give a high-level overview of some of the work you've been doing, and then we can talk about how to take it and put it into practice for the Server Edition?
16:12:04 <sgallagh> Dodji: ?
16:12:21 <Dodji> right
16:13:35 <Dodji> After we started looking into how to come up with an automatic scheme to detect incompatible ABI changes, we realized we needed a tool that would compare two binary *packages* and detect potentially problematic changes.
16:14:12 <Dodji> We came up with abipkgdiff (https://sourceware.org/libabigail/manual/abipkgdiff.html)
16:14:56 <Dodji> that tool takes two versions of a package, say glib-version-one and glib-version-two, along with the debug info packages of the two packages
16:15:13 <Dodji> and emits a report of the ABI changes that are potentially problematic.
16:15:57 <Dodji> In parallel, there has been work done in the taskotron infrastructure, to allow a task to retrieve:
16:16:34 <Dodji> 1/ the latest previous *stable* package, for a given N-V-R
16:16:52 <Dodji> 2/ the debug info package, for a given N-V-R.
16:17:46 <sgallagh> Dodji: Can you show an example of what the output of that check looks like?
16:17:53 <Dodji> yes
16:17:58 <sgallagh> Is it human-readable, or is other tooling needed for that?
16:19:16 <Dodji> http://paste.fedoraproject.org/314933/25139145
16:19:25 <Dodji> it's human-readable
16:19:59 <Dodji> so that's on the Glib package; comparing the rawhide one with the latest stable in F23
16:20:15 <sgallagh> Cool
16:20:37 <Dodji> so
16:21:06 <Dodji> Provided someone stands up to write (and maintain) the taskotron task, I think it's now possible to write the task that will fire on each koji build
16:21:07 <nirik> nice
16:21:23 <sgallagh> Very, very cool
16:21:33 <Dodji> and compare the abi of the build that just completed against the latest stable build that matches
16:21:45 <Dodji> there are two catches, though.
16:21:57 <Dodji> One that I think is handled, and another one that is not yet handled.
16:22:03 <sgallagh> I assume we'd also want to notify the creator of the build that the ABI had changed.
16:22:04 <Dodji> I'll start by the one that is handled.
16:22:20 <sgallagh> Maybe even automatically untag the build so it doesn't hit the Rawhide mirrors in the nightly mash
16:22:31 <Dodji> 1/ We need somehow to provide a way for packagers to cut false positives
16:22:35 <sgallagh> (If it was intentional breakage, the packager could tag it back in)
16:22:46 <sgallagh> Sorry, please continue.
16:22:55 <Dodji> sgallagh: right, I'll come to your valid points in a bit.
16:22:58 <Dodji> so, 1/
16:23:41 <Dodji> For instance, in the case I just showed in the fpaste, that ABI change is a change to an implementation detail
16:24:17 <Dodji> why? because the change is on a type (a struct) that is *defined* somewhere that is not a public header file.
16:24:40 <Dodji> it says:
16:24:43 <Dodji> 1 data member insertion:
16:24:43 <Dodji> 'guint _GSocketPrivate::connected_read', at offset 256 (in bits) at gsocket.c:245:1
16:24:51 <Dodji> see, gsocket.c
16:25:03 <sgallagh> Right, so an opaque struct grew a new member
16:25:07 <Dodji> right
16:26:35 <Dodji> http://paste.fedoraproject.org/314938/53825587
16:26:55 <sgallagh> So we need a way for the package maintainer to acknowledge that certain changes are not public ABI
16:27:05 <Dodji> so, look at the output of the tool, when we tell it to not look at type changes that happen outside of *.h files
16:27:09 <sgallagh> And that the size of the struct is not supposed to be relied upon.
16:27:45 <Dodji> so basically, just like you can provide valgrind with "suppressions", you can provide abipkgdiff with suppression specification files.
16:28:27 <Dodji> and the way it's implemented speficially for fedora is that, if a file ends up with *.abignore is found *in the package* abipkgdiff takes it as a suppression specification
16:28:37 <Dodji> and uses it as such.
16:28:44 <sgallagh> Define "in the package" please?
16:28:52 <Dodji> Source: foo.abignore
16:28:56 <Dodji> (for instance)
16:28:58 <sgallagh> OK, so in dist-git
16:29:01 <Dodji> in the .spec file
16:29:06 <Dodji> sgallagh: right.
16:29:06 <sgallagh> Not in the produced binary RPM
16:29:31 <Dodji> sgallagh: ah, woops.  My bad. sorry.  It has to be in the binary RPM
16:29:40 <sgallagh> Ah, hmm.
16:29:50 * Dodji confuses this with some other internal project he works on
16:29:51 <Dodji> sorry
16:30:05 <nirik> I guess it could be a %doc ?
16:30:08 <Dodji> but if people want something else, I am open.
16:30:11 <Dodji> <nirik> I guess it could be a %doc ?
16:30:12 <Dodji> yes
16:30:37 <Dodji> I wanted to provide *something* first.
16:30:40 <sgallagh> I think we'd probably want to have a %abignore macro instead, really.
16:30:50 <bochecha> maybe rpmbuild could put it in the -debuginfo packages even?
16:31:12 <Dodji> yeah, that would work too.
16:31:17 <sgallagh> Yeah, I'd rather see it in -debuginfo, then it's not adding unnecessary cruft to installed packages.
16:31:29 <simo> Dodji: so N-V-R has .abignore and N-V-R+1 too, which one do you pick up ?
16:31:52 <Dodji> simo: the +1 one.
16:32:15 <simo> I mean what if the suppressions  should be different between X-.X+1 and X-1 -> X+1 is that possible ?
16:32:27 <simo> Dodji: my concern is if it wiill become a maintenance burden
16:32:31 <Dodji> simo: actually we only look inside the second package for *.abignore files
16:33:07 <simo> Dodji: so a packager needs to build a packge, get errors, create a new .abiignore file and then rev up the package
16:33:14 <Dodji> right
16:33:36 <simo> wouldn't it be better to pick .abignore from dist-git so that all you need is to commit it and then just rerun the tool w/o having to touch the RPM ?
16:33:40 <sgallagh> simo: Well, we can also probably arrange it so that we have `fedpkg local --abicheck` or something
16:33:41 <Dodji> but the packager can see the error right away when building the package
16:33:48 <Dodji> abipkgdiff is an offline tool
16:33:55 <Dodji> just like rpmlint
16:34:06 <simo> sure
16:34:20 <simo> but truth is, people will end up going through the churn
16:34:29 <Dodji> yeah
16:34:31 <simo> so I'd rather have the file hosted externally to the RPM
16:34:35 * Dodji is opened to ideas :-)
16:34:43 <sgallagh> simo: Right, but we can avoid the churn in koji at least
16:34:45 <simo> because after all some suppressions will be diff-version specific anyway
16:34:50 <sgallagh> By making it part of `fedpkg local`
16:35:00 <Dodji> simo: right;  I just wanted to provide "a way" at least.
16:35:05 <simo> sgallagh: I don't always run fedpkg local :)
16:35:07 <Dodji> I am not against better ways :-)
16:35:13 <sgallagh> simo: tsk tsk
16:35:27 <Dodji> and then there is the second catch, that is not yet handled.
16:35:37 <simo> Dodji: the thing that makes me lean for externally hosted is that a suppression is by definition not a change of anything ni the code
16:35:40 <sgallagh> Right, so let's assume that this is a solvable problem
16:35:45 <Dodji> I think we need a way to whitelist the packages that we actually want to check the ABI for.
16:35:47 <simo> so we go through a whole package build just to add a file in the rpm
16:35:50 <dgilmore> sgallagh: running rpmlint abicheck etc post build is okay to me
16:35:55 <simo> just to suppress a tool used during build
16:36:07 <simo> sounds like a churn we should spare koji from
16:36:16 <sgallagh> dgilmore: In the case of packages that take a long time to build, avoiding having to rebuild in koji seems sensible though
16:36:23 <Dodji> i.e, restrain the ABI check to core packages only
16:36:25 <sgallagh> (Especially with ARM builds taking so long)
16:36:32 <Dodji> yeah
16:36:39 <dgilmore> sgallagh: ehhh
16:36:52 <simo> Dodji: that again seem to be external data koji should retrieve from "somewhere" to me :)
16:36:54 <sgallagh> Anyway, let's avoid prematurely optimizing this
16:37:10 <Dodji> simo: :)
16:37:22 <sgallagh> A mechanism for telling the tool what to skip exists
16:37:24 <Dodji> simo: yes, and I am not sure where exactly
16:37:32 <Dodji> sgallagh: ah
16:37:32 <simo> like we have comps and all, maybe we have a virtual group of packages that defines the whitelist
16:37:36 <sgallagh> Let's all agree that we can figure out an appropriate way to host it as we get down the path
16:37:41 <Dodji> sgallagh: useable from a taskotron task?
16:38:08 <simo> does taskotron have access to dist-git ?
16:38:15 <sgallagh> Dodji: What I mean is that we don't have to solve this today.
16:38:20 <Dodji> okay
16:38:27 <simo> (because another way is again a file in dist-git that tells taskotron if it should run at all
16:38:32 <simo> 2 birds one stone :)
16:38:41 <Dodji> sorry.  I just listed the issues I see to have the system be implemented :)
16:38:50 <Dodji> so, notifications.
16:38:55 <sgallagh> Dodji: No, that's good. We will absolutely follow up on it.
16:39:07 <Dodji> I think we could use the same notification mechanisms used for the taskotron tasks today
16:39:10 <sgallagh> (I just don't want to lose the remaining 20 minutes on that rat-hole)
16:39:15 <Dodji> I.e, the that runs rpmlint.
16:39:29 <Dodji> the *task* that runs rpmlint, sorry.
16:39:45 <Dodji> it sends me notifications on IRC, even.
16:39:53 <sgallagh> Does that run on Koji builds?
16:39:58 <Dodji> yes.
16:40:01 <sgallagh> I thought rpmlint only ran on Bodhi updates
16:40:10 <tflink> sgallagh: that's depcheck and upgradepath
16:40:12 <Dodji> hmmh, now I am not sure anymore :-)
16:40:19 <sgallagh> Huh... I don't see those rpmlint messages.
16:40:23 <simo> isn't it all flowing into fedmsg ?
16:40:25 <sgallagh> Maybe my fedmsg isn't set up to send them
16:40:27 <tflink> rpmlint is run on every koji build, notifications are handled through fmn
16:40:39 <Dodji> right, thanks tflink
16:40:46 <sgallagh> Good to know
16:41:06 <sgallagh> OK, if it's going through fedmsg, that makes notification (relatively) easy, doesn't it?
16:41:26 <tflink> at least standardize-able, if not easy
16:41:51 <tflink> the config in fmn may end up needing some tweaking, though
16:42:31 <Dodji> so, that's all for the work we did.
16:42:40 <sgallagh> Dodji: So what is the open question on notifications?
16:43:05 <Dodji> sgallagh: ah, no, on notification, I was actually replying to your question at the beginning
16:43:23 <Dodji> <sgallagh> I assume we'd also want to notify the creator of the build that the ABI
16:43:23 <Dodji> had changed.
16:43:46 <Dodji> sgallagh: I didn't have any open question on notification myself.
16:44:01 <Dodji> my open question, though, is who is going to take care of the Task?
16:44:11 <Dodji> the taskotron task, that is.
16:44:26 <sgallagh> *everyone suddenly looks urgently busy*
16:44:43 <Dodji> lol
16:44:45 <sgallagh> What language are taskotron tasks written in?
16:45:02 <tflink> they can be written in just about whatever. for the moment, we support python the most
16:45:30 <tflink> so long as it can return results in a form that we can understand, it could be written in just about anything
16:46:07 <Dodji> for the record, I'll provide support through hand holding and code to anything needed on the libabigail side.
16:46:10 <Dodji> I am committed to this.
16:46:49 <sgallagh> bconoboy: I'm going to try to drag you into this conversation, if only because I know this topic is relevant to your interests as well
16:46:57 <sgallagh> And you might be able to find a resource to help with this :)
16:47:25 <tflink> fwiw, I'm doing a workshop on writing taskotron tasks @ devconf
16:47:50 <sgallagh> #info tflink is running a workshop on writing taskotron tasks at DevConf.cz
16:48:13 * bconoboy wakes up
16:48:15 <sgallagh> #info Help wanted for writing a taskotron task for automatically running ABI checks on important packages
16:48:35 <bconoboy> sgallagh: was there a specific question for me?
16:48:36 <Dodji> I am going to be at Fosdem, but not at devconf, sigh.
16:48:57 <masta> seems like the scope is not limited to server
16:48:58 * Dodji always misses the cool stuff.
16:49:03 <Dodji> masta: indeed.
16:49:05 <masta> but this is neato
16:49:17 <sgallagh> #info Server SIG should work with Dodji to resolve open questions about whitelisting packages to be scanned as well as mechanisms for hosting the suppression file.
16:49:46 <sgallagh> masta: Most definitely not, but Server SIG has been talking about wanting to stabilize its advertised ABI for some time now.
16:49:55 <sgallagh> So this is a direct benefit to our goals.
16:50:12 <masta> ah! yes an advertised abi is nice feature for server
16:50:41 <sgallagh> s/nice/mandatory, IMHO :)
16:50:42 <Dodji> sgallagh: ah.  Also.  For the advertising.  I didn't talk about this, but we do have a format for specifying an ABI, in libabigail.
16:50:52 <sgallagh> Oh, interesting
16:50:56 <Dodji> the tools' name is abidw
16:51:18 <Dodji> you say 'abidw the-binary > the-binary.abi'
16:51:18 <sgallagh> What exactly does that mean?
16:51:25 <sgallagh> "specifying an ABI"
16:51:42 <bconoboy> <randomnotion> I would think a stable ABI is equally applicable to Desktop/Workstation users, possibly more so since there are more moving parts </randomnotion>
16:51:45 <sgallagh> What does that look like?
16:51:47 <Dodji> the-binary.abi is then a file which contains the description of all the functions, variables *and their types*
16:51:58 <Dodji> sgallagh: it's an XML file, sorry.
16:52:02 <sgallagh> bconoboy: I'm not disagreeing, but someone has to drive it.
16:52:12 <sgallagh> Dodji: Well, at least you apologize ;-)
16:52:19 <Dodji> sgallagh: but then, you can do: "abidiff foo.abi foo"
16:52:28 <Dodji> sgallagh: to compare a binary against a given abi
16:52:35 <Dodji> just like you compare the abi of two binaries.
16:52:48 <sgallagh> I'm guessing this is an intermediate representation you use when comparing the binaries?
16:52:49 <masta> yeah I was wondering how the ABI graph would be expressed. xml, json, etc
16:53:02 <Dodji> sgallagh: yes
16:53:17 <Dodji> sgallagh: it's the "prettyfied" dump of the internal representation used by libabigail
16:53:20 <sgallagh> This could still be useful in generating a skeleton for an API specification I suppose
16:53:35 <sgallagh> s/still/also/
16:53:47 <bconoboy> sgallagh: completely agree
16:54:00 <sgallagh> bconoboy: To which part?
16:54:05 <bconoboy> some has to drive it
16:54:40 <Dodji> sgallagh: you asked what it looked like: http://paste.fedoraproject.org/314950/53827262
16:54:43 <bconoboy> I'm catching up on scrollback, but basically I think the abi we ship a release with is one we should be checking updates against, minimally
16:55:12 <Dodji> masta: ^^ an example of the abi graph representation
16:55:16 <sgallagh> Dodji: Well, at least it's more readable than YAML or PERL
16:55:26 <Dodji> sgallagh: :)
16:56:04 <sgallagh> Dodji: Thank you again for talking with us about this today. We're going to want to follow up (probably after DevConf).
16:56:04 <Dodji> it's very very detailed though.
16:56:23 <Dodji> sgallagh: you are welcome.  I am looking forward to it.
16:56:38 <nirik> good stuff Dodji. Thanks.
16:56:41 <sgallagh> Unfortunately, we're at the top of the hour and I have another meeting I need to get to.
16:56:45 <Dodji> nirik: thanks :-)
16:56:56 <Dodji> sgallagh: no pb.  I talk too much, I know.
16:56:59 * Dodji has a meeting too.
16:57:02 <sgallagh> Dodji: I'll see what I can do (or convince bconoboy to do) about finding someone to own that taskotron task
16:57:09 <Dodji> thanks.
16:57:31 <sgallagh> I think the advantages to it are obvious
16:57:45 * nirik nods
16:57:47 <sgallagh> #topic Open Floor
16:58:00 <sgallagh> Two remaining minutes for Open Floor. Anyone have anything here?
16:58:09 * nirik has nothing off hand
16:59:21 <sgallagh> Thanks for coming, everyone.
16:59:59 <sgallagh> #endmeeting