16:00:35 <sgallagh> #startmeeting Server SIG Weekly Meeting (2016-01-26) 16:00:35 <zodbot> Meeting started Tue Jan 26 16:00:35 2016 UTC. The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:35 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:35 <zodbot> The meeting name has been set to 'server_sig_weekly_meeting_(2016-01-26)' 16:00:36 <sgallagh> #meetingname ServerSIG 16:00:36 <zodbot> The meeting name has been set to 'serversig' 16:00:36 <sgallagh> #chair sgallagh mizmo nirik stefw adamw simo danofsatx mhayden jds2001 16:00:36 <sgallagh> #topic roll call 16:00:36 <zodbot> Current chairs: adamw danofsatx jds2001 mhayden mizmo nirik sgallagh simo stefw 16:00:45 <mhayden> .hello mhayden 16:00:46 <zodbot> mhayden: mhayden 'Major Hayden' <major@mhtx.net> 16:00:50 <sgallagh> .hello sgallagh 16:00:50 <nirik> .hello kevin 16:00:51 <Dodji> Hello 16:00:53 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com> 16:00:57 <zodbot> nirik: kevin 'Kevin Fenzi' <kevin@scrye.com> 16:00:59 <sgallagh> Dodji: Thanks for joining us today. 16:01:14 <Dodji> thank you for inviting me 16:01:19 <simo> .hello simo 16:01:20 <zodbot> simo: simo 'Simo Sorce' <ssorce@redhat.com> 16:01:20 <danofsatx> .hello dmossor 16:01:26 <zodbot> danofsatx: dmossor 'Dan Mossor' <danofsatx@gmail.com> 16:02:18 <Dodji> .hello Dodji 16:02:19 <zodbot> Dodji: Sorry, but you don't exist 16:02:23 <Dodji> :) 16:02:38 <danofsatx> Dodji: it's FAS account 16:02:47 <Dodji> .hello dodji 16:02:48 <zodbot> Dodji: dodji 'Dodji Seketeli' <dodji@seketeli.org> 16:02:51 <Dodji> better 16:03:08 <Dodji> danofsatx: thanks 16:03:13 <danofsatx> yw 16:04:23 <sgallagh> We technically have quorum, but we're missing at least one person that I really want to be part of this conversation... 16:04:26 <sgallagh> adamw: Are you around? 16:06:21 <sgallagh> OK, well, I guess we may as well get started. 16:06:24 <sgallagh> #topic Agenda 16:06:40 <sgallagh> #info Agenda Topic: ABI/API Compatibility 16:06:55 <sgallagh> Other topics for this week? 16:09:07 <sgallagh> OK, guess not 16:09:15 <sgallagh> #topic ABI/API Compatibility. 16:10:15 <sgallagh> Dodji: Would you be willing to give a high-level overview of some of the work you've been doing, and then we can talk about how to take it and put it into practice for the Server Edition? 16:12:04 <sgallagh> Dodji: ? 16:12:21 <Dodji> right 16:13:35 <Dodji> After we started looking into how to come up with an automatic scheme to detect incompatible ABI changes, we realized we needed a tool that would compare two binary *packages* and detect potentially problematic changes. 16:14:12 <Dodji> We came up with abipkgdiff (https://sourceware.org/libabigail/manual/abipkgdiff.html) 16:14:56 <Dodji> that tool takes two versions of a package, say glib-version-one and glib-version-two, along with the debug info packages of the two packages 16:15:13 <Dodji> and emits a report of the ABI changes that are potentially problematic. 16:15:57 <Dodji> In parallel, there has been work done in the taskotron infrastructure, to allow a task to retrieve: 16:16:34 <Dodji> 1/ the latest previous *stable* package, for a given N-V-R 16:16:52 <Dodji> 2/ the debug info package, for a given N-V-R. 16:17:46 <sgallagh> Dodji: Can you show an example of what the output of that check looks like? 16:17:53 <Dodji> yes 16:17:58 <sgallagh> Is it human-readable, or is other tooling needed for that? 16:19:16 <Dodji> http://paste.fedoraproject.org/314933/25139145 16:19:25 <Dodji> it's human-readable 16:19:59 <Dodji> so that's on the Glib package; comparing the rawhide one with the latest stable in F23 16:20:15 <sgallagh> Cool 16:20:37 <Dodji> so 16:21:06 <Dodji> Provided someone stands up to write (and maintain) the taskotron task, I think it's now possible to write the task that will fire on each koji build 16:21:07 <nirik> nice 16:21:23 <sgallagh> Very, very cool 16:21:33 <Dodji> and compare the abi of the build that just completed against the latest stable build that matches 16:21:45 <Dodji> there are two catches, though. 16:21:57 <Dodji> One that I think is handled, and another one that is not yet handled. 16:22:03 <sgallagh> I assume we'd also want to notify the creator of the build that the ABI had changed. 16:22:04 <Dodji> I'll start by the one that is handled. 16:22:20 <sgallagh> Maybe even automatically untag the build so it doesn't hit the Rawhide mirrors in the nightly mash 16:22:31 <Dodji> 1/ We need somehow to provide a way for packagers to cut false positives 16:22:35 <sgallagh> (If it was intentional breakage, the packager could tag it back in) 16:22:46 <sgallagh> Sorry, please continue. 16:22:55 <Dodji> sgallagh: right, I'll come to your valid points in a bit. 16:22:58 <Dodji> so, 1/ 16:23:41 <Dodji> For instance, in the case I just showed in the fpaste, that ABI change is a change to an implementation detail 16:24:17 <Dodji> why? because the change is on a type (a struct) that is *defined* somewhere that is not a public header file. 16:24:40 <Dodji> it says: 16:24:43 <Dodji> 1 data member insertion: 16:24:43 <Dodji> 'guint _GSocketPrivate::connected_read', at offset 256 (in bits) at gsocket.c:245:1 16:24:51 <Dodji> see, gsocket.c 16:25:03 <sgallagh> Right, so an opaque struct grew a new member 16:25:07 <Dodji> right 16:26:35 <Dodji> http://paste.fedoraproject.org/314938/53825587 16:26:55 <sgallagh> So we need a way for the package maintainer to acknowledge that certain changes are not public ABI 16:27:05 <Dodji> so, look at the output of the tool, when we tell it to not look at type changes that happen outside of *.h files 16:27:09 <sgallagh> And that the size of the struct is not supposed to be relied upon. 16:27:45 <Dodji> so basically, just like you can provide valgrind with "suppressions", you can provide abipkgdiff with suppression specification files. 16:28:27 <Dodji> and the way it's implemented speficially for fedora is that, if a file ends up with *.abignore is found *in the package* abipkgdiff takes it as a suppression specification 16:28:37 <Dodji> and uses it as such. 16:28:44 <sgallagh> Define "in the package" please? 16:28:52 <Dodji> Source: foo.abignore 16:28:56 <Dodji> (for instance) 16:28:58 <sgallagh> OK, so in dist-git 16:29:01 <Dodji> in the .spec file 16:29:06 <Dodji> sgallagh: right. 16:29:06 <sgallagh> Not in the produced binary RPM 16:29:31 <Dodji> sgallagh: ah, woops. My bad. sorry. It has to be in the binary RPM 16:29:40 <sgallagh> Ah, hmm. 16:29:50 * Dodji confuses this with some other internal project he works on 16:29:51 <Dodji> sorry 16:30:05 <nirik> I guess it could be a %doc ? 16:30:08 <Dodji> but if people want something else, I am open. 16:30:11 <Dodji> <nirik> I guess it could be a %doc ? 16:30:12 <Dodji> yes 16:30:37 <Dodji> I wanted to provide *something* first. 16:30:40 <sgallagh> I think we'd probably want to have a %abignore macro instead, really. 16:30:50 <bochecha> maybe rpmbuild could put it in the -debuginfo packages even? 16:31:12 <Dodji> yeah, that would work too. 16:31:17 <sgallagh> Yeah, I'd rather see it in -debuginfo, then it's not adding unnecessary cruft to installed packages. 16:31:29 <simo> Dodji: so N-V-R has .abignore and N-V-R+1 too, which one do you pick up ? 16:31:52 <Dodji> simo: the +1 one. 16:32:15 <simo> I mean what if the suppressions should be different between X-.X+1 and X-1 -> X+1 is that possible ? 16:32:27 <simo> Dodji: my concern is if it wiill become a maintenance burden 16:32:31 <Dodji> simo: actually we only look inside the second package for *.abignore files 16:33:07 <simo> Dodji: so a packager needs to build a packge, get errors, create a new .abiignore file and then rev up the package 16:33:14 <Dodji> right 16:33:36 <simo> wouldn't it be better to pick .abignore from dist-git so that all you need is to commit it and then just rerun the tool w/o having to touch the RPM ? 16:33:40 <sgallagh> simo: Well, we can also probably arrange it so that we have `fedpkg local --abicheck` or something 16:33:41 <Dodji> but the packager can see the error right away when building the package 16:33:48 <Dodji> abipkgdiff is an offline tool 16:33:55 <Dodji> just like rpmlint 16:34:06 <simo> sure 16:34:20 <simo> but truth is, people will end up going through the churn 16:34:29 <Dodji> yeah 16:34:31 <simo> so I'd rather have the file hosted externally to the RPM 16:34:35 * Dodji is opened to ideas :-) 16:34:43 <sgallagh> simo: Right, but we can avoid the churn in koji at least 16:34:45 <simo> because after all some suppressions will be diff-version specific anyway 16:34:50 <sgallagh> By making it part of `fedpkg local` 16:35:00 <Dodji> simo: right; I just wanted to provide "a way" at least. 16:35:05 <simo> sgallagh: I don't always run fedpkg local :) 16:35:07 <Dodji> I am not against better ways :-) 16:35:13 <sgallagh> simo: tsk tsk 16:35:27 <Dodji> and then there is the second catch, that is not yet handled. 16:35:37 <simo> Dodji: the thing that makes me lean for externally hosted is that a suppression is by definition not a change of anything ni the code 16:35:40 <sgallagh> Right, so let's assume that this is a solvable problem 16:35:45 <Dodji> I think we need a way to whitelist the packages that we actually want to check the ABI for. 16:35:47 <simo> so we go through a whole package build just to add a file in the rpm 16:35:50 <dgilmore> sgallagh: running rpmlint abicheck etc post build is okay to me 16:35:55 <simo> just to suppress a tool used during build 16:36:07 <simo> sounds like a churn we should spare koji from 16:36:16 <sgallagh> dgilmore: In the case of packages that take a long time to build, avoiding having to rebuild in koji seems sensible though 16:36:23 <Dodji> i.e, restrain the ABI check to core packages only 16:36:25 <sgallagh> (Especially with ARM builds taking so long) 16:36:32 <Dodji> yeah 16:36:39 <dgilmore> sgallagh: ehhh 16:36:52 <simo> Dodji: that again seem to be external data koji should retrieve from "somewhere" to me :) 16:36:54 <sgallagh> Anyway, let's avoid prematurely optimizing this 16:37:10 <Dodji> simo: :) 16:37:22 <sgallagh> A mechanism for telling the tool what to skip exists 16:37:24 <Dodji> simo: yes, and I am not sure where exactly 16:37:32 <Dodji> sgallagh: ah 16:37:32 <simo> like we have comps and all, maybe we have a virtual group of packages that defines the whitelist 16:37:36 <sgallagh> Let's all agree that we can figure out an appropriate way to host it as we get down the path 16:37:41 <Dodji> sgallagh: useable from a taskotron task? 16:38:08 <simo> does taskotron have access to dist-git ? 16:38:15 <sgallagh> Dodji: What I mean is that we don't have to solve this today. 16:38:20 <Dodji> okay 16:38:27 <simo> (because another way is again a file in dist-git that tells taskotron if it should run at all 16:38:32 <simo> 2 birds one stone :) 16:38:41 <Dodji> sorry. I just listed the issues I see to have the system be implemented :) 16:38:50 <Dodji> so, notifications. 16:38:55 <sgallagh> Dodji: No, that's good. We will absolutely follow up on it. 16:39:07 <Dodji> I think we could use the same notification mechanisms used for the taskotron tasks today 16:39:10 <sgallagh> (I just don't want to lose the remaining 20 minutes on that rat-hole) 16:39:15 <Dodji> I.e, the that runs rpmlint. 16:39:29 <Dodji> the *task* that runs rpmlint, sorry. 16:39:45 <Dodji> it sends me notifications on IRC, even. 16:39:53 <sgallagh> Does that run on Koji builds? 16:39:58 <Dodji> yes. 16:40:01 <sgallagh> I thought rpmlint only ran on Bodhi updates 16:40:10 <tflink> sgallagh: that's depcheck and upgradepath 16:40:12 <Dodji> hmmh, now I am not sure anymore :-) 16:40:19 <sgallagh> Huh... I don't see those rpmlint messages. 16:40:23 <simo> isn't it all flowing into fedmsg ? 16:40:25 <sgallagh> Maybe my fedmsg isn't set up to send them 16:40:27 <tflink> rpmlint is run on every koji build, notifications are handled through fmn 16:40:39 <Dodji> right, thanks tflink 16:40:46 <sgallagh> Good to know 16:41:06 <sgallagh> OK, if it's going through fedmsg, that makes notification (relatively) easy, doesn't it? 16:41:26 <tflink> at least standardize-able, if not easy 16:41:51 <tflink> the config in fmn may end up needing some tweaking, though 16:42:31 <Dodji> so, that's all for the work we did. 16:42:40 <sgallagh> Dodji: So what is the open question on notifications? 16:43:05 <Dodji> sgallagh: ah, no, on notification, I was actually replying to your question at the beginning 16:43:23 <Dodji> <sgallagh> I assume we'd also want to notify the creator of the build that the ABI 16:43:23 <Dodji> had changed. 16:43:46 <Dodji> sgallagh: I didn't have any open question on notification myself. 16:44:01 <Dodji> my open question, though, is who is going to take care of the Task? 16:44:11 <Dodji> the taskotron task, that is. 16:44:26 <sgallagh> *everyone suddenly looks urgently busy* 16:44:43 <Dodji> lol 16:44:45 <sgallagh> What language are taskotron tasks written in? 16:45:02 <tflink> they can be written in just about whatever. for the moment, we support python the most 16:45:30 <tflink> so long as it can return results in a form that we can understand, it could be written in just about anything 16:46:07 <Dodji> for the record, I'll provide support through hand holding and code to anything needed on the libabigail side. 16:46:10 <Dodji> I am committed to this. 16:46:49 <sgallagh> bconoboy: I'm going to try to drag you into this conversation, if only because I know this topic is relevant to your interests as well 16:46:57 <sgallagh> And you might be able to find a resource to help with this :) 16:47:25 <tflink> fwiw, I'm doing a workshop on writing taskotron tasks @ devconf 16:47:50 <sgallagh> #info tflink is running a workshop on writing taskotron tasks at DevConf.cz 16:48:13 * bconoboy wakes up 16:48:15 <sgallagh> #info Help wanted for writing a taskotron task for automatically running ABI checks on important packages 16:48:35 <bconoboy> sgallagh: was there a specific question for me? 16:48:36 <Dodji> I am going to be at Fosdem, but not at devconf, sigh. 16:48:57 <masta> seems like the scope is not limited to server 16:48:58 * Dodji always misses the cool stuff. 16:49:03 <Dodji> masta: indeed. 16:49:05 <masta> but this is neato 16:49:17 <sgallagh> #info Server SIG should work with Dodji to resolve open questions about whitelisting packages to be scanned as well as mechanisms for hosting the suppression file. 16:49:46 <sgallagh> masta: Most definitely not, but Server SIG has been talking about wanting to stabilize its advertised ABI for some time now. 16:49:55 <sgallagh> So this is a direct benefit to our goals. 16:50:12 <masta> ah! yes an advertised abi is nice feature for server 16:50:41 <sgallagh> s/nice/mandatory, IMHO :) 16:50:42 <Dodji> sgallagh: ah. Also. For the advertising. I didn't talk about this, but we do have a format for specifying an ABI, in libabigail. 16:50:52 <sgallagh> Oh, interesting 16:50:56 <Dodji> the tools' name is abidw 16:51:18 <Dodji> you say 'abidw the-binary > the-binary.abi' 16:51:18 <sgallagh> What exactly does that mean? 16:51:25 <sgallagh> "specifying an ABI" 16:51:42 <bconoboy> <randomnotion> I would think a stable ABI is equally applicable to Desktop/Workstation users, possibly more so since there are more moving parts </randomnotion> 16:51:45 <sgallagh> What does that look like? 16:51:47 <Dodji> the-binary.abi is then a file which contains the description of all the functions, variables *and their types* 16:51:58 <Dodji> sgallagh: it's an XML file, sorry. 16:52:02 <sgallagh> bconoboy: I'm not disagreeing, but someone has to drive it. 16:52:12 <sgallagh> Dodji: Well, at least you apologize ;-) 16:52:19 <Dodji> sgallagh: but then, you can do: "abidiff foo.abi foo" 16:52:28 <Dodji> sgallagh: to compare a binary against a given abi 16:52:35 <Dodji> just like you compare the abi of two binaries. 16:52:48 <sgallagh> I'm guessing this is an intermediate representation you use when comparing the binaries? 16:52:49 <masta> yeah I was wondering how the ABI graph would be expressed. xml, json, etc 16:53:02 <Dodji> sgallagh: yes 16:53:17 <Dodji> sgallagh: it's the "prettyfied" dump of the internal representation used by libabigail 16:53:20 <sgallagh> This could still be useful in generating a skeleton for an API specification I suppose 16:53:35 <sgallagh> s/still/also/ 16:53:47 <bconoboy> sgallagh: completely agree 16:54:00 <sgallagh> bconoboy: To which part? 16:54:05 <bconoboy> some has to drive it 16:54:40 <Dodji> sgallagh: you asked what it looked like: http://paste.fedoraproject.org/314950/53827262 16:54:43 <bconoboy> I'm catching up on scrollback, but basically I think the abi we ship a release with is one we should be checking updates against, minimally 16:55:12 <Dodji> masta: ^^ an example of the abi graph representation 16:55:16 <sgallagh> Dodji: Well, at least it's more readable than YAML or PERL 16:55:26 <Dodji> sgallagh: :) 16:56:04 <sgallagh> Dodji: Thank you again for talking with us about this today. We're going to want to follow up (probably after DevConf). 16:56:04 <Dodji> it's very very detailed though. 16:56:23 <Dodji> sgallagh: you are welcome. I am looking forward to it. 16:56:38 <nirik> good stuff Dodji. Thanks. 16:56:41 <sgallagh> Unfortunately, we're at the top of the hour and I have another meeting I need to get to. 16:56:45 <Dodji> nirik: thanks :-) 16:56:56 <Dodji> sgallagh: no pb. I talk too much, I know. 16:56:59 * Dodji has a meeting too. 16:57:02 <sgallagh> Dodji: I'll see what I can do (or convince bconoboy to do) about finding someone to own that taskotron task 16:57:09 <Dodji> thanks. 16:57:31 <sgallagh> I think the advantages to it are obvious 16:57:45 * nirik nods 16:57:47 <sgallagh> #topic Open Floor 16:58:00 <sgallagh> Two remaining minutes for Open Floor. Anyone have anything here? 16:58:09 * nirik has nothing off hand 16:59:21 <sgallagh> Thanks for coming, everyone. 16:59:59 <sgallagh> #endmeeting