[trustable-software] [Trustable-distros] Provenance, git and freedesktop-sdk

[Ben Brewer]

On 26/09/18 16:23, Paul Sherwood wrote:
> On 2018-09-26 13:53, Ben Brewer wrote:
>> Nobody has an opinion on this?
>
> I think there are opinions, but no easy answers.
I agree, but there'd be no point discussing it if the answers were easy :)

>>> Recently we've been moving to have minimal-distro source elements
>>> from freedesktop-sdk using BuildStream junctions.
>>>
>>> As far as was practically possible we attempted to build
>>> minimal-distro from git, while freedesktop-sdk uses tarballs for
>>> most packages.
>>>
>>> We're currently working on the assumption that provenance is
>>> provided by building from git source, and that this is a worthy goal
>>> to work towards, however the practicality of this approach has
>>> become more questionable as we've tried to implement it.
>>>
>>> Here are the issues we've run into:
>>>
>>> * Many projects don't have any repository, or don't have a git
>>> repository:
>>>
>>> * bzip2 has no repository at all
>>> * mpfr (a gcc dependency) only has an SVN repository
>>>
>>> * gmp (a gcc dependency) only has a Mercurial repository
>
> I still advocate the git.baserock.org/trove approach. The history 
> matters IMO, so I'd suggest we use g.b.o for these cases by default.

The trove approach describes two things:

 1. Mirroring all sources so we don't depend on external infrastructure
    to build
 2. Putting all sources in git

I agree that we need to mirror all sources so we can build any previous 
systems.

I don't agree that putting tarballs in git, or having git mirrors for 
svn/cvs/mercurial/etc. makes sense.

We've had issues with mirroring SVN repositories when they are too 
large, and it just adds a layer of complexity that I don't
believe is needed.

Most of our packages come from freedesktop-sdk junctions, would the 
freedesktop-sdk project accept using git.baserock.org as a mirror?
I think we'd also need to include Pedro (CC'd) in this discussion as 
he's currently maintaining the baserock infrastructure.

Also worth mentioning that the trove tool is currently not maintained.


>>> * In many cases, the process of building from git source to
>>> produce a release is undocumented. In these cases a fair amount of
>>> reverse engineering effort is required to gain parity with a tarball
>>> release.
>>> * Many projects require patches applied on-top of git in-order to
>>> build.
>>>
>>> * These patches require maintenance
>>> * These patches may be incorrect especially when changing versions
>>>
>>> * Many projects required different versions of autotools in order
>>> to build the configure file
>>>
>>> * GCC requires a specific version of autoconf (2.64) to build from
>>> git
>
> Could we fix that?
GCC deliberately uses a specific version of autoconf (2.64) because the 
commit their configure script to the repository; they do this partially 
to ensure there aren't large diffs between the configure scripts. The 
only safe fix would be to use an old copy of autoconf as a build dependency.

>>> * ucl (a dependency of SYSLINUX) requires a version of autoconf so
>>> old, I've not been able to build it.
>>> * Many git repositories contain a configure file that can't easily
>>> be generated from the source for these reasons, and in those cases
>>> pose the same issues for provenance that a tar file does.
>
> Not quite... if the configure file is hosted/provided by upstream it 
> means something, doesn't it?
I don't think it does: If you commit a binary to a git repository, does 
that mean the binary can be trusted?

Configure scripts are large,complicated and very hard to audit; I don't 
see any difference between executing a configure script and a random binary.

>>> * Some projects require closed source binaries to build a tar
>>> release from git.
>
> Eek... which ones?
UPX which is a dependency of modern SYSLINUX requires binaries for the 
compression algorithms which pack the boot image. As described below.

>>> * SYSLINUX requires upx to build, and upx requires closed source
>>> binaries to build properly. When building upx from git using ucl, it
>>> prints a warning stating that it's a beta and not to be used in
>>> production. I don't believe it's easy to trust such a build.
>
> Weren't we going to move away from SYSLINUX?
We were going to move away from it, but our research indicated that it 
wasn't really all that easy, likely this would take quite a bit of 
effort to achieve unless we can find someone with specific experience.

We looked into systemd-boot (previously gummiboot) but there wasn't a 
great deal of documentation and it seems that it requires systemd to be 
in the booted image.

We looked into grub which is documented in Linux From Scratch, and while 
it's possible to configure this, it doesn't gain us very much. The 
original Baserock team indicated that SYSLINUX was used because it was 
the simplest and I trust their judgement.

The third option I guess would be uboot for x86, but I've not looked 
into this as I've never seen anyone using it.

I don't currently see any significant gains to be had by updating the 
bootloaders, we'd have to have built everything else from source before 
we're ready to use that as an excuse to drop SYSLINUX.

>>> * The dependency list for the git build is much larger (and
>>> usually undocumented) than the dependency list for the tarball
>
> I suppose if we document, it becomes documented?

It's technically possible to move a mountain with a spoon.
I guess it's partially my fault for not indicating the scope of the 
work, but I think we're talking man years to get this done. It's worth 
bearing in mind that building from git was a goal for Baserock and was 
never achieved for a very basic system.
Being able to build from git sources is the first stage in documenting 
how to build the sources from git.

>>> * When updating "file" to git, I discovered that it has
>>> undocumented dependencies
>>>
>>> * In many cases the increased dependencies would also be required
>>> as part of the bootstrap, which would inflate the bootstrap size by
>>> an impractically large amount.
>>>
>>> The Mercurial and SVN issues can be resolved either by developing
>>> these plugins for BuildStream or using a trove style approach where
>>> we create a git mirror for Mercurial/SVN repositories.
>>>
>>> The remaining issues I believe are a little more tricky to resolve.
>>>
>>> Any thoughts or ideas are welcome.
>
> Can/should we start raising issues on the applicable upstreams, to see 
> if there is any interest in addressing them, or in us helping them to 
> address them?
It's possible we can do this, however most of the projects that are 
difficult to build are GNU or similar projects which have little project 
infrastructure and in many cases have been ignored for years.
We should aim to do this in the future where possible though.

Regards,
Ben

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.trustable.io/pipermail/trustable-software/attachments/20180927/10ff5a78/attachment-0001.html>