Add compression for build release #7

2021-03-15T17:55:24Z

mycognosist commented

2021-03-15 17:55:24 +00:00

(Migrated from github.com)

@mhfowler

Using xz with the default compression level (-6), I was able to get the 2.1 GB image down to 477.2 MB. I think this is a great improvement with minimal change to the build process.

One thing I'd love your opinion on is the way I've handled the failure state for the compression. I highly doubt the compression will ever fail, but if it does then the uncompressed release image is simply copied to the RELEASE_DIR instead of the compressed image.

@mhfowler Using `xz` with the default compression level (`-6`), I was able to get the 2.1 GB image down to 477.2 MB. I think this is a great improvement with minimal change to the build process. One thing I'd love your opinion on is the way I've handled the failure state for the compression. I highly doubt the compression will ever fail, but if it does then the uncompressed release image is simply copied to the `RELEASE_DIR` instead of the compressed image.

mycognosist commented

2021-03-15 17:59:11 +00:00

(Migrated from github.com)

@mhfowler

One extra detail I didn't mention: the -k flag (--keep) in the xz command means that the original .img file is kept, and a second (compressed) file is created (.img.xz).

I've chosen this approach as it feels safer and more conservative. We could remove the -k flag if we wish to save disk space, especially if we end up building a lot of versions of the image at a later stage. Not a major issue but just something I wanted to point out.

@mhfowler One extra detail I didn't mention: the `-k` flag (`--keep`) in the `xz` command means that the original `.img` file is kept, and a second (compressed) file is created (`.img.xz`). I've chosen this approach as it feels safer and more conservative. We could remove the `-k` flag if we wish to save disk space, especially if we end up building a lot of versions of the image at a later stage. Not a major issue but just something I wanted to point out.

mhfowler commented

2021-03-16 08:01:23 +00:00

(Migrated from github.com)

@mycognosist I was thinking of using this image compression script that gunnar was using for the raspberry/debian image https://salsa.debian.org/raspi-team/image-specs/-/blob/master/compress.sh -- It also uses xz at the end

I also just now was planning to finish a python script for building the image, to replace this build.sh file, which in addition to building the image also updates the releases HTML appropriately

I will incorporate the compression into this script too, definitely a big improvement

@mycognosist I was thinking of using this image compression script that gunnar was using for the raspberry/debian image https://salsa.debian.org/raspi-team/image-specs/-/blob/master/compress.sh -- It also uses xz at the end I also just now was planning to finish a python script for building the image, to replace this build.sh file, which in addition to building the image also updates the releases HTML appropriately I will incorporate the compression into this script too, definitely a big improvement

mhfowler commented

2021-03-16 08:32:09 +00:00

(Migrated from github.com)

not sure how much room the extra compression stuff the gunnar script does saves but may be worth trying it out

👍 1

mycognosist commented

2021-03-16 14:06:19 +00:00

(Migrated from github.com)

@mhfowler

I trust Gunnar when it comes to image compression. It'll be interesting to see how much of a difference those extra steps make (I don't think it'll be more than 100 MB smaller, but every MB counts when I think about how many times the image might be downloaded).

@mhfowler I trust Gunnar when it comes to image compression. It'll be interesting to see how much of a difference those extra steps make (I don't think it'll be more than 100 MB smaller, but every MB counts when I think about how many times the image might be downloaded).

mhfowler commented

2021-03-16 14:13:54 +00:00

(Migrated from github.com)

@mycognosist I tried for about 10 minutes to get gunnar's compression script to work, but it kept throwing different errors. So for now I gave up and just used the xz command you added which works great and makes such a big difference

I'll add a github issue to look further into the compression script, but seems not pressing

@mycognosist I tried for about 10 minutes to get gunnar's compression script to work, but it kept throwing different errors. So for now I gave up and just used the xz command you added which works great and makes such a big difference I'll add a github issue to look further into the compression script, but seems not pressing

mycognosist commented

2021-03-16 14:18:54 +00:00

(Migrated from github.com)

@mhfowler

OK great. Yeah simple is best sometimes. I'm happy with the level of compression we've achieved and agree that further tweaking is not a priority.

@mhfowler OK great. Yeah simple is best sometimes. I'm happy with the level of compression we've achieved and agree that further tweaking is not a priority.

Sign in to join this conversation.

No reviewers