At Transloadit, we desire to give you the best possible encoding experience, and one of the big pillars in that is
making sure that we use the latest encoding tools.

It’s no secret that we’re heavily relying on open source technology for this, and so we make an effort
to give back to the giants whose shoulders we’re standing on by for instance providing free hardware to the ImageMagick
project.

Even though we have an intimate relationship with the project, it’s proven a very hard problem for us to modernize
our stack.

The ‘fun’ parts of maintaining encoding software

Our customers rely on behavior of the old version (even on its bugs, more on that later), so “just upgrading”, which we naively tried some many years back, breaks their workflow. To ensure we can provide a consistent experience even between operating system upgrades, that means compiling ImageMagick and all of its 31 dependencies. Yes 31, because by adding libraries such as pixman, pango and ufraw we gain support for 131 more formats than what a standard apt-get install imagemagick would provide on Ubuntu. And by including libraries all the way down to zlib, we also shield ourselves from any backwards compatibility breakage that could occur in the face of operating system upgrades, and allows our encoding capabilities to be a constant factor that our customers can build businesses on.

Besides vast support for formats and this guarantee on continuity, the advantage of compiling is that you can create a static binary. This is a single executable file that has every dependency baked into it. This way it becomes easy to run very different versions of the same software alongside one another without any conflicts, giving our customers an upgrade path that is opt-in. This is how we have been proving upgrades to our video encoding software.

The problem now however with upgrading our image stack, is that compiling ImageMagick and the 31 dependencies is that this process would take 2 hours, and that is on a beefy server. If one particular library version conflicted with another, you may only find out after 1.5 hours, you’d have to think of a possible fix and try all over. That’s hardly workable/maintainable, and certainly does not amount to the level of agility that we aspire to have in our development.

The introduction of Docker containers, we thought, would solve some of this struggle in that we could at least run versions alongside easier, and speed up our build times through the use of layers. Also, instead of having single massive binaries to recompile, we could just recompile 1 shared library if needed. Our initial joy was tempered when we learned that there was no way to start containers without dangerous system privileges. This meant we either had to keep containers running and let our non-privileged API user proxy commands to those, or be okay with giving this user the privilege to start containers. Proxying would be fragile however, and by giving privilege an attacker could theoretically mount any directory on the host machine as a --volume into the container, and then read/write/delete whatever they wanted with it. We learned about work done in the runC project to launch containers without root access (and we’ll likely take that approach in the future) but it wasn’t at the level of maturity that we needed yet.

We also considered running a separate cluster for the new version, but this would come with serious costs in additional hardware, maintenance and operational overhead.

We dove deep into all these different directions and each time had to back out as they had serious drawbacks. How should the future of our stack maintenance look like? What mature enough solution could give us continuity guarantees while maintaining our agility?

Nix

We’d been following Nix for a long time as a possible solution. We have been building our software with a simple compile framework that Felix wrote back in 2012 which implemented some of Nix’ basic ideas and qualities already. It’s been working very well for us all these years, but Nix takes these, and additional concepts much further.

Nix is a functional programming language that let’s you define how to build software in a declarative fashion. By making use of a global cache, you only have to compile things that are unique to your environment, but in many cases you can just pull the resulting software straight from this cache. This can greatly improve build times as there is no building *waves hand *

After a good amount of learning and experimentation we figured out how Nix and depmake could co-exist, so that we can already make use of the new, without throwing away the old.

Future

This opens a floodgate to new software and is going to make it extremely easy for us to roll out new stack, customize it, ensure it keeps it working in the same way forever, and allow multiple versions of the same software alongside one another.

The gist of the work in providing new stack to our customers has now moved from building to testing it, which is a great improvement.

We’re still considering wrapping our Nix-built software in containers and running them root-less, but this would primarily be to deliver added security on top of the containment that we already provide.

Will we ever deprecate our old versions? Yes. But we’ll do so with grace-periods where we gently nudge people towards the new.

Without further ado.. version two!

So now we present to you our new ImageMagick stack nicked v2.0.3 (not SemVer, any change is potentially BC breaking). It’s been in private beta for two weeks and is now entering public beta. It’s based on the latest ImageMagick version that Nix provides but with custom additions such as libraw so that our customers can enjoy those 131 extra formats.

Besides RAW and all the other formats that we already support, customers will love that the new stack comes with support for WEBP and DjVu formats.

We have updated our supported formats page so you can more easily discover differences between our stack versions, just click the “Compare” button.

For your convenience, here are the full lists:

Gained read support: 3FR, 3G2, 3GP, CANVAS, CLIP (previously only write), FILE, FTP, HALD, HTTP, IIQ, JNX, MAC, MEF, NRW, PES, RAW, RMF, RW2, SCREENSHOT, WMF, WMZ.

Gained write support: BRF, DDS (previously only read), INLINE (previously only read), ISOBRL, ISOBRL6, JSON, SPARSE, UBRL, UBRL6

Gained read/write support: AAI, BGRA, BGRO, CAL, CALS, DXT1, DXT5, G4, GROUP4, HDR, JPE, JPS, MASK, MKV, PNG00, PNG48, PNG64, PSB, RGF, SIX, SIXEL, TIFF64, VIPS, WEBP

Besides more formats, our new ImageMagick version also enables HDRI (High Dynamic Range Imaging) by default, so you can expect more accurate image processing results.

Aside from these customer facing upgrades, we’re quite happy to that the new version has operational benefits for us like the plugging of a number of security vulnerabilities, memory leaks, etc.

Notable breaking changes

Our previous ImageMagick had a bug in colorspace handling resulting in darker images and incorrectly reporting sRGB vs RGB. This is now fixed, but if you had worked around this bug, you are relying on it and so will need to compensate when you choose to upgrade.

We no longer support reading PGX and JPX files, and for the formats: BRG, GBR, GRB, RBG it is now recommended to create an RGB format and then swap channels afterwards.

For reference, you can find a full changelog at the bottom of this post.

Upgrading

Transloadit recommends to start using the new ImageMagick stack for testing and non-critical workloads. You can do this by adding imagemagick_stack: "v2.0.3" in your /image/resize Robot Step.

Our apologies for taking so long to deliver on this upgrade, you can expect much timelier updates thanks to our new way of stack herding :smile:

We’d love to hear your findings so do let us know what you think via the well known speech bubble on our site!

Changelog

For reference, here is the a full changelog:

  • Add additional checks to DCM reader to prevent data-driven faults (bug report from Hanno Böck)
  • Add final ClampToQuantum in sigmoidal colormap loop
  • Add support for languages that require complex text layout (reference)
  • Add tanh/atanh clone of legacy sigmoidal map (faster & more accurate)
  • Added define psd:additional-info to preserve the additional information in a PSD file
  • Added define psd:preserve-opacity-mask to preserve the opacity mask in a PSD file
  • Added layer RLE compression to the PSD encoder
  • Added layer RLE compression to the PSD encoder
  • Added support for GROUP4 compression to the FAX coder
  • Added support for RGB555, RGB565, ARGB4444 and ARGB1555 to the BMP encoder (reference)
  • Allow the use of set and escapes when no images in memory (unless you attempt to access per-image meta-data) Currently does not include %[fx:...] and %[pixel:...]
  • Apply Debian patches (reference)
  • Backoff finite precision epsilon (reference)
  • Bump Magick++ SO. Previously a global replace changed matteColor to alphaColor
  • Can read geo-related EXIF metdata once-again (reference)
  • Check for buffer overflow in magick/draw.c/DrawStrokePolygon()
  • Coder path traversal is not authorized (bug report provided by Masaaki Chida)
  • coders/png.c: Added support for a proposed new PNG chunk (exIf read-write, eXIf read-only) that is currently being discussed on the png-mng-misc at lists.sourceforge.net mailing list
  • coders/png.c: Added support for a proposed new PNG chunk (zxIf, read-only) that is currently being discussed on the png-mng-misc at lists.sourceforge.net mailing list. Enable exIf and zxIf with CPPFLAGS="-DexIf_SUPPORTED -DxzIf_SUPPORTED" If exIf is enabled, only the uncompressed exIF chunk will be written and the hex-encoded zTXt chunk containing the raw Exif profile won’t be written.
  • Correct for numerical instability (reference)
  • Correction to composite Over operator (reference)
  • Define CompositeChannels mask to Red, Green, Blue, Alpha, and Black
  • Deny indirect reads by policy, remove policy to permit, e.g., convert caption:@mytext.txt
  • Distort no longer converts grayscale image to sRGB (reference)
  • Do not close path for linejoins of round (reference)
  • Document behavior change in the security policy (thanks to yoya)
  • Don’t interpret -fx option arguments (reference)
  • Don’t return a zero bounding box for QueryMultilineFontMetrics() (reference)
  • Don’t set background for transparent tiled images (reference)
  • Don’t set update trait on alpha channel (private e-mail concerning -levels-colors option)
  • Don’t sync pixel cache in AcquireAuthenticCacheView() (bug report from Hanno Böck)
  • Eliminate compiler warning
  • Enable alpha channel if background color is non-opaque (reference)
  • Evaluate lazy pixel cache morphology to prevent buffer overflow (bug report from Ibrahim M. El-Sayed)
  • Fix compile error in opencl.c (reference)
  • Fix drawing glitch for stroke widths greater than 2 (reference)
  • Fix for possible security vulnerabilities (reference)
  • Fix GetNextToken() off by one error
  • Fix memory leak in the MPC format
  • Fix MVG stroke-opacity (reference)
  • Fix pixel cache on disk regression (reference)
  • Fix possible buffer overflow when writing compressed TIFFS (vulnerability report from Cisco Talos, CVE-2016-8707)
  • Fix re-declaration of i (at the top, and inside a conditional)
  • Fix small memory leak (patch provided by Андрей Черный)
  • Fix stroke offset problem for -annotate (reference)
  • Fixed fd leak for webp coder (reference)
  • Fixed improper scaling of certain FITS images (reference)
  • Fixed incorrect padding calculation in PSD encoder
  • Fixed incorrect parsing with ordered dither (reference)
  • Fixed incorrect RLE decoding when reading a DCM image that contains multiple segments
  • Fixed incorrect RLE decoding when reading an SGI image (reference)
  • Fixed issue where the display window was used instead of the data window when reading EXR files (reference)
  • Fixed memory leak when creating nested exceptions in Magick++ (reference)
  • Fixed proper placement of text annotation for east / west gravity
  • Fixed reading DXT1 images with an alpha channel.
  • If a convenient line break is not found, force it for caption: (reference)
  • Implemented a private PNG caNv (canvas) chunk for remembering the original dimensions and offsets when an image is cropped. Previously we used the oFFs and vpAg chunks for this purpose, but this had potential conflicts with other applications that also use the oFFs chunk
  • Increase memory allocation for TIFF pixels (reference)
  • Initialize draw_info alpha member to OpaqueAlpha
  • Initialize index channel to get expected results from the stegano coder
  • Iterate channels over source image rather than destination (bug report from Hanno Böck)
  • Mask composite produces proper results for the convert utility (reference)
  • Monochrome images no longer have inverted colors (reference)
  • Note alpha channel when combining 4 or more images (reference)
  • Off by 1 error when computing the standard deviation (reference)
  • Off by one memory allocation (reference)
  • Patch so -kuwahara option can preserve colormapped edges
  • Permit EPT images with just a TIFF or EPS image, not both (reference)
  • Prevent buffer overflow (bug report from Max Thrane)
  • Prevent buffer overflow and other problems in SIXEL, PDB, MAP, TIFF and CALS coders (bug report from Donghai Zhu)
  • Prevent buffer overflow in BMP & SGI coders (bug report from pwchen&rayzhong of tencent)
  • Prevent buffer overflow when streaming an image (reference)
  • Prevent fault in MSL interpreter (reference)
  • Prevent memory use after free (reference)
  • Prevent possible buffer overflow when reading TIFF images (bug report from Shi Pu of MS509 Team)
  • Prevent possible shell command injection vulnerability through the authenticate parameter of the PDF, PCL and XPS coders (report from Erez Turjeman)
  • Prevent random pixel data for corrupt JPEG image (bug report from Hirokazu Moriguchi, Sony)
  • Prevent spurious removal of MPC cache files (reference)
  • Process channels independently for -channel -equalize (reference)
  • Properly auto-fit caption (reference)
  • Properly center text label (reference)
  • Properly initialze PES blocks (reference)
  • Quote passwords when passed to a delegate program.
  • Rather than replicate options into artifacts make a link from image to image_info and lookup a global option if no artifact is defined.
  • Recognize XML policy closing tags (reference)
  • Remove https delegate
  • Remove OpenMP calls from colormap update loops
  • Remove support for internal ephemeral coder
  • Renamed read_vpag_chunk_callback() function to png_user_chunk_callback() in coders/png.c
  • Render to clip mask rather than image for clip-path MVG graphics primitive
  • Replace show delegate title with image filename rather than label
  • Replaced CoderSeekableStreamFlag with CoderDecoderSeekableStreamFlag and CoderEncoderSeekableStreamFlag
  • Respect connected-components:area-threshold define (reference)
  • Respect gravity option (reference)
  • Restore -mattecolor option
  • Return correct offset for negative index for -fx option (reference)
  • Return unbiased standard deviation for image statistics (reference)
  • Revert patch that did not set update trait on alpha channel
  • RLE check for pixel offset less than 0 (heap overflow report from Craig Young)
  • Sanitize all delegate emedded formatting characters
  • Sanitize comments that include braces for the MIFF image format (reference)
  • Sanitize input filename for http / https delegates (improved patch)
  • Security improvements to TEXT coder broke it (reference)
  • Set alpha member of draw structure to OpaqueAlpha (reference)
  • Set colorspace to sRGB if -append has non-homogenous colorspaces (reference)
  • sigmoidal-contrast: Direct computation, without LUT
  • sigmoidal-contrast: Remove unnecessary initial ClampToQuantum
  • Support -region option (reference)
  • Support configure script --enable-indirect-reads option to enable indirect reads (@) in filenames
  • Support configure script --enable-pipes option to enable pipes (|) in filenames
  • Support pixel-cache and shred security policies
  • Support read-masks for the -modulate option
  • Support the compare -read-mask option
  • Support the phash:colorspaces and phash:normalize options
  • The -clone option no longer leaks memory
  • The -extent option now matches the results of IMv6 (reference)
  • The -stream option now increments the pixel pointer properly (reference)
  • The histogram coder now returns the correct extent
  • To comply with the SVG standard, use stroke-opacity for transparent strokes
  • Turn off alpha channel for the compare difference image (reference)
  • Unbreak build without JPEG support (reference)
  • Uninitialized data in MAT image format (reference)
  • Unit test pass again after small SUN image patch
  • Use CopyMagickString() rather than CopyMagickMemory() for strings
  • Validation unit test for MNG works again
  • Web pages were broken when we moved to HTTPS protocol



Source link