Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Falls back to blocky decoding when one pass is interrupted by another #521

Open
jakearchibald opened this issue May 18, 2021 · 9 comments
Open
Assignees
Labels

Comments

@jakearchibald
Copy link

jakearchibald commented May 18, 2021

Have you searched the existing issues (both open and closed) in the libjpeg-turbo issue tracker to ensure that this bug report is not a duplicate? #343 seems related, but that issue is closed.

Does this bug report describe one of the two known and unsolvable issues with the JPEG format? I don't think so.

Clear and concise description of the bug:

While rendering a progressive image, block smoothing appears to stop when one pass interrupts another.

Steps to reproduce the bug (using only libjpeg-turbo):

Apologies, I'm not familiar enough with libjpeg-turbo, but I have a browser demo, and I'm assured that part of the issue is in libjpeg-turbo, although there may need to be changes in Chrome and Firefox too.

  • Images
  • App (unfortunately the app only works in Chrome due to missing primitives in Firefox)

The app will slowly load a given image, stopping at a particular byte. From the set of images provided, here's cat.jpg stopped at 45538 bytes:

Screenshot 2021-05-18 at 15 27 33

In this example, block smoothing isn't working on the luma channel (at least).

Here's ic-pr.jpg, stopped at 47424 bytes:

Screenshot 2021-05-18 at 15 30 27

In this example, block smoothing isn't working on at least one chroma channel (see the stripes on the jumper).

Expected behavior:

The decoding goes from smooth to sharp, rather than switching to blocky.

Additional information:

I can only reproduce this by causing multiple passes, if I simply truncate the file it renders as expected. The app also supports simple truncation (see the dropdown).

@mo271
Copy link

mo271 commented May 18, 2021

Thank you for finding that! The fact that it is not a problem with truncation points towards being caused by skipping scans when decoding, e.g. displaying the 7th scan on top and the 1st scan (DC only) on the bottom, perhaps this has something to do with what coef_bits_latch stores.

@dcommander
Copy link
Member

It's probably expected behavior, the result of not enough information yet being decoded to support block smoothing. I'll investigate, though.

@jakearchibald
Copy link
Author

jakearchibald commented May 18, 2021

The app has a couple of options:

  • "partial load, multi pass" - This loads the image at 2g speeds and stops at a particular byte
  • "partial load" - This loads the image as fast as possible and stops at a particular byte

Only the former recreates the bug. I think this is because multiple scans are rendered in the former, but not the latter. The latter produces the same partial image, but without the blockiness. Given that they're working with the same amount of data, I think something else is going on here.

@jakearchibald
Copy link
Author

To illustrate the above, here's what it looks like when the image downloads at ~2g speeds and stops at 45625 bytes (resulting in many render passes):

Screenshot 2021-05-18 at 17 06 09

Whereas here's what it looks like when the browser receives all 45625 bytes at once (resulting in one render pass):

Screenshot 2021-05-18 at 17 08 13

@dcommander
Copy link
Member

OK, well, in that case, someone else needs to investigate it and figure out whether it's actually a bug in libjpeg-turbo. It takes a while for me to dig into these block smoothing issues, and since the specific funding for that feature has been exhausted, I'm not willing to spend any of the General Fund on this unless it can be shown to be a legitimate issue.

@jyrkialakuijala
Copy link

jyrkialakuijala commented Jun 16, 2021

The first variation where no interpolated AC coefficients are preserved for area which did not receive any AC coefficients seems like a bug that likely could be fixed.

The second image is more complicated.

My possibly flawed understanding of this:
The smoothing correctly and beautifully computes new virtual AC coefficients from the DC. This forms a consistent image.

When some actual AC coefficients are loaded, they don't make a consistent image neither together with the virtual AC coefficients, nor when the virtual AC coefficients have been replaced by zeros. Another strategy would be needed to get a consistent image at that stage.

Two possibilities to fix this come to mind:

  1. smoothing the image at such situations. Smoothing in pixel space is a lot more computation, i.e., deployments would have more performance risk, particularly so on low powered devices.

  2. wait before redrawing the image until a sufficient amount of coefficients have been delivered. The heuristic for delaying the second draw could be something like at least 14 AC coefficients have been send with no more than 2 bits missing from them (I'm just guessing, didn't do actual testing if this leads to better perceived quality than the initial smoothing guess). Another visually solid alternative would be to wait for the second draw until all but last bit of all AC coefficients has been sent.

Intuitively 2) will be easier to implement in real-life systems.

@dcommander
Copy link
Member

The other piece of information I need with regard to this issue is whether it also exists in libjpeg-turbo 2.0.x. I am still not convinced that it is a bug, and you'll probably observe that the behavior in libjpeg-turbo 2.0.x is worse, but it's still a good data point to have.

@dcommander
Copy link
Member

This issue has been languishing for over a year. Can anyone provide any additional details on it?

@mo271
Copy link

mo271 commented Jul 5, 2022

Thanks for bringing this up again! I don't have time to look at it right now, but plan to do is in the coming months.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants