Discussion:
The conversion from PDF to DJVU loses too much quality
(too old to reply)
Andrea Zanni
2017-01-24 11:03:00 UTC
Permalink
Raw Message
I added this issue to IA-upload tool on github:
https://github.com/Tpt/ia-upload/issues/14

Unfortunately, the new PDF > DJVU conversion is useless, as it loses too
much quality.
Can we find a solution?
The IA-Upload tool is a great asset for the whole international community,
and it's very simple to teach librarians to upload stuff on IA and then use
it to port it on Commons and Wikisource.
But when they upload new stuff on IA, we don't have the IA djvu anymore.
So the tool converts the original PDF to a new DJVU, and this is the part
of the process that is failing.

I can think of 2 solutions:
* integrate this script from Alex brollo into the tool:
https://it.wikisource.org/wiki/Progetto:Bot/Programmi_
in_Python_per_i_bot/jp2todjvu.py
the script creates a good quality djvu
* have a toggle/top-down menu which allow the user to use directly the PDF.

Andrea
Alex Brollo
2017-01-24 16:16:26 UTC
Permalink
Raw Message
My tool is very rough, and some recent tests show that is not sufficiently
generalized - it fails into some IA items; so I thing that it can
considered simply a proof that a tool, that uses IA jp2 images (that are
shown into the IA viewer IMHO) and djvy_xml can be merged into a
high-quality djvu file.

Alex brollo
Post by Andrea Zanni
https://github.com/Tpt/ia-upload/issues/14
Unfortunately, the new PDF > DJVU conversion is useless, as it loses too
much quality.
Can we find a solution?
The IA-Upload tool is a great asset for the whole international community,
and it's very simple to teach librarians to upload stuff on IA and then
use it to port it on Commons and Wikisource.
But when they upload new stuff on IA, we don't have the IA djvu anymore.
So the tool converts the original PDF to a new DJVU, and this is the part
of the process that is failing.
https://it.wikisource.org/wiki/Progetto:Bot/Programmi_in_
Python_per_i_bot/jp2todjvu.py
the script creates a good quality djvu
* have a toggle/top-down menu which allow the user to use directly the PDF.
Andrea
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Yann Forget
2017-01-24 18:05:35 UTC
Permalink
Raw Message
Hi,

The quality of the result depends of paramaters.
I use Abby FineReader, and I use the high quality possible, which gives
good quality DjVu, but of high volume.
It is always a trade-off between quality and volume.

Regards,

Yann
Post by Andrea Zanni
https://github.com/Tpt/ia-upload/issues/14
Unfortunately, the new PDF > DJVU conversion is useless, as it loses too
much quality.
Can we find a solution?
The IA-Upload tool is a great asset for the whole international community,
and it's very simple to teach librarians to upload stuff on IA and then
use it to port it on Commons and Wikisource.
But when they upload new stuff on IA, we don't have the IA djvu anymore.
So the tool converts the original PDF to a new DJVU, and this is the part
of the process that is failing.
https://it.wikisource.org/wiki/Progetto:Bot/Programmi_in_
Python_per_i_bot/jp2todjvu.py
the script creates a good quality djvu
* have a toggle/top-down menu which allow the user to use directly the PDF.
Andrea
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Sam Wilson
2017-01-25 00:45:34 UTC
Permalink
Raw Message
Yann, do you mean you're getting good quality DjVu generated from the
PDF? Or from the original scan Jpegs?


Aubrey: when you say directly use the PDF, you mean for the tool to copy that across to Commons and not create a DjVu?


I can have a look at adding that feature perhaps? (Anyone else
working on this?)


—sam
Post by Yann Forget
Hi,
The quality of the result depends of paramaters.
I use Abby FineReader, and I use the high quality possible, which
gives good quality DjVu, but of high volume.
It is always a trade-off between quality and volume.
Regards,
Yann
Post by Andrea Zanni
https://github.com/Tpt/ia-upload/issues/14
Unfortunately, the new PDF > DJVU conversion is useless, as it loses
too much quality.
Can we find a solution?
The IA-Upload tool is a great asset for the whole international community,
and it's very simple to teach librarians to upload stuff on IA and
then use it to port it on Commons and Wikisource.
But when they upload new stuff on IA, we don't have the IA djvu anymore.
So the tool converts the original PDF to a new DJVU, and this is the
part of the process that is failing.
https://it.wikisource.org/wiki/Progetto:Bot/Programmi_in_Python_per_i_bot/jp2todjvu.py
the script creates a good quality djvu
* have a toggle/top-down menu which allow the user to use directly the PDF.
Andrea
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_________________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Andrea Zanni
2017-01-25 07:27:27 UTC
Permalink
Raw Message
Yann, do you mean you're getting good quality DjVu generated from the PDF?
Or from the original scan Jpegs?
AFAIU, Yann is using ABBYY finereader to generate a djvu and then uploads
it directly to Commons. So outside of our ia-upload tool.
Aubrey: when you say directly use the PDF, you mean for the tool to copy
that across to Commons and not create a DjVu?
Yes.
If the Djvu quality is much lower than the PDF there's no reason to use the
djvu over the pdf :-(
I can have a look at adding that feature perhaps? (Anyone else working on
this?)
Please ;-)
Aubrey
Sam Wilson
2017-01-25 07:40:27 UTC
Permalink
Raw Message
On Wed, Jan 25, 2017 at 1:45 AM, Sam Wilson
Post by Sam Wilson
__
Yann, do you mean you're getting good quality DjVu generated from the
PDF? Or from the original scan Jpegs?
AFAIU, Yann is using ABBYY finereader to generate a djvu and then
uploads it directly to Commons. So outside of our ia-upload tool.
Ah, okay. So if it could be done in the tool, that'd be nicer.
Post by Sam Wilson
Aubrey: when you say directly use the PDF, you mean for the tool to
copy that across to Commons and not create a DjVu?
Yes.
If the Djvu quality is much lower than the PDF there's no reason to
use the djvu over the pdf :-(
Are we saying that we *never* want to use the IA PDF? That if there's a
DjVu we use it, and if there isn't we generate our own DjVu from the JP2
and djvu.xml files? Or should the tool user make this call and we give
them a drop-down list of "PDF only", "Generate DjVu from PDF", and
"Generate DjVu from original scans" with a note about the last of these
being higher quality but slower?


I think I'm in favour of just generating a high-quality DjVu and
making it simpler for the end user. But we want to be flexible too.
jayantanth mentioned[1] that he'd like to be able to just upload the
PDF for example.
Post by Sam Wilson
I can have a look at adding that feature perhaps? (Anyone else
working on this?)
Please ;-)
I can try! :-)
Aubrey
_________________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Links:

1. https://github.com/Tpt/ia-upload/issues/15
Andrea Zanni
2017-01-25 09:07:49 UTC
Permalink
Raw Message
Post by Sam Wilson
Are we saying that we *never* want to use the IA PDF? That if there's a
DjVu we use it, and if there isn't we generate our own DjVu from the JP2
and djvu.xml files? Or should the tool user make this call and we give them
a drop-down list of "PDF only", "Generate DjVu from PDF", and "Generate
DjVu from original scans" with a note about the last of these being higher
quality but slower?
I think I'm in favour of just generating a high-quality DjVu and making it
simpler for the end user. But we want to be flexible too. jayantanth
mentioned <https://github.com/Tpt/ia-upload/issues/15> that he'd like to
be able to just upload the PDF for example.
Ideally, a menu is perfect, with the "default" of the best solution :-)
Right now, unfortunately, the djvu is not sufficient quality, when
processed from a PDF.
Yann Forget
2017-01-25 10:35:31 UTC
Permalink
Raw Message
Yann, do you mean you're getting good quality DjVu generated from the PDF?
Or from the original scan Jpegs?
AFAIU, Yann is using ABBYY finereader to generate a djvu and then uploads
it directly to Commons. So outside of our ia-upload tool.
Ah, okay. So if it could be done in the tool, that'd be nicer.
Yes, it is a question of settings.
Aubrey: when you say directly use the PDF, you mean for the tool to copy
that across to Commons and not create a DjVu?
Yes.
If the Djvu quality is much lower than the PDF there's no reason to use
the djvu over the pdf :-(
DjVu has to advantages over PDF: better compression, so small files for
the same content, and better management of the text layer.
Over if the compression is too high, the quality is not good. It is a
question of a compromise between quality and size.

Yann
Are we saying that we *never* want to use the IA PDF? That if there's a
DjVu we use it, and if there isn't we generate our own DjVu from the JP2
and djvu.xml files? Or should the tool user make this call and we give them
a drop-down list of "PDF only", "Generate DjVu from PDF", and "Generate
DjVu from original scans" with a note about the last of these being higher
quality but slower?
I think I'm in favour of just generating a high-quality DjVu and making it
simpler for the end user. But we want to be flexible too. jayantanth
mentioned <https://github.com/Tpt/ia-upload/issues/15> that he'd like to
be able to just upload the PDF for example.
I can have a look at adding that feature perhaps? (Anyone else working on
this?)
Please ;-)
I can try! :-)
Aubrey
*_______________________________________________*
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Alex Brollo
2017-01-26 10:26:39 UTC
Permalink
Raw Message
By now IA pdf too are very compressed, sometimes too much - the result
being impredictable; the problem is, that viewer doesn't uses djvu nor pdf
IMHO, so the quality of pdf (and of resulting djvu by pdf2djvu) doesn't
mirror at all the quality of viewer images.

The IA pdf needs a good review before upload it into Commons.

There are subltle advantages using djvu instead of pdf, i.e. fixing errors
into source file (adding/deleting/moving pages, manipulating text layer);
djvu is a great "wiki" format since it is *open*.

Alex
Post by Yann Forget
Post by Sam Wilson
Yann, do you mean you're getting good quality DjVu generated from the
PDF? Or from the original scan Jpegs?
AFAIU, Yann is using ABBYY finereader to generate a djvu and then uploads
it directly to Commons. So outside of our ia-upload tool.
Ah, okay. So if it could be done in the tool, that'd be nicer.
Yes, it is a question of settings.
Aubrey: when you say directly use the PDF, you mean for the tool to copy
that across to Commons and not create a DjVu?
Yes.
If the Djvu quality is much lower than the PDF there's no reason to use
the djvu over the pdf :-(
DjVu has to advantages over PDF: better compression, so small files for
the same content, and better management of the text layer.
Over if the compression is too high, the quality is not good. It is a
question of a compromise between quality and size.
Yann
Post by Sam Wilson
Are we saying that we *never* want to use the IA PDF? That if there's a
DjVu we use it, and if there isn't we generate our own DjVu from the JP2
and djvu.xml files? Or should the tool user make this call and we give them
a drop-down list of "PDF only", "Generate DjVu from PDF", and "Generate
DjVu from original scans" with a note about the last of these being higher
quality but slower?
I think I'm in favour of just generating a high-quality DjVu and making
it simpler for the end user. But we want to be flexible too. jayantanth
mentioned <https://github.com/Tpt/ia-upload/issues/15> that he'd like to
be able to just upload the PDF for example.
I can have a look at adding that feature perhaps? (Anyone else working on
this?)
Please ;-)
I can try! :-)
Aubrey
*_______________________________________________*
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Andrea Zanni
2017-01-26 10:35:16 UTC
Permalink
Raw Message
The problem for me is that librarians and other people who are genuinely
interested in Wikisource and IA
don't understand why
* they upload a good scan on IA
* see a good book on IA, via the viewer
* get an horrible djvu on Wikisource.

This is the issue we should try to solve, otherwise we will lose a
potential important ally, content and new userbase.

Aubrey
Post by Alex Brollo
By now IA pdf too are very compressed, sometimes too much - the result
being impredictable; the problem is, that viewer doesn't uses djvu nor pdf
IMHO, so the quality of pdf (and of resulting djvu by pdf2djvu) doesn't
mirror at all the quality of viewer images.
The IA pdf needs a good review before upload it into Commons.
There are subltle advantages using djvu instead of pdf, i.e. fixing errors
into source file (adding/deleting/moving pages, manipulating text layer);
djvu is a great "wiki" format since it is *open*.
Alex
Post by Yann Forget
Post by Sam Wilson
Yann, do you mean you're getting good quality DjVu generated from the
PDF? Or from the original scan Jpegs?
AFAIU, Yann is using ABBYY finereader to generate a djvu and then
uploads it directly to Commons. So outside of our ia-upload tool.
Ah, okay. So if it could be done in the tool, that'd be nicer.
Yes, it is a question of settings.
Aubrey: when you say directly use the PDF, you mean for the tool to copy
that across to Commons and not create a DjVu?
Yes.
If the Djvu quality is much lower than the PDF there's no reason to use
the djvu over the pdf :-(
DjVu has to advantages over PDF: better compression, so small files for
the same content, and better management of the text layer.
Over if the compression is too high, the quality is not good. It is a
question of a compromise between quality and size.
Yann
Post by Sam Wilson
Are we saying that we *never* want to use the IA PDF? That if there's a
DjVu we use it, and if there isn't we generate our own DjVu from the JP2
and djvu.xml files? Or should the tool user make this call and we give them
a drop-down list of "PDF only", "Generate DjVu from PDF", and "Generate
DjVu from original scans" with a note about the last of these being higher
quality but slower?
I think I'm in favour of just generating a high-quality DjVu and making
it simpler for the end user. But we want to be flexible too. jayantanth
mentioned <https://github.com/Tpt/ia-upload/issues/15> that he'd like
to be able to just upload the PDF for example.
I can have a look at adding that feature perhaps? (Anyone else working
on this?)
Please ;-)
I can try! :-)
Aubrey
*_______________________________________________*
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Sam Wilson
2017-01-26 23:10:56 UTC
Permalink
Raw Message
Post by Andrea Zanni
The problem for me is that librarians and other people who are
genuinely interested in Wikisource and IA
don't understand why
* they upload a good scan on IA
* see a good book on IA, via the viewer
* get an horrible djvu on Wikisource.
This is the issue we should try to solve, otherwise we will lose a
potential important ally, content and new userbase.
Aubrey
Definitely!



On a related note: most (all?) IA-scanned books have e.g. *_jp2.zip
files containing all the original scan images, but is there any standard
for user-uploaded books? Like your librarians above, I assume they're
uploading individual jpg/png files? Do these get combined into a single
zip? I'm thinking that they don't, and that ia-upload needs to provide
the option of using any of the following sources:
* .djvu
* _jp2.zip (there's also _jpg.zip and _raw_jp2.zip, but I guess we
don't need to use them?)
* *.jpg + *.jp2 + *.png (i.e. use all images in the item, apart from
_cover_image.jpg)
* .pdf


Sound complete? Or are there other ways?
Andrea Zanni
2017-01-26 23:40:16 UTC
Permalink
Raw Message
AFAIK, IA always produce the jp2 files by himself.

I suggest GLAMs to upload zipped folders of jpegs,
so IA can do his magic and produce a book viewer and a PDF as well as the
jp2.
Post by Andrea Zanni
The problem for me is that librarians and other people who are genuinely
interested in Wikisource and IA
don't understand why
* they upload a good scan on IA
* see a good book on IA, via the viewer
* get an horrible djvu on Wikisource.
This is the issue we should try to solve, otherwise we will lose a
potential important ally, content and new userbase.
Aubrey
Definitely!
On a related note: most (all?) IA-scanned books have e.g. *_jp2.zip files
containing all the original scan images, but is there any standard for
user-uploaded books? Like your librarians above, I assume they're uploading
individual jpg/png files? Do these get combined into a single zip? I'm
thinking that they don't, and that ia-upload needs to provide the option of
- .djvu
- _jp2.zip (there's also _jpg.zip and _raw_jp2.zip, but I guess we
don't need to use them?)
- *.jpg + *.jp2 + *.png (i.e. use all images in the item, apart from
_cover_image.jpg)
- .pdf
Sound complete? Or are there other ways?
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Sam Wilson
2017-01-26 23:53:51 UTC
Permalink
Raw Message
Good to know, thanks!



So, we just stick with jp2.zip



And I love the IA magic :)
Post by Andrea Zanni
AFAIK, IA always produce the jp2 files by himself.
I suggest GLAMs to upload zipped folders of jpegs,
so IA can do his magic and produce a book viewer and a PDF as well as
the jp2.
On Fri, Jan 27, 2017 at 12:10 AM, Sam Wilson
Post by Sam Wilson
__
Post by Andrea Zanni
The problem for me is that librarians and other people who are
genuinely interested in Wikisource and IA
don't understand why
* they upload a good scan on IA
* see a good book on IA, via the viewer
* get an horrible djvu on Wikisource.
This is the issue we should try to solve, otherwise we will lose a
potential important ally, content and new userbase.
Aubrey
Definitely!
On a related note: most (all?) IA-scanned books have e.g. *_jp2.zip
files containing all the original scan images, but is there any
standard for user-uploaded books? Like your librarians above, I
assume they're uploading individual jpg/png files? Do these get
combined into a single zip? I'm thinking that they don't, and that
ia-upload needs to provide the option of using any of the following
* .djvu
* _jp2.zip (there's also _jpg.zip and _raw_jp2.zip, but I guess we
don't need to use them?)
* *.jpg + *.jp2 + *.png (i.e. use all images in the item, apart from
_cover_image.jpg)
* .pdf
Sound complete? Or are there other ways?
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_________________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Alex Brollo
2017-01-27 07:24:20 UTC
Permalink
Raw Message
Yes, presently IA jp2.zip are the source files for all derived ones and for
OCR. All the derived ones are omologous - t.i. *relative* coordinates of
any element inside images are identical, even if image size varies. This
means that mapping of elements (images or text) can be exported into any
derived file.

Just an example: when an user crops an image from a djvu file by the
excellent CropTool by Danmichaelo, coordinated of the cropping could be
used to crop high-resolution jp2 or jpg image, or to get coordinates of any
piece of text mapped by OCR.

Alex
Post by Sam Wilson
Good to know, thanks!
So, we just stick with jp2.zip
And I love the IA magic :)
AFAIK, IA always produce the jp2 files by himself.
I suggest GLAMs to upload zipped folders of jpegs,
so IA can do his magic and produce a book viewer and a PDF as well as the
jp2.
The problem for me is that librarians and other people who are genuinely
interested in Wikisource and IA
don't understand why
* they upload a good scan on IA
* see a good book on IA, via the viewer
* get an horrible djvu on Wikisource.
This is the issue we should try to solve, otherwise we will lose a
potential important ally, content and new userbase.
Aubrey
Definitely!
On a related note: most (all?) IA-scanned books have e.g. *_jp2.zip files
containing all the original scan images, but is there any standard for
user-uploaded books? Like your librarians above, I assume they're uploading
individual jpg/png files? Do these get combined into a single zip? I'm
thinking that they don't, and that ia-upload needs to provide the option of
- .djvu
- _jp2.zip (there's also _jpg.zip and _raw_jp2.zip, but I guess we
don't need to use them?)
- *.jpg + *.jp2 + *.png (i.e. use all images in the item, apart from
_cover_image.jpg)
- .pdf
Sound complete? Or are there other ways?
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
*_______________________________________________*
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Sam Wilson
2017-01-27 08:44:58 UTC
Permalink
Raw Message
It's a pretty cool format. :)



I have got the beginnings of a PHP rewrite of your python script
running https://github.com/Tpt/ia-upload/pull/18 (but it's not at all
finished yet).


What is the best way to decrease the size of the jpgs before creating
the djvu? Just scale them to ~1000 px or something? How do you handle
that? (Sorry, I've read your code, but am confused...) I'm using
imagemagick to do it, so any transformation it can do is easy to
implement.


—Sam
Post by Alex Brollo
Yes, presently IA jp2.zip are the source files for all derived ones
and for OCR. All the derived ones are omologous - t.i. *relative*
coordinates of any element inside images are identical, even if image
size varies. This means that mapping of elements (images or text) can
be exported into any derived file.
Just an example: when an user crops an image from a djvu file by the
excellent CropTool by Danmichaelo, coordinated of the cropping could
be used to crop high-resolution jp2 or jpg image, or to get
coordinates of any piece of text mapped by OCR.
Alex
Post by Sam Wilson
__
Good to know, thanks!
So, we just stick with jp2.zip
And I love the IA magic :)
Post by Andrea Zanni
AFAIK, IA always produce the jp2 files by himself.
I suggest GLAMs to upload zipped folders of jpegs,
so IA can do his magic and produce a book viewer and a PDF as well
as the jp2.
Post by Sam Wilson
__
Post by Andrea Zanni
The problem for me is that librarians and other people who are
genuinely interested in Wikisource and IA
don't understand why
* they upload a good scan on IA
* see a good book on IA, via the viewer
* get an horrible djvu on Wikisource.
This is the issue we should try to solve, otherwise we will lose a
potential important ally, content and new userbase.
Aubrey
Definitely!
On a related note: most (all?) IA-scanned books have e.g. *_jp2.zip
files containing all the original scan images, but is there any
standard for user-uploaded books? Like your librarians above, I
assume they're uploading individual jpg/png files? Do these get
combined into a single zip? I'm thinking that they don't, and that
ia-upload needs to provide the option of using any of the following
* .djvu
* _jp2.zip (there's also _jpg.zip and _raw_jp2.zip, but I guess we
don't need to use them?)
* *.jpg + *.jp2 + *.png (i.e. use all images in the item, apart
from _cover_image.jpg)
* .pdf
Sound complete? Or are there other ways?
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_________________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_______________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
_________________________________________________
Wikisource-l mailing list
https://lists.wikimedia.org/mailman/listinfo/wikisource-l
Federico Leva (Nemo)
2017-01-27 07:33:09 UTC
Permalink
Raw Message
Post by Sam Wilson
On a related note: most (all?) IA-scanned books have e.g. *_jp2.zip
files containing all the original scan images, but is there any standard
for user-uploaded books? Like your librarians above, I assume they're
uploading individual jpg/png files? Do these get combined into a single zip?
https://en.wikisource.org/wiki/Help:DjVu_files#The_Internet_Archive
covers this.

Nemo
Loading...