crengine page mode is slow on opening some epubs

[dracodoc]

I found the newest build v2012.11-45-ge183bce is very slow on opening a medium size epub, which is only 1.6M, converted from mobi, "Structure and Interpretation of Computer".

The previous version take about 1 min to open the book, then it will open the book in several seconds as long as kpv is not closed then launch again -- I'm not sure the crengine cache still works after kpv launch again, it works in same session.

However with the current version, it took about 3 mins to open the same book. I believe it's related to the page view mode, and I cannot switch the mode before opening any epub file.

Besides, the go to position doesn't work in page mode too, I input 1 percent but it jump to the end of book, and it say current position is 1 for the end of book. I guess the go to position dialog is still in scroll mode.

[houqp]

Could you help me test it in scroll mode also? I want to make sure it's just page mode slows it down.

By editing default.lua, you should be able to set default view mode back to scroll mode, that way you can test it in view mode.

Thanks a lot.

[houqp]

Regarding the second bug, you are right, I have added the fix in RP

[kai771]

@dracodoc Can you please give Librerator a try, too? In Librerator, in page mode go to percent is replaced with go to page. There are a lot of other changes too, and I'm interested to hear your opinion. You can get it here.

[dracodoc]

@houqp , I changed the mode to scroll mode after I opened that epub, so I can open it without editing default.lua. I found with newly started kpv (it seemed kpv cannot really use the cache after a new session), it took a long time to open that epub even in scroll mode. My estimate is:

1. old version of kpv: 20 seconds.
2. new version page mode: about 3 mins.
3. new version scroll mode: about 1.5 mins.

Can you also have a look at the cache? If it works between sessions, the problem will be much smaller.

I put my epub here
https://www.box.com/s/2wzby5pl24tiamq74jrm

[dracodoc]

@kai771 , I tried Librerator with that epub.

1. the opening speed is same with kpv, i.e. it took 3 mins to open the epub in new session (even it has been opened before and have a cache available). It doesn't matter for scroll mode or page mode. If the cache is working (the epub has been opened in same session), the epub could be opened very quickly again in both modes.
2. the go to page works in Librerator.
   3.I found kpv doesn't recognize the table of contents of that epub(opening any item will go to the beginning of book), while Librerator can. What's the difference? Can you make it back to kpv to make it work too?
3. Another bug in kpv is solved in Librerator:
   I tried to choose font for rifont, but kpv can only get the host fonts for rifont, while kpv can get the other font I put in the fonts folder for tfont. Librerator don't have this problem, I can select my font under fonts folder for rifont.

Librerator looks good to me, though there are lots of changes and I'll need to move all my history files to start to use Librerator. I'm wondering if it's easy for you to incorporate new developed features in kpv to Librerator? Because I may want to use Librerator, but I also would like to see new features in kpv go to Librerator too.

[tigran123]

@dracodoc the toc problem is fixed in KPV too, but not yet committed to the master tree.

Concerning the fonts problem --- it is possible that you simply named it incorrectly in the KPV's "fonts" directory but correctly in Librerator's "fonts" directory. Please check that the filenames are exactly identical.

The goto page was fixed in KPV too, it is already committed into master, but no nightly build was generated yet.

[dracodoc]

It's not the problem of file name. When I first reported the problem in mobileread thread, I was comparing two font settings both in KPV. You can try this:

1. put some fonts under kpv fonts folder.
2. go to font setting, tfont, you can see all the droid fonts, host fonts and the custom fonts. Then if you go to rifont, you can only see host fonts.
   So there are two kinds of behavior in font settings, rifont, cfont can only read host fonts, but the other fonts can read all the fonts include custom fonts.

And now I found Librerator is same in this. Maybe I mistaken other settings in Librerator.

Besides, which fonts is responsible for the pdf table of contents fonts in KPV? It seemed that rifont only work for the reading position, not for the table of contents. The table of contents will use the same fonts of cfont, which can only be selected from kindle host fonts rightnow.

[kai771]

@dracodoc As far as I know, there are no features present in KPV that are not present in Librerator. Some things are done differently, but I tried to make it as configurable as possible. (for example, I prefer old style of "bookmark added" dialog, so that one is the default. But you can have your style if you set DKPV_STYLE_BOOKMARKS to true in defaults.lua.
History files didn't change. You should be able to copy the whole history folder from kindlepdfviewer to librerator folder without problems.

[dracodoc]

Yes I noticed that. The first thing I did is to change the bookmark style to KPV style, since that was actually "my style" :PP. I also copied the history files and began to use Librerator.

What I worried is if there are lots of difference between KPV and Librerator, it could very cumbersome to introduce new features into Librerator, and there is only you to do all the work.

[kai771]

@dracodoc Well, yeah, there are a lot of differences... and there will be much more when KPV releases touch support. Of course, there are no guarantees that I won't abandon it tomorrow... but it's open source, so...

Btw, have you tried screen rotation yet? ;)

[dracodoc]

What screen rotation? I noticed there are some changes in the screen rotation key but I didn't try it.

[kai771]

You can now rotate to landscape and back to portrait using the same key :). Try it. Both J and K work. And if you're interested, do have a look at History on wiki. It is a bit long, but you might find it interesting.

[houqp]

thanks @dracodoc , good to know that it's not page mode that slows everything down. I will take a look at it tonight and also the cache problem.

[dracodoc]

Yeah I skimmed through the history in wiki but didn't really notice the same key rotation. It's great, I like that, because I usually will always use one direction to rotate, so using one key to handle all rotations is good for me. I don't even remember if I suggested this by myself some time ago.

[kai771]

@dracodoc Yeah, you did... I implemented it specially for you :P.

[dracodoc]

Thanks, I'm honored...

This long discussion actually covered several problems, I'll list them here, maybe submit them separately after a while if they need more time to working on:

1. slow in open epub.
2. epub cache seemed not working between sessions. It could be it is still working but just need some time to read the cache, but I guess it's not the case because some epub cache are quite small and the reading cache should not take that much time.
3. font setting: rifont and cfont can only read host fonts, while some other font setting can read all fonts including droid fonts and custom fonts under fonts folder.
   //update: it's caused by select host fonts, once host font was selected, only host fonts can be seen. It could be related to fonthack too.
4. pdf table of content font is same with cfont (so the description should be more clear), which can only read host fonts now.
   //update: this is not a problem anymore if host fonts are avoided.

[houqp]

OK, I just did a quick check on font setting. I cannot reproduce it in my DXG, So far I cannot think of a reason why it does not work for you. Very likely this can be fixed by wipe out and re-install KPV from scratch. Or if you really want to help us fix that bug, please pack your whole KPV folder and put the link here.

[dracodoc]

I found the deeper cause of the problem:

1. I replaced one kindle font with Chinese font so that I can see Chinese file name in kindle original software. It was the Serif_Bold.ttf.
2. Once I choose this font as one font setting, this font setting will only recognize host fonts later. Actually it apply to all host fonts. So you can try this: select any host font in one font setting, quit then open this font setting again (you may need to quit kpv and start kpv, sometimes you don't), then you can only see host fonts, and there are 2 messed up font name in fonts list.

Since this happen to all host fonts, it is not the problem of my Chinese font.

I changed the entry in settings.lua back to droid fonts, then it can read all the fonts again. So the font setting will fall into some trap once it was set to host fonts. One simple solution is just ignore host fonts all together, user can always copy the font they want to fonts folder, even if they want to use some host fonts they can just copy them.

[houqp]

@dracodoc , you should go to bed at this time ;p

Nice found, but I still cannot reproduce it. Following is what I did:

• press f to invoke font menu
• press u to select font for cfont
• select one of the font
• exit KPV
• start KPV again and try to choose font for cfont

I can still see all the fonts in the font list, did I missed anything?

EDIT: I am going out now, will check back tonight.

[dracodoc]

I'm in U.S. now so it was noon for me :)

First, did you have fonthack? I installed fonthack to show Chinese file name, but I only changed one font to Chinese font.

I tested again, at least host/serif_bold, host/serif_Regular, host/Mono_Regular all have this effect.(I expect more have same result, but every attempt will trap one font into host mode so I didn't try all fonts). You don't even need quit kpv or quit font settings, just quit to the font setting screen then back to the font selection, you will see the effect.

Since I only changed Serif_bold, the other fonts are original kindle fonts, I don't think it's the problem of my fonts.

I've been wondering about the empty "host" folder in fonts folder for a long time, I'm guessing that was for a symbolic link to fonthack host folder, once some fonts there was chose, the path of fonts have been changed so the font selection can only see host fonts. The simple solution is just remove host folder and that link to avoid this trap.

[kai771]

@dracodoc host folder is where KPV mount binds system fonts on start, and unmounts them on exit. That's why it's empty if KPV is not running. If they're not mounted there, KPV would not be able to see system fonts, I think. In that case you would have to copy fonts to fonts folder in order for KPV to find them.

I'm not using fonthack, so I don't know what changes it does to the system.

[dracodoc]

If the font setting problem is not obvious to solve, I think it's ok to leave it as is. It should only affect fonthack user, and once user know the trap he can avoid using host fonts. If he already selected some host fonts, it can be modified in settings.lua. And the problem about pdf table of content font also can be avoided.

The major problem now is the slow opening of epub, and the cache problem. I think the cache problem is more important and easier to track.

[kai771]

@dracodoc
I downloaded the epub you attached, but can't reproduce your results on my K3 and Librerator R1. It took about 55-57 sec on first opening, if cr3cache was empty. If I close the file and open it immediately after, it takes around 35 seconds. If I close Librerator, and open it again, opening that epub takes a bit less than 50 sec. I measured all this a few times, and got consistent results. Default LBR settings, that is page mode, header on. Not sure what to think of this all.

I'm thinking that maybe hyphenation might be one of the reasons why opening is slower (no evidence, just an idea).
Although the file is not that big in bytes, it is 1080 pages. Quite a lot imno. I do hope houpq will be able to "easily track" the cache problem :). I wouldn't even know where to begin :).

[dracodoc]

OK, I tested with stopwatch again.
In Kindel DXG, newest nightly build
1.open with cache exists in first time of same kpv session: 64 seconds.
2. close document and open it again in same kpv session: 4 seconds. Your result for cache have something wrong, I always can open cached epub in less than 10 seconds in same session, from the previous version till this version.
3. I deleted the cache, open it again: 90 seconds.

So maybe the cache is working, but why the same session opening is much faster? something still in RAM?

I don't know why your k3 need 35 seconds to open it.

Another difference in my settings:
I used a slightly different epub.css, changed only this line
text-indent: 40px;
And I'm using PMN Caecilia LT as crengine default font. These are all the customization related to epub I did for a new kpv install.

[kai771]

@dracodoc
I tried last official release of KPV (2012.11). Same file, on first opening - 39 sec!
If closed, then opened again immediatelly - less than 2 sec!
If KPV is closed, then started again, and then immediatelly opened your epub - 35 sec.

Now, this is the good part:
If KPV is closed, then started again, and we open some other epub (not the one you attached) and then close it, and then open your epub - again less than 2 sec!

My previous measurments were done with Librerator - although I thought I didn't change anything about the cashing and stuff, I might've accidentally done something. Will be interesting to find out what, but will have to wait at least until tomorrow.

One other thing, might be helpful: in Cool Reader 3, if we open your epub for the first time, it takes around 30 sec. If we close it and start Cool Reader 3 again (it opens the last file by default), it again takes some time to show the page (I didn't exactly time this, but let's say it's about 25 sec). After Cool Reader is up and running, switching between epubs is much faster. Maybe there's some initialization for crengine that we're failing to do, that Cool Reader 3 does on startup? Just a thought.

Oh, and emulator exhibits similar behaviour - first opening of epub, on freshly started Emu session, is noticeably slower than subsequent ones.

[dracodoc]

Yes I forgot the part of open other epub can "warm up" crengine and make the opening big epub much quicker, I found this behavior before and thought maybe crengine need some time to be loaded into RAM, since at that time the opening time was always less than 1 min, I didn't think there is a problem.

[houqp]

Aha, I found out the culprit ;p It turns out that most of the time is wasted on loading fonts. Currently, we open a document without setting default font, so crengine will choose one for us, most of the time, it's not what we want. Then latter we read the document config and update the font. Now we load font two times and the first time is useless.

Crengine warm up is also related to font loading. Every time we exit KPV and open again, all the fonts have to be reloaded again.

Here is my test result with the new patch:

• clear cache file
• enter KPV
• first open: 26s (no cache and no crengine warm up)
• exit document and open again: 2s (with cache and crengine warm up)
• exit KPV
• enter KPV
• open with document: 3s (with cache, but no warm up)

We can make it a little bit more faster because I see from the log that crengine is still choosing font for us in one or two places. But I think this is good enough?

Here is the download link for the patched KPV I am using (it also includes all the other bug fixes):
https://bitbucket.org/houqp/kindlepdfviewer/downloads/kindlepdfviewer-cache-fix.zip

[dracodoc]

@houqp , this is great news! Now the performance is very good! I'll try the patched version now.

I tried it, it took about 1 min (I didn't measue with stopwatch) to open that epub without cache, and about 6 seconds with cache but no warm up. It's very good.

One problem is, when I navigated to some place in page mode, if I switch to scroll mode it will jump to the beginning of the book instead of the same place of page mode. And if I was in scroll mode it will jump to the end of book if I switch to page mode. Though if I switch back to the original mode it will remember the location of that mode. So right now the two mode are using 2 set of histories.

[houqp]

Got it. I will fix that bug tonight.

BTW, why your DXG is a lot slower than mine? Did you have other hack running at the same time?

[dracodoc]

I tested again with stopwatch, it's 27 seconds, so my first estimation is way off :PP

Another question, does crengine have to hyphenate? I found Chinese epub with english text will have bad effect with crengine, for example, English text inside parentheses would break early and left a very short line, probably because it cannot recognize English, Chinese and parentheses properly to hyphenate correctly.

OK, I found that I can modify epub.css to remove hyphenate. The Chinese epub is better now, though there is still some time the English text inside parentheses is broken into 2 lines, maybe I need to adjust justify too, but that could harm English epub display.

I think there should be 2 different css for Chinese and English epub. Maybe I can use the cr3 extension and modify the css, make all Chinese epub to cr3 extension and use that css.

On second thought, a better method will be enable loading customized css within a book. So user can prepare several different css and put them inside data folder, open a book, press a key to load with certain css. This will solve these problems:
1.Chinese epub and English epub need different css.
2.Some epub don't have the beginning of a paragraph indented, so you will need css to indent for them. However some epub do have the indent by space (this is not the best method, though many epub are not build following all the best practices), the previous css will cause double indent.

Another question, I like the page mode display but can we hide the title bar in page mode?Since the kpv convention is always full screen and show status by press menu key, I'd like to have the page mode in full screen and show title bar only when user press menu key or other key.

[houqp]

wow, so many stuffs added to TODO list ;p

I created #710, #711 and #712 to keep track of the feature requests. All of them will be possible. But for the full screen page mode, you might only want to see the native KPV status bar because toggling full screen mode is actually re-typsetting the document and thus is time consuming.

Since we are waiting for the Jan release now, I would prefer to freeze the code for bug fixes only and add these features after the release. It will take a couple of days.

[houqp]

closing for now since patch is merged to master.

[dracodoc]

If the font setting problem will not be solved (since it only affect fonthack user and not many people reported this problem), I'll note this problem in the wiki to tell people who did meet this problem how to avoid it.

It's here
https://github.com/hwhw/kindlepdfviewer/wiki/Usage-Tips
font setting trap for fonthack users

[houqp]

Thanks for the tip :)