Thoughts on In-Memory Cache Extension of the Cache System

[da3dsoul]

I don't know if it's an issue with the way that the SQL/database schema is put together, but even with cache, the bottleneck is still retrieving info.

We generally don't keep much loaded, and we could speed it up immensely by implementing an ORM In-Memory Cache.

[bigretromike]

Aint kodi has built in memory cache solution with `toCache` or something for listitems objects?

…

On March 19, 2019 7:46:58 PM GMT+01:00, da3dsoul ***@***.***> wrote: I don't know if it's an issue with the way that the SQL/database schema is put together, but even with cache, the bottleneck is still retrieving info. We generally don't keep much loaded, and we could speed it up immensely by implementing an ORM In-Memory Cache. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #303

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[da3dsoul]

Not that I'm aware of?

[da3dsoul]

Okay, I tried for a bit. At least for now, I'm marking this as not feasible. It's a shame, as we could make loading almost instantaneous reading it from memory.

[bigretromike]

I have an idea... I will try to do PoC and we will see if its worth the hassle or not.

[da3dsoul]

Go for it. My idea was to use service to persist the memory cache, but it kept getting mysteriously disposed, so I gave up.

[bigretromike]

awww, thats what I was thinking after reading in-memory sqlite documentation 👍 so no point in wasting time then ;p

[bigretromike]

but maybe I will try the PoC maybe I will if while else goto enough it will persist ;-)

[da3dsoul]

It's already got a loop. I couldn't even figure out why it was happening. Let me know if you can get a variable to persist. If you can manage that then I know enough from my attempts to get the rest working.

[da3dsoul]

Specifically, see if you can get a variable to be set from plugin and retrieved from a different menu

[bigretromike]

ok, I will then skip the cache part and focus on 'ANY' variable to persist in in-memory database, and if that possible we going forward

[da3dsoul]

Cool

[da3dsoul]

My next project is implementing the search menu. I may wait for the cast menu, as I want to do so much more with it.

[bigretromike]

what about rest of issues from #301 ?

[da3dsoul]

I'll get to them, but I want it to be "feature complete", or at least to a level that the more important features, before I worry about minor bugs

[bigretromike]

to be honest I was refering only to this "Default sorting Method for Episodes" because without that shit my experience is low so beta for now cant replace my stable build on main htpc ;p so i test it on smaller setup

[da3dsoul]

That was going to be the first one I look at. It depends on how I feel at the time whether I try to fix that or do search first

[bigretromike]

how did you communicate between service and plugin ?

[da3dsoul]

I tried doing it directly. On the attempt to cache it, I set nakamori_service.cache_database = database.connect(':memory:'). My guess is that each addon has a different Python instance, so they can't talk to each other unless you use networking, but then you have serialization latency. If we could use a method that has super fast serialization, then that could work, but idk. There's always the option of doing straight bitstreams, as it's communicating with localhost and compression is unnecessary.

[bigretromike]

so you used import ? to access service objects?

[da3dsoul]

http://romanvm.github.io/script.module.simpleplugin/storage.html
https://github.com/romanvm/script.module.simpleplugin
Look at that.

I did this basically

import nakamori_service
nakamori_service.variable1 = 'cheese'

[bigretromike]

I found your localization shit: http://romanvm.github.io/script.module.simpleplugin/gettext.html
also did you try that memstorage?

[da3dsoul]

That isn't in active development and may not work, but the idea is still useful.

[bigretromike]

I read about that, he store data inside xbmc.windows

self._id = storage_id
        self._window = xbmcgui.Window(window_id)
        try:
            self['__keys__']
        except KeyError:
            self['__keys__'] = []

Thats why they persist

[da3dsoul]

I didn't try anything from that

[bigretromike]

IMO, its more of a hack than solution - that could break anytime :-)

[da3dsoul]

Indeed

[da3dsoul]

https://stackoverflow.com/questions/7839786/efficient-python-to-python-ipc
https://docs.python.org/3/library/multiprocessing.html
http://www.celeryproject.org/

These are all overengineered for our purpose, but we can use fragments and concepts from them.

[bigretromike]

thats too much for me, im doing simpler :-)

[da3dsoul]

Haha okay. It's there just in case

[bigretromike]

ok, its working and persist

[bigretromike]

memory_cache.zip

Its ugly as hell but it works as PoC;
If you want to check it replace 2 files, first send "add" with plugin, then exit plugin, replace add with "get" and you should get "test" that was added on first "add" send.

Just have in mind that its Proof of Concept, not a production code xD

your mistake was that when importing service you import it in plugin process so, when plugin die it dies with it, while service working in background. The service you want to access it the one that kodi itself started and not the one you imported. Thats why you need to communicate with it.

[da3dsoul]

Haha thanks for the warning

[da3dsoul]

We are busy with other stuff, and that zip will likely need to be converted into a patch. That can be done simply, so don't worry about it. The zip was created at 84cf620, so it'll be diff'd from there.

[bigretromike]

in plugin I added 1 line and replace it with another, and the cache is what was important :-)
but making it simple I wrote a socket server in service to comunicate with and exchange data between.
That was one of few ways to make service exchange data because you need to access the cache inside service via service, not via include. When you include it you make another copy of it, that's why data didn't persist

[da3dsoul]

Fair

[bigretromike]

ok, it was a PoC almost half a year ago.
Concept was run service with sqlite in memory and listen to socket/udp, get and set data was via that udp/socket.

It worked as a PoC.

If you ever need it, it will be here - our by that time maybe other solution will be available.