Added audio cache for wav tracks

[hamsterksu]

hi @hrosa i have found issue
stream = cache.putIfAbsent(uri, new AudioStreamCache(uri));

putIfAbsent returns null if no such mapping existed http://www.ehcache.org/apidocs/3.1.1/org/ehcache/Cache.html#putIfAbsent-K-V-

unfortunately i removed prev fork so i have created new, applied pull request as patch and made fix

[hrosa]

@hamsterksu thank you for the patch, I will take some time to load test this branch. If everything is OK I will merge it.

In the meantime, can you please update the MS docs to include information about the caching mechanism?

https://github.com/RestComm/mediaserver/blob/master/docs/sources-asciidoc/src/main/asciidoc/concept-chapter-Configuring_the_Media_Server.adoc#resources-configuration

Thank you!

[jaimecasero]

It would be nice to have an option for disk caches. Just using the builder with different option http://www.ehcache.org/apidocs/3.1.0/org/ehcache/config/builders/ResourcePoolsBuilder.html#disk-long-org.ehcache.config.units.MemoryUnit-boolean-

It would require a new flag coming form configuration

[hrosa]

I also would like to offer more flexibility when setting the cache size.
The configuration file should allow users to setup sizes in String format like 10b, 10k, 10m,10g.

[hamsterksu]

@hrosa currently we support Mb only
as for me it's enough, otherwise cache doesn't make sense
if you need Gb you can setup it in Mb

[hamsterksu]

@jaimecasero we did plan to add disk cache at all.
because restcomm-connect has own disk cache

[hrosa]

Hi @hamsterksu

I've been load testing an application like this:

<?xml version="1.0" encoding="UTF-8"?>
<%! int counter = 0; %>
<%! int size = 338; %>
<% counter++; %>
<Response>
<Say>Hello. This is announcement number <%= counter%size %></Say>
</Response>

The application will spawn 338 wav tracks with ~60Kb each. In total, we have ~25Mb.

With a 100Mb cache, capable of containing all data we have the following results:

This is pretty good, as it shows only network activity when the cache loads files for first time.
Without cache, the network activity spikes for the entire duration of the test, as shown below:

Then I wanted to test overhead of cache eviction by reducing the size of cache in such way that it could not contain all 338 wav tracks. I tried reducing the cache size to 1Mb but no luck.
Based on profiler, seems 1Mb cache can hold all wav tracks because we only have network activity first time files are read.

Can you provide some pointer on how to estimate cache size?

I've tried hacking the code using SizeOf to measure size of each stream. SizeOf claims 24 bytes for each one.
Maybe understanding SizeOf is key to understand cache sizing? Can you investigate this?

The only way I had to test cache eviction was to hardcode cache size to a few Kb.
Here are the results:

Still better than no caching, and CPU overhead is small. Still, we're talking about a small cache.

Would you be able to test impact of CPU, Memory and overall performance for an evicting cache that is +100Mb?

Great job so far!

[hamsterksu]

@hrosa

i don't know how to estimate cache size. As for me you just need to calculate count of unique audio files for some time period. in most cases you know a size of each file. so it will be required cache size

WDYT?

[hrosa]

@hamsterksu I thought so myself, but I got contradictory results as explained.
I fed 25Mb (60Kb * 338) of files to a 1Mb cache and it seemed the cache was able to store them all. Profiler only showed active sockets for the first time the files were accessed.

Can you dig a bit into this and check if you obtain similar results?
Apparently ehcache uses SizeOf to calculate available size, so there must be a catch there.

[hrosa]

any news on this @hamsterksu ?

[hamsterksu]

hi @hrosa sorry for delay
i think i have found issue: we put lazy object to cache
so i think cache manager calculates object size when we add it and doesn't recalculate it when add new object.
i'm working on fix

[hrosa]

ah understood @hamsterksu
OK, let me know once you have news.

[hamsterksu]

@hrosa i have rebased your master to my branch and pushed it
so it contains the following fixes:

1. remove uri class member to save some memory
2. add workaround for lazy cache objects - re-put them to cache when data is downloaded
3. disable cache by default

i have uploaded profiler snapshots to dropbox

1. w/o cache - https://www.dropbox.com/s/2k5srbwsotrkid3/without_cache.snapshot?dl=0
2. with 1m cache - https://www.dropbox.com/s/sp90khokd9i1a6v/with_1m_cache.snapshot?dl=0

in my test case i used the following app:

    bucket := count / 100
    position := count % 20
    fmt.Fprint(w, fmt.Sprintf(
        "<Response><Say>%s %d%d</Say><Hangup/></Response>",
        "Hello. This is announcement number", bucket, position))

and sipp test with with following settings:

MAXIMUM_CALLS=200 
SIMULTANEOUS_CALLS=20 
CALLS_PER_SEC=10 

[hrosa]

many thanks @hamsterksu
Will do some tests on my side and review.

[hrosa]

Hi @hamsterksu

After doing some tests, I feel like the cache is now always allocating space for existing URI.
Below you can find socket activity for a load test with a 200MB cache and 50Mb of sample files:

Before you worked on latest changes, the cache would only create sockets for the first time a file was read (given there was space in cache), as showed in my examples from #252 (comment) (1st picture).

To back this up, I did manual tests by calling 3 times the +1234 (hello-world) application. Three times the same track was played, three times the cache size changed.
See cache-test.txt for log traces.

You will find 3 occurrences of:

10:15:27,667 DEBUG [CachedRemoteStreamProvider] onSizeChanged for http://localhost:8080/restcomm/cache/ACae6e420f425248d6a26948c17a9e2acf/35ea210b73dcaa203f471c0e30304811a8b89b94a598aa9c44f1c3a5d8a7ce88.wav

Can you look into this?

Thanks

[hamsterksu]

@hrosa i will look on it.
i see message
10:14:42,995 WARN [OnHeapStore] Max Object Graph Size reached for the object : java.nio.DirectByteBuffer[pos=0 lim=8192 cap=8192]
after
10:14:42,982 DEBUG [CachedRemoteStreamProvider] onSizeChanged for http://localhost:8080/restcomm/cache/ACae6e420f425248d6a26948c17a9e2acf/35ea210b73dcaa203f471c0e30304811a8b89b94a598aa9c44f1c3a5d8a7ce88.wav

Seems changed object extrusions it from cache.

[hamsterksu]

@hrosa sorry for a lot of iterations

i have found an issue: lock object was blocked and cache could not calculate size.
also i have added test for cache overflow case

[hrosa]

Hi @hamsterksu

I've been testing your latest work and everything seems OK now. Great job!
Only saw one issue with memory consumption, in cases where cache size cannot contain all data and evictions happen. It seems that we are unable to GC evicted data during minor GC collections, so we will have major collection (stop the world) more frequently.

Here is proof from my tests, where initial cache size is 200Mb. After 28min of testing, we see 1.2Gb of memory consumption (old gen).
Tested with 300 concurrent calls, 30 calls per second. Duration of each call is 1 minute.

It seems to me that we (urgently) need to improve cleanup of evicted payloads:

This issue DOES NOT happen when cache is big enough to store all data. That being said, I'm willing to merge this PR (since we can disable cache or assign it a safe size to ensure evictions won't happen during runtime).
But I will open a ticket for cache eviction optimisation that we need to work on. I would like to ask you if can do some investigation on this matter?

Many thanks for this valuable contribution. Highly appreciated!