Refactor handling of non-SOH generations #1688

VSadov · 2020-01-13T20:01:18Z

Soon LOH will be not the only non-SOH generation. That will break a few existing assumptions and common patterns.

This change makes the code to expect that in addition to LOH, there could be other non-SOH generations without introducing one (yet).

In particular:

we are not using loh suffix for members applicable to non-SOH scenario in general. We now use more generic suffix aoh instead.
we are not using loh_generation unless it is something specific to large object heap.
For example iterating non-SOH generations is done like

for (int i = max_generation + 1; i < total_generation_count; i++)
{
   . . .
}

Inverting a common pattern for going through segments.
We used to go through segments in the outer loop, and switch to LOH upon reaching the end of the list and breaking out when reaching the end once more. With more non-SOH generations this pattern is not very convenient.
Now we iterate generations in the outer loop and segments in the inner.
generally avoid math with generation indices or conditions that assume that: LOH is the last generation, the only non-SOH generation, that non-SOH generation has #3 index, etc...

VSadov · 2020-01-13T20:06:18Z

I am no longer able to add commits to the old PR.

After the runtime repo went public, my clone lost the upstream relationship (this happens on GitHub when parent fork changes visibility). The PR form the old clone was working for a while, but not any more. It looks like the time to make a new PR.

This is continuation of #235
#WontFix

jkotas · 2020-01-14T14:21:54Z

Meant to close the other one #WontFix

Maoni0

here are the comments so far from where I stopped on #235. more to go; will pick up tomorrow.

Maoni0 · 2020-01-16T06:46:53Z

src/coreclr/src/gc/gc.cpp

-    size_t eph_gen_starts = (Align (min_obj_size)) * (max_generation + 1);
+
+    // See comment in can_expand_into_p why we need this size.
+    size_t eph_gen_starts = gc_heap::eph_gen_starts_size + Align (min_obj_size);


nit - no need for gc_heap:: here #Resolved

Maoni0 · 2020-01-16T07:42:59Z

src/coreclr/src/gc/gc.cpp

@@ -20632,9 +20576,8 @@ size_t gc_heap::get_total_committed_size()
    return total_committed;
 }

-size_t gc_heap::committed_size (bool loh_p, size_t* allocated)
+size_t gc_heap::committed_size (int gen_number, size_t* allocated)


you are changing the meaning of this method which appears to return the committed size for a given generation but this does not work for gen0/1. if you want to make gen0/1 work you'd have to start counting from their gen start. but since we currently just use this for loh (I meant this to be for either the total soh or loh thus it took loh_p as an arg), I would suggest to just rename this to uoh_committed_size. #Resolved

Maoni0 · 2020-01-16T07:48:40Z

src/coreclr/src/gc/gc.cpp

-                        loh_committed, loh_allocated,
-                        dd_fragmentation (dynamic_data_of (max_generation + 1)),
+                    size_t uoh_allocated = 0;
+                    size_t uoh_committed = committed_size (gen_number, &uoh_allocated);


this committed_size only gets the committed size for the given generation though so this should stay loh_committed, not uoh_committed (which would mean for all uoh generations). #Resolved

this is not necessarily loh generation either. I will just call it allocated and committed. They are just numbers reported by the dprintf

In reply to: 367271669 [](ancestors = 367271669)

sure they are not necessarily loh gen, but the point is for any uoh gen you will be returning the correct # for committed/allocated and that's not the case for gen0/1. #Resolved

It is now:

else // not a SOH generation { uint32_t memory_load = 0; uint64_t available_physical = 0; get_memory_info (&memory_load, &available_physical); #ifdef TRACE_GC if (heap_hard_limit) { size_t allocated = 0; size_t committed = uoh_committed_size (gen_number, &allocated); dprintf (1, ("GC#%Id h%d, GMI: UOH budget, UOH commit %Id (obj %Id, frag %Id), total commit: %Id (recorded: %Id)", (size_t)settings.gc_index, heap_number, committed, allocated, dd_fragmentation (dynamic_data_of (gen_number)), get_total_committed_size(), (current_total_committed - current_total_committed_bookkeeping))); } #endif //TRACE_GC . . .

what should it be?

In reply to: 368124892 [](ancestors = 368124892)

not sure what you mean? you said "this is not necessarily loh generation either.", and I said that's true but it will return correct info for other uoh gen anyway. the updated code looks fine to me. #Resolved

I just meant the loh should not be a part of the name either.

In reply to: 368133805 [](ancestors = 368133805)

Maoni0 · 2020-01-16T07:51:30Z

src/coreclr/src/gc/gc.cpp

-        gen_data->free_list_space_after = generation_free_list_space (large_object_generation);
-        gen_data->free_obj_space_after = generation_free_obj_space (large_object_generation);
-        gen_data->npinned_surv = out;
+        for (int i = gen_number + 1; i < total_generation_count; i++)


nit
for (int i = (gen_number + 1); i < total_generation_count; i++) #Resolved

Maoni0 · 2020-01-16T07:58:07Z

src/coreclr/src/gc/gc.cpp

+        for (int i = max_generation; i < total_generation_count; i++)
        {
-            committed_mem += heap_segment_committed (seg) - heap_segment_mem (seg);
-            seg = heap_segment_next (seg);
-        }
-        seg = generation_start_segment (generation_of (max_generation + 1));
-        while (seg)
-        {
-            committed_mem += heap_segment_committed (seg) - heap_segment_mem (seg);
-            seg = heap_segment_next (seg);
+            heap_segment* seg = generation_start_segment (generation_of (i));
+            while (seg)
+            {
+                committed_mem += heap_segment_committed (seg) - heap_segment_mem (seg);
+                seg = heap_segment_next (seg);
+            }
        }


this could actually just be replaced by a simple committed_size() call... my bad that I didn't notice this before #Resolved

Maoni0 · 2020-01-16T08:09:28Z

src/coreclr/src/gc/gc.cpp

@@ -33940,7 +33837,7 @@ void gc_heap::process_n_background_segments (heap_segment* seg,
                                             heap_segment* prev_seg,
                                             generation* gen)
 {
-    assert (gen != large_object_generation);
+    assert (gen->gen_num <= max_generation);


I'm ok with just deleting the process_n_background_segments method as it's unused #Resolved

Maoni0

here are the comments on the rest of the gc.cpp changes... tomorrow I will look at the other files.

Maoni0 · 2020-01-17T01:24:17Z

src/coreclr/src/gc/gc.cpp

+            next_sweep_obj = o;
+
+            allow_fgc();
+            uint8_t* end = background_next_end (seg, i > max_generation);


nit
uint8_t* end = background_next_end (seg, (i > max_generation)); #Resolved

Maoni0 · 2020-01-17T01:29:27Z

src/coreclr/src/gc/gc.cpp

+                            allow_fgc();
+                            current_num_objs = 0;
+                        }
+                    } while (o < end && background_object_marked(o, TRUE));


nit
} while ((o < end) && background_object_marked(o, TRUE)); #Resolved

Maoni0 · 2020-01-17T01:35:42Z

src/coreclr/src/gc/gc.cpp

+                        current_num_objs = 0;
+                    }
+                }
+            } // while (o < end)


is the comment here some accidental leftover from some previous changes? #Resolved

Refactoring these loops required some extra care. I think I have added a comment to the closing brace to not lose track what it matches with while refactoring and forgot to remove.
It is no longer needed.

In reply to: 367733008 [](ancestors = 367733008)

Maoni0 · 2020-01-17T01:39:21Z

src/coreclr/src/gc/gc.cpp

-                    current_num_objs = 0;
-                }
+            enter_spin_lock (&more_space_lock_uoh);
+            add_saved_spinlock_info (true, me_acquire, mt_bgc_loh_sweep);


we should rename mt_bgc_loh_sweep to mt_bgc_uoh_sweep then. #Resolved

Maoni0 · 2020-01-17T01:39:43Z

src/coreclr/src/gc/gc.cpp

-                {
-                    break;
-                }
+            concurrent_print_time_delta ("Swe LOH took msl");


nit -
concurrent_print_time_delta ("Swe UOH took msl"); #Resolved

Maoni0 · 2020-01-17T07:04:45Z

src/coreclr/src/gc/gc.cpp

@@ -34754,13 +34606,14 @@ void gc_heap::relocate_in_large_objects ()
    }
 }

-void gc_heap::mark_through_cards_for_large_objects (card_fn fn,
+void gc_heap::mark_through_cards_for_uoh_objects (card_fn fn,
+                                                    int gen_num,


nit - spaces need to be adjusted here. the coding convention says that the parameters have the same indentation. #Resolved

Maoni0 · 2020-01-17T07:15:21Z

src/coreclr/src/gc/gc.cpp

@@ -35843,7 +35645,7 @@ gc_heap::verify_free_lists ()
                                 (size_t)free_list));
                    FATAL_GC_ERROR();
                }
-                if ((gen_num != max_generation+1)&&(object_gennum (free_list)!= gen_num))
+                if ((gen_num <= max_generation)&&(object_gennum (free_list)!= gen_num))


nit - would be good to take the opportunity to fix the spacing here -

if ((gen_num <= max_generation) && (object_gennum (free_list)!= gen_num)) #Resolved

Maoni0 · 2020-01-17T07:16:53Z

src/coreclr/src/gc/gc.cpp

@@ -35884,7 +35686,9 @@ gc_heap::verify_heap (BOOL begin_gc_p)
    BOOL            large_brick_p = TRUE;
    size_t          curr_brick = 0;
    size_t          prev_brick = (size_t)-1;
-    int             curr_gen_num = max_generation+1;
+
+    // go through all generations starting with loh


would be better to start with (total_generation_count - 1) here - inevitably we'd want to verify all generations. #Resolved

Maoni0 · 2020-01-17T07:38:45Z

src/coreclr/src/gc/gc.cpp

-                                     0);
-
-                if (should_collect_optimized (dd1, low_memory_p))
+                if (!should_collect && should_check_uoh)


you still need to call should_collect_optimized on max_generation always; you just also need to check uoh if should_check_uoh is true (and should_collect_optimized on max_generation is false).

also if it's changed to uoh it should really check all uoh generations #Resolved

Maoni0 · 2020-01-17T07:41:51Z

src/coreclr/src/gc/gc.cpp

+            }
+
+            //discount the fragmentation
+            generation* gen2 = pGenGCHeap->generation_of (i);


nit - uoh_gen seems like a better name than gen2 (I'm aware that it says seg2 above but gen2 means something specific) #Resolved

VSadov · 2020-01-17T21:19:56Z

        concurrent_print_time_delta (concurrent_p ? "Cov SOH" : "Nov SOH");

this now wouldn't be correct since this is executed when soh is not true. strictly speaking this wasn't exactly working before in the dependent handle scenario but at least when there isn't, this works by design (when we added DH it didn't use concurrent_print_time_delta). you could put this line under the if (small_object_segments) check before so at least it isn't worse than before. #Resolved

Refers to: src/coreclr/src/gc/gc.cpp:20318 in 2d3c483. [](commit_id = 2d3c483, deletion_comment = False)

VSadov · 2020-01-17T22:03:00Z

uint32_t pct = (gen_num != max_generation) ? fgn_loh_percent : fgn_maxgen_percent;

same here - since this is checking for allocations would be a bit better to do
(gen_num >= uoh_start_generation) #Resolved

Refers to: src/coreclr/src/gc/gc.cpp:12345 in 2d3c483. [](commit_id = 2d3c483, deletion_comment = False)

VSadov · 2020-01-17T22:03:57Z

assert (new_alloc == (ptrdiff_t)Align (new_alloc, get_alignment_constant (gen_number <= max_generation)));

seems better to replace places that say (max_generation + 1) with uoh_start_generation in the allocation path since UOH specially says it's user allocated #Resolved

Refers to: src/coreclr/src/gc/gc.cpp:12231 in 2d3c483. [](commit_id = 2d3c483, deletion_comment = False)

VSadov · 2020-01-17T22:05:49Z

                (i < loh_generation) ? old_brick_table : NULL,

you could do
((i < uoh_start_generation) ? old_brick_table : NULL),
which is saying we do not use brick table for UOH gens.
you could also imagine adding yet another enum that says "generations that use the brick table" but that's not that necessary for now.
#Resolved

Refers to: src/coreclr/src/gc/gc.cpp:8363 in 2d3c483. [](commit_id = 2d3c483, deletion_comment = False)

VSadov · 2020-01-17T22:07:07Z

    generation_alloc_context (large_object_generation);

i #Resolved

Refers to: src/coreclr/src/gc/gc.cpp:6434 in 2d3c483. [](commit_id = 2d3c483, deletion_comment = False)

VSadov · 2020-01-17T22:09:12Z

size_t default_seg_size = min_uoh_segment_size;

this shouldn't be min_uoh_segment_size because POH has much smaller seg size. we can have a method that returns the min seg size for each generation. #Resolved

Refers to: src/coreclr/src/gc/gc.cpp:13450 in 2d3c483. [](commit_id = 2d3c483, deletion_comment = False)

VSadov · 2020-01-17T22:13:08Z

size_t default_seg_size = min_uoh_segment_size;

I think it will be easier to address once POH is introduced. We may have to tune not just the segment size. ( https://github.com/dotnet/coreclr/issues/27696 )

In reply to: 575814638 [](ancestors = 575814638)

Refers to: src/coreclr/src/gc/gc.cpp:13450 in 2d3c483. [](commit_id = 2d3c483, deletion_comment = False)

VSadov · 2020-01-17T22:56:34Z

I have looked through logs for the previous PR and moved unaddressed comments here.
I think all the comments up to this moment are addressed. #Resolved

Maoni0

ok I've gone through all changes for this PR

Maoni0 · 2020-01-18T00:39:37Z

src/coreclr/src/debug/daccess/request.cpp

@@ -701,7 +701,7 @@ ClrDataAccess::GetThreadAllocData(CLRDATA_ADDRESS addr, struct DacpAllocData *da
    Thread* thread = PTR_Thread(TO_TADDR(addr));

    data->allocBytes = TO_CDADDR(thread->m_alloc_context.alloc_bytes);
-    data->allocBytesLoh = TO_CDADDR(thread->m_alloc_context.alloc_bytes_loh);
+    data->allocBytesLoh = TO_CDADDR(thread->m_alloc_context.alloc_bytes_uoh);


do we want to rename this allocBytesLoh field? this GetThreadAllocData is only used by sos I think (not by clrmd?)? @leculver @mikem8361 could you please confirm? for context we are renaming the alloc_bytes_loh field on the alloc_context to alloc_bytes_uoh because we are introducing a concept of uoh which includes loh and additional generations. #Resolved

It looks like SOS doesn't use this API. Not sure about CLRMD. #Resolved

I am suggesting changing this only when clients can use it. No benefit from changing this now, for sure, since there is still only loh in this change, but we may break something.

In reply to: 368189692 [](ancestors = 368189692)

We already have a tracking bug to revisit this - https://github.com/dotnet/coreclr/issues/27692

In reply to: 368208096 [](ancestors = 368208096,368189692)

For ClrMD we reimplement all of this interface in C#. Changing names won't affect us at all, but we require these structures do not change in layout or meaning for backwards compat.

When real changes go in that affect the underlying shape of the GC heap and/or requires changing the shape or usage of DAC data structures it's best to implement a new interface (ISOSDacX or similar) that we can query for to tell us there's a new style of heap involved.

Maoni0 · 2020-01-18T00:41:03Z

src/coreclr/src/gc/gcee.cpp

        allocation_3 +=
-            dd_desired_allocation (hp->dynamic_data_of (max_generation+1))-
-            dd_new_allocation (hp->dynamic_data_of (max_generation+1));
+            dd_desired_allocation (hp->dynamic_data_of (loh_generation))-


seems odd that you have different code for WKS and SVR.. #Resolved

odd. Also it seems that allocation_0 and allocation_3 are computed, but not used. ..

In reply to: 368189884 [](ancestors = 368189884)

I will just remove this code as unused.

In reply to: 368208845 [](ancestors = 368208845,368189884)

Also this seems to be the last actionable comment on this PR that I am aware of.

In reply to: 368247232 [](ancestors = 368247232,368208845,368189884)

ah, this was because the code for perf counters was removed and allocation_0/3 was used by perf counters but was mistakenly not removed.

Maoni0 · 2020-01-18T00:43:18Z

src/coreclr/src/gc/gcinterface.dac.h

@@ -17,6 +17,8 @@
 #define MAX_EXPAND_MECHANISMS_COUNT     6
 #define MAX_GC_MECHANISM_BITS_COUNT     2
 #define MAX_GLOBAL_GC_MECHANISMS_COUNT  6
+
+// for backward compatibility the number of generations for DAC must stay 4


I don't think this is the case - you can make it work, it's just work that we need to do. it would be very odd if we just forever only process 4 generations. please talk to the diagnostics folks about this. #Resolved

It looks like the SOS DAC api (in dacprivate.h starting at line 712) assume that there are only 4 generations. The DacpGcHeapDetails request and DacpGenerationAllocData assume 4 generations. SOS and probably CLRMD use this API so we need to come up with a backward compatible way to support this. Like a new Request (DacpGcHeapDetails2 ?) that the 5.0 DAC supports. #Resolved

Not that this is impossible, but it is not something to be changed lightly. It is a breaking change.
It is also not clear whether if it is better to introduce a new generation for POH or other generations that we may have after, or just fold everything UOH together.
There is a work item for this - https://github.com/dotnet/coreclr/issues/27693

In reply to: 368190184 [](ancestors = 368190184)

I certainly wasn't suggesting changing this lightly. I was suggesting we should not have an incorrect comment. we could say that it will require work to change this. feel free to open an issue for changing it in the future.

I will remove the comment. It does not make a lot of sense in this change, and related work is tracked in a workitem.

Maoni0 · 2020-01-21T20:32:13Z

if you have not run stress and perf pass on this change I certainly would suggest to do so (I would guess you have not run perf or it should have shown regression due to the changes in BGC sweep). most of the changes are very benign but at this scale we could be making accidental serious mistakes.

VSadov · 2020-01-21T20:34:50Z

Yes, I will definitely run stress/perf once there are no more changes. There could be bugs.

leculver · 2020-01-21T20:35:53Z

Just catching up on this. I read through the changes but I'm not clear on whether this change would require updates to SOS and ClrMD for heap enumeration. Obviously changes will need to happen once this is true:

Soon LOH will be not the only non-SOH generation

But does this particular refactor change the layout of the GC heap at all?

Maoni0 · 2020-01-21T20:36:11Z

great, thanks @VSadov!

VSadov · 2020-01-21T20:36:14Z

I did run stress a while ago, but since then there were many changes, so that does not count.

VSadov · 2020-01-21T20:43:42Z

@leculver - this is just a refactoring change. No new functionality. The questions about SOS/DAC/ClrMD are mostly forward-looking.

leculver · 2020-01-21T20:44:20Z

Thanks, please keep me cc'ed on future changes of this nature!

Maoni0 · 2020-01-24T23:53:45Z

src/coreclr/src/gc/gc.cpp

@@ -34117,7 +34120,7 @@ void gc_heap::background_sweep()

        if (i == max_generation)
        {
-            start_seg = ephemeral_heap_segment;
+            start_seg = eph_seg;


this is already saved as saved_sweep_ephemeral_seg in background_ephemeral_sweep

I did not notice that we are already saving that. Will use that one then

VSadov · 2020-01-28T23:52:29Z

src/coreclr/src/gc/gc.cpp

@@ -3170,7 +3167,7 @@ void gc_heap::fire_per_heap_hist_event (gc_history_per_heap* current_gc_data_per
               current_gc_data_per_heap->mechanisms[gc_heap_expand],
               current_gc_data_per_heap->heap_index,
               (void *)(current_gc_data_per_heap->extra_gen0_committed),
-               (max_generation + 2),
+               total_generation_count - 1,


@Maoni0 - looks like it is this change that interferes with event format. I did -1 to report the same number of heaps when POH was added and did not readjust this back.

By looking at how version of this events is calculated/validated, this should be just total_generation_count. PerfView seems to permit extra generations here - for future versions. Going below 4 is unexpected though.

ahh, okie. can you please rerun the perf test and make sure you are now seeing the correct data?
it's strange why the 0gb is the only scenario that's exhibiting the behavior. and also it seems like the LOH data should be the one that gets messed up.

PR feedback

more PR feedback

VSadov · 2020-01-29T01:55:26Z

fixed GCPerHeapHistory_V3 event.

VSadov · 2020-01-29T02:04:55Z

also rebased due to merge conflicts

VSadov · 2020-01-29T14:18:09Z

Thanks!!

Refactor handling of non-SOH generations (dotnet#1688)

…ns (#1688) this is supposed to get the user allocations, before #1688 it was getting gen0 and gen3. with the refactor it should get gen0 and all UOH allocations.

A recent refactoring PR dotnet#1688 has regressed GC processing of mark overflow. If GC heap segments in the segment linked list are not ordered in an ascending order by their addresses, the mark overflow processing misses segments on lower addresses if they follow segments on higher addresses. This leads to some objects that are alive to not to be reported and freed. This change fixes the problem by making sure the segment order doesn't matter.

A recent refactoring PR #1688 has regressed GC processing of mark overflow. If GC heap segments in the segment linked list are not ordered in an ascending order by their addresses, the mark overflow processing misses segments on lower addresses if they follow segments on higher addresses. This leads to some objects that are alive to not to be reported and freed. This change fixes the problem by making sure the segment order doesn't matter.

VSadov mentioned this pull request Jan 13, 2020

Refactor handling of non-SOH generations #235

Closed

VSadov requested a review from Maoni0 January 13, 2020 20:08

jkotas added the area-GC-coreclr label Jan 14, 2020

jkotas closed this Jan 14, 2020

jkotas reopened this Jan 14, 2020

Maoni0 requested changes Jan 16, 2020

View reviewed changes

Maoni0 requested changes Jan 17, 2020

View reviewed changes

VSadov force-pushed the genRefactor branch from 2d3c483 to 305c74d Compare January 17, 2020 22:31

Maoni0 requested changes Jan 18, 2020

View reviewed changes

VSadov force-pushed the genRefactor branch 3 times, most recently from 7994fe0 to bc07b39 Compare January 24, 2020 23:38

Maoni0 reviewed Jan 24, 2020

View reviewed changes

VSadov commented Jan 28, 2020

View reviewed changes

VSadov added 11 commits January 28, 2020 17:16

refactoring

22bbcda

ploh-->uoh

71c7219

fix Linux build

d6761f0

PR feedback

some PR feedback

326741b

more PR feedback

7a5bf91

more PR feedback

more PR feedback

b488709

address comments in UpdatePreGCCounters

e8396d7

removed a confusing comment

1acd514

stress fix in background sweep

f421d32

use saved_sweep_ephemeral_seg in background sweep.

536f3d1

fixed GCPerHeapHistory_V3 event.

59510de

VSadov force-pushed the genRefactor branch from 278e464 to 536f3d1 Compare January 29, 2020 01:35

re-implemented dotnet#2103 on top of refactoring (it was hard to merge)

1dba378

VSadov force-pushed the genRefactor branch from f052511 to 1dba378 Compare January 29, 2020 02:46

Maoni0 approved these changes Jan 29, 2020

View reviewed changes

VSadov merged commit 1336238 into dotnet:master Jan 29, 2020

VSadov deleted the genRefactor branch January 29, 2020 14:18

Maoni0 added a commit to Maoni0/runtime that referenced this pull request May 2, 2020

fix a regression introduced by 1336238

d3d22e9

Refactor handling of non-SOH generations (dotnet#1688)

Maoni0 mentioned this pull request May 2, 2020

fix a regression in gc_heap::get_current_allocated #35744

Merged

janvorli mentioned this pull request Jul 9, 2020

Fix mark overflow processing #38969

Merged

ghost locked as resolved and limited conversation to collaborators Dec 11, 2020

Refactor handling of non-SOH generations #1688

Refactor handling of non-SOH generations #1688

Conversation

VSadov commented Jan 13, 2020 • edited Loading

VSadov commented Jan 13, 2020 • edited Loading

jkotas commented Jan 14, 2020 • edited by VSadov Loading

Maoni0 left a comment

Choose a reason for hiding this comment

Maoni0 Jan 16, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 16, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 16, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maoni0 Jan 16, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 16, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 16, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 left a comment

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Maoni0 Jan 17, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

VSadov commented Jan 17, 2020 • edited Loading

VSadov commented Jan 17, 2020 • edited Loading

VSadov commented Jan 17, 2020 • edited Loading

VSadov commented Jan 17, 2020 • edited Loading

VSadov commented Jan 17, 2020 • edited Loading

VSadov commented Jan 17, 2020 • edited Loading

VSadov commented Jan 17, 2020

VSadov commented Jan 17, 2020 • edited Loading

Maoni0 left a comment

Choose a reason for hiding this comment

Maoni0 Jan 18, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

mikem8361 Jan 18, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VSadov Jan 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maoni0 Jan 18, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maoni0 Jan 18, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

mikem8361 Jan 18, 2020 • edited by VSadov Loading

Choose a reason for hiding this comment

VSadov Jan 18, 2020 • edited Loading

Choose a reason for hiding this comment

VSadov commented Jan 13, 2020 •

edited

Loading

VSadov commented Jan 13, 2020 •

edited

Loading

jkotas commented Jan 14, 2020 •

edited by VSadov

Loading

Maoni0 Jan 16, 2020 •

edited by VSadov

Loading

Maoni0 Jan 16, 2020 •

edited by VSadov

Loading

Maoni0 Jan 16, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 16, 2020 •

edited by VSadov

Loading

Maoni0 Jan 16, 2020 •

edited by VSadov

Loading

Maoni0 Jan 16, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

Maoni0 Jan 17, 2020 •

edited by VSadov

Loading

VSadov commented Jan 17, 2020 •

edited

Loading

VSadov commented Jan 17, 2020 •

edited

Loading

VSadov commented Jan 17, 2020 •

edited

Loading

VSadov commented Jan 17, 2020 •

edited

Loading

VSadov commented Jan 17, 2020 •

edited

Loading

VSadov commented Jan 17, 2020 •

edited

Loading

VSadov commented Jan 17, 2020 •

edited

Loading

Maoni0 Jan 18, 2020 •

edited by VSadov

Loading

mikem8361 Jan 18, 2020 •

edited by VSadov

Loading

VSadov Jan 18, 2020 •

edited

Loading

Maoni0 Jan 18, 2020 •

edited by VSadov

Loading

Maoni0 Jan 18, 2020 •

edited by VSadov

Loading

mikem8361 Jan 18, 2020 •

edited by VSadov

Loading

VSadov Jan 18, 2020 •

edited

Loading