packages: add general io utils #17
Conversation
Vicente-Cheng
force-pushed
the
add-io-util
branch
3 times, most recently
from
97e77c6 to
edc9fd7
Compare
Vicente-Cheng
force-pushed
the
add-io-util
branch
from
edc9fd7 to
a379f38
Compare
Vicente-Cheng
force-pushed
the
add-io-util
branch
2 times, most recently
from
0bcf9b1 to
2598c88
Compare
io/io.go
Outdated
| _, err := PReadExact(src, buf, chunkSize, uint64(i*chunkSize)) | ||
| if err != nil { | ||
| fmt.Printf("Error reading data: %v\n", err) | ||
| ioError = err | ||
| continue | ||
| } | ||
| objs <- Content{offset: uint64(i * chunkSize), buf: buf} |
There was a problem hiding this comment.
Maybe we should have continuous direct reads in a single go routine?
There was a problem hiding this comment.
Could you elaborate more on your concerns about the single go routine (single producer) instead of the multi-producer?
There was a problem hiding this comment.
I don't mean to only have a single producer. IIUC, We will have 8 (from producerNum) producers, and in each producer, the read address is not continuous. For example, the first producer will read the address of chunk numbers 0, 8, 16, 24 ... so on. It could be just a nit for non-rotational devices. I'm ok with the current design, just a discussion. WDYT?
There was a problem hiding this comment.
Sounds good to me. Now, each producer will handle the continuous offset to read.
Vicente-Cheng
force-pushed
the
add-io-util
branch
2 times, most recently
from
df1fa6b to
79007d3
Compare
Thanks @WebberHuang1118. So, I dropped the dynamic alignment size when calling posix_memalign, which would also simplify the code. |
Vicente-Cheng
force-pushed
the
add-io-util
branch
from
79007d3 to
2d3c9ae
Compare
Vicente-Cheng
force-pushed
the
add-io-util
branch
3 times, most recently
from
22df88a to
47189b1
Compare
Vicente-Cheng
force-pushed
the
add-io-util
branch
from
47189b1 to
e92c23d
Compare
- suppport parallel ioReader/ioWriter
- skip the zero buffer
Signed-off-by: Vicente Cheng <[email protected]>
Vicente-Cheng
force-pushed
the
add-io-util
branch
from
e92c23d to
1982bab
Compare
It will be used for general volume clones.