[linux-mm-cc] (no subject)

[John Kielkopf]

Could be. ?That depends on the particular card you use, tho,

since the

algorithm used by the translation layer varies.

The only relevant thing that varies from SD card to SD card is
the size of the erase block.?? Any re-mapping SDs (and
most? SSDs) do are at the erase block level.

Wear
leveling algorithms vary -- but as far as I know,? the majority are
still based around the erase block.? (e.g.: When a group of sectors
written are smaller than the size of an erase? block, the remaining
sectors are read from the erase block, combined with the new data, and
then written to a free block with the lowest amount of writes.)

I'm interested in seeing an SD card that remaps at a finer grain than
its erase block.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.laptop.org/pipermail/linux-mm-cc/attachments/20100831/caf58207/attachment.htm

[Stefan Monnier]

Wear leveling algorithms vary -- but as far as I know,? the majority
are still based around the erase block.? (e.g.: When a group of
sectors written are smaller than the size of an erase? block, the
remaining sectors are read from the erase block, combined with the
new data, and then written to a free block with the lowest amount
of writes.)

AFAIK the SD cards don't support any kind of "tagged command queuing",
so after writing the first sector in a sequence, the card is supposed to
be in a state where the write is completed (even if power goes out at this
particular moment), so I wonder how they handle this while at the same
time being able to wait for the next write to see if it happens to be
to the next sector.


Stefan

[John Kielkopf]

Post by Stefan Monnier
AFAIK the SD cards don't support any kind of "tagged command

queuing",

Post by Stefan Monnier
so after writing the first sector in a sequence,

the card is supposed to

Post by Stefan Monnier
be in a state where the write is

completed (even if power goes out at this

Post by Stefan Monnier
particular moment),

so I wonder how they handle this while at the same

Post by Stefan Monnier
time being

able to wait for the next write to see if it happens to be

Post by Stefan Monnier
to

the next sector.
SD cards support a bulk data
transfer or "multiple block write" operation.? In this
case, it is up to the OS file system driver, device driver and/or SD host
controller to buffer contiguous writes, and then stream the buffer to the
SD card with a multiple block write operation. For the most efficient
writes, you would want your multiple block writes to be aligned with and
the size of the erase block in the underlying NAND -- something that the
application committing data would need to be aware of.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.laptop.org/pipermail/linux-mm-cc/attachments/20100901/e7b6a48c/attachment.htm

[Stefan Monnier]

SD cards support a bulk data transfer or "multiple block write"
operation.? In this case, it is up to the OS file system driver,
device driver and/or SD host controller to buffer contiguous writes,
and then stream the buffer to the SD card with a multiple block write
operation. For the most efficient writes, you would want your
multiple block writes to be aligned with and the size of the erase
block in the underlying NAND -- something that the application
committing data would need to be aware of.

Ah, now that makes sense.
So yes, it would be good to be able to tweak Linux's swap handling so it
tries to write in chunks that are erase-block sized and aligned, as far
as possible. Tho maybe trying to just maximize the chunk size (without
paying particular attention to a particular size or alignment) would be
good enough.


Stefan

[John Kielkopf]

Post by Stefan Monnier
Ah, now that makes sense.
So yes, it would be good to be able to tweak Linux's swap handling so it
tries to write in chunks that are erase-block sized and aligned, as far
as possible. Tho maybe trying to just maximize the chunk size (without
paying particular attention to a particular size or alignment) would be
good enough.

It would seem, rather than starting from scratch, compcache could be a
useful starting point. The compression compcache provides should help
greatly with the the general bandwidth issues of any NAND based device,
reducing the amount of data transferred. And because of compcache's
compression, I'm assuming it already needs to re-map compressed pages -- an
area that could hopefully be tweaked to re-map into neat erase block sized
buffers before finally committing them to a block storage device.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.laptop.org/pipermail/linux-mm-cc/attachments/20100902/1c301c21/attachment.htm

[John Kielkopf]

Post by Stefan Monnier
Ah, now that makes sense.
So yes, it would be good to be able to tweak Linux's swap handling so it
tries to write in chunks that are erase-block sized and aligned, as far
as possible. Tho maybe trying to just maximize the chunk size (without
paying particular attention to a particular size or alignment) would be
good enough.

Rather than starting from scratch, could compcache be a useful starting
point? The compression compcache provides should help greatly with the the
general bandwidth issues of any NAND based device, reducing the amount of
data transferred. I'm assuming compcache already needs to re-map compressed
pages -- an area that could hopefully be tweaked to re-map into neat erase
block sized buffers before finally committing them to a block storage
device.

To recap my thoughts on swapping to NAND:

While we've established that write performance can be increased by paying
close attention to erase blocks, writes should still be avoided as much as
possible. I can think of two ways to reduce the amount of data written:

1) Compress pages -- just like compcache does today, but for different
reasons.

2) Keep a unique hash* of each swapped page in memory. Before swapping out a
page, check if it has been swapped before. If it has, just map to the
already swapped entry**. (I suspect, in memory constrained systems, that
the same data is swapped in and out very often.)

* While MD5, SHA1 and others have been used in storage deduplication
systems, I'm sure there will be a concern of hash collision/silent data
corruption no matter what algorithm is used. A less expensive algorithm
could be used, along with a read verification of the page on systems where
hash collisions must be prevented at all costs. Provided the hash does not
collide often, and knowing that reads are usually much less expensive than
writes, byte for byte verification of pages with duplicate hashes should
still improve performance -- especially if the pages are compressed before
being hashed.

** Pages would need to be left on block storage as long as possible, even
after being swapped back in. Since NAND memory is relativly inexpensive,
using more space as a page cache shouldn't be a problem.

I've searched and have not been able to find a working implementation of
either. The only compressed flash swapping system I found (see:
http://www.celinux.org/elc08_presentations/belyakov_elc2008_compressed_swap_final_doc.pdf)
was discontinued when Numonyx was aquired by Micron, and the source
code
was not and apparently will not be released under GPL. I can find no "swap
de-duplication" projects.

There is room to improve performance in swapping to nand based block
devices, but I'm left wondering why it appears to have little interest. It
seems a natural progression for a project like compcache.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.laptop.org/pipermail/linux-mm-cc/attachments/20100905/6c1f0d0a/attachment.htm