Related
What are the chances of being able to ditch all the concerns that go along with Ext2, Ext3, and Ext4 hacks to improve read-write on the Captivate by going with YAFFS or YAFFS2?
http://en.wikipedia.org/wiki/YAFFS
YAFFS is an optimized filesystem for use with NAND flash memory, would it provide the stability and speed we're looking for?
I know this idea has been discussed lightly before in the various "lag fix" threads, but I was wondering if users with knowledge could enlighten the rest of us on the YAFFS filesystem.
I may be way wrong here, but judging by a few commands I've seen regarding busybox install issues, is it possible the stock kernel supports this already?? In which case could we modify the current lag fixes to use it instead?
Since this is related and I already posted this in Nuka's thread:
http://forum.xda-developers.com/showthread.php?t=750663
Those wacky gents over in the I9000 section have managed to get YAFFS2 compatibility into the kernel. This is something we should probably look into once we've figured out the wake-lag issue (maybe sooner )
In other news, assuming we can get YAFFS2 plugged into our kernel, how feasible (albiet probably complicated) would it be to actually REPLACE RFS and then load the OS on top of YAFFS2? There's just something about cascading file systems written on top of each other that sounds like trouble to me.
Zilch25 said:
Since this is related and I already posted this in Nuka's thread:
http://forum.xda-developers.com/showthread.php?t=750663
Those wacky gents over in the I9000 section have managed to get YAFFS2 compatibility into the kernel. This is something we should probably look into once we've figured out the wake-lag issue (maybe sooner )
Click to expand...
Click to collapse
Nice find!
I agree, the wake-up-lag issue is bothersome. Looking forward to mimocan possibly helping us out with his solution that he offered the Samset people.
kennethpenn said:
What are the chances of being able to ditch all the concerns that go along with Ext2, Ext3, and Ext4 hacks to improve read-write on the Captivate by going with YAFFS or YAFFS2?
http://en.wikipedia.org/wiki/YAFFS
YAFFS is an optimized filesystem for use with NAND flash memory, would it provide the stability and speed we're looking for?
I know this idea has been discussed lightly before in the various "lag fix" threads, but I was wondering if users with knowledge could enlighten the rest of us on the YAFFS filesystem.
Click to expand...
Click to collapse
Using a FS designed for Flash media will help with some of the fixes but not all. For example the mimocan fix it should help, as it will reduce overhead and increase the longevity of the Flash. Also since the mimocan fix is on the external card, it is properly un-mounted at shut down so this helps stop corruption.
Conversly it won't really do anything for the ext2 fix because that is a virtual disk sitting on top of another FS, which is not un-mounted at shutdown.
Potentially a custom ROM could replace the FS on the internal flash, but Samsung must have did what they did for a reason. It is hard to tell if they did what they did because they were lazy / didn't know any better, or they knew something we don't.
Bjd223 said:
Potentially a custom ROM could replace the FS on the internal flash, but Samsung must have did what they did for a reason. It is hard to tell if they did what they did because they were lazy / didn't know any better, or they knew something we don't.
Click to expand...
Click to collapse
Samsung use rfs because for some misguided and twisted reason some manager there thinks it is good and proprietary so they have a leg up on the competition. They go on about how quick rfs mounts as a benefit (yaffs and other flash files systems are slow to mount).
RFS isn't exactly terrible in the right places - the /dbdata symlink hack works because RFS-on-NAND is faster than RFS-on-SD, even though raw reads on the internal SD are faster than on the OneNAND partitions. I suspect it simply interacts poorly with devices that already do their own remapping and wear-leveling, or perhaps that the performance hit is a result of using much larger blocks on /data. I wonder what would happen if one just formatted /data with smaller blocks than 16KiB?
kennethpenn said:
YAFFS is an optimized filesystem for use with NAND flash memory
Click to expand...
Click to collapse
That already explains it all. I do not think it can be used on SD. This is copy-paste from UBIFS, but I believe it is the same for YAFFS:
This is why UBIFS does not work on MMC cards and the like - they look like block devices to the outside world because they implement FTL (Flash Translation Layer) support in hardware, which simply speaking emulates a block device on top of the built-in raw flash.
Click to expand...
Click to collapse
For the ext2 hack with loop device (file formated as ext2) I have no idea if it is any better.
Why do I feel like i'm the only one without the wake up lag issue? .
tbae2 said:
Why do I feel like i'm the only one without the wake up lag issue? .
Click to expand...
Click to collapse
I bet u r on stock kernel
Sent from my SAMSUNG-SGH-I897 using XDA App
Doesn't the Captivate already use yaffs2? Try running "mount" in a terminal emulator or in adb shell.
junknuts said:
Doesn't the Captivate already use yaffs2? Try running "mount" in a terminal emulator or in adb shell.
Click to expand...
Click to collapse
no, it uses rfs for everything...
yaffs2 is not really any better than rfs, both file systems, however, are not intended for use on SD memory. In other words you would be exerting great effort to get yaffs2 as your file system for the NAND part of your phone, and get little to no improvement. As for the mimocam fixes and the like that use your SD card or Internal Memory (Internal SD Card essentially), ext2, 3, or 4 would still be better options than yaffs2 or rfs as they are intended for true NAND Flash memory not block based SD memory.
tbae2 said:
Why do I feel like i'm the only one without the wake up lag issue? .
Click to expand...
Click to collapse
Ok, silly question...what is "wake up lag"? Either thats too technical of a term, or too vague, I dont really get it.
derek4484 said:
Ok, silly question...what is "wake up lag"? Either thats too technical of a term, or too vague, I dont really get it.
Click to expand...
Click to collapse
When you don't use your phone for a while, the phone goes into a "sleep" mode where the screen is totally black and does not respond to any touch. To "wake up" the phone, you press the power button, after which you can see the phone locked screen, and the screen responds to touch again. On a normal Captivate with the precompiled kernel installed by Samsung, or on a Captivate where you have flashed the same precompiled kerenel via Odin, the delay between pressing the power button, and seeing the phone locked sceen is usually less than a second or so. On phones where a kernel compiled from the Samsung released source code (either with or without any modifications) is flashed on, there is a much longer delay between the power button press, and the phone waking up to show the phone locked screen. This is what we mean by wake up lag. We still don't know what causes the rebuilt kernels to lag on wake up.
*Moved question to General*
Flash memory: NAND vs SD?
Hello,
Mr Taco above made a distinction that I can't figure out. He suggests that YAFFS is for "NAND" memory and not "block based SD." Here is my confusion.
From what I have read, it appears as though typical flash multimedia drives, such as MMC and other SD cards, are all created from NAND chips (as opposed to the less-memory-dense NOR chips). So I do not understand the distinction between "NAND" and "SD."
Also, he mentions "block based" devices. But as far as I can tell, "block level" has to do with the *interface* to storage and not the nature of the storage itself. For example, one could map a bunch of random access memory and present it as a block device (say, so it could be treated as a disk with a filesystem). Similarly, mmap and other techniques use disk storage with a memory-like interface (as opposed to a block interface).
So here are some questions I have that might help me figure this out:
1. Presumably the Captivate has some true RAM, that is, non-NAND chips that give truly random access memory that is not persistent upon shutdown.
2. That RAM has a normal (non-MTD) interface (512MB?).
3. Besides this RAM, the Captivate has an internal NAND-based flash drive (2gb).
4. It also has another internal NAND-based flash drive (16GB).
5. It has an external slot for a micro sd card, a NAND-based flash device accepting up to 32GB.
If these are all true, I have questions.
A. Is mimocam's (sp?) solution to use actual RAM, synced to flash, to load certain partitions?
B. Or is his solution implemented on the internal 2GB flash?
C. Certainly, other solutions have used external and internal flash (the ext2 solution, for example).
D. So, isn't it still true that any solution that would rely on the 2GB, internal 16GB or external NAND-flash would profit from using YAFFS?
Thank you in advance,
-0
zeroaltitude said:
Mr Taco above made a distinction that I can't figure out. He suggests that YAFFS is for "NAND" memory and not "block based SD." Here is my confusion.
From what I have read, it appears as though typical flash multimedia drives, such as MMC and other SD cards, are all created from NAND chips (as opposed to the less-memory-dense NOR chips). So I do not understand the distinction between "NAND" and "SD."
Click to expand...
Click to collapse
I'll repeat once again, it is from UBIFS page, but afaik it is the same for yaffs and others:
One thing people have to understand when dealing with UBIFS is that UBIFS is very different to any traditional file system - it does not work on top of block devices (like hard drives, MMC/SD cards, USB flash drives, SSDs, etc). UBIFS was designed to work on top of raw flash, which has nothing to do with block devices. This is why UBIFS does not work on MMC cards and the like - they look like block devices to the outside world because they implement FTL (Flash Translation Layer) support in hardware, which simply speaking emulates a block device on top of the built-in raw flash.
Click to expand...
Click to collapse
"One thing people have to understand when dealing with UBIFS is that UBIFS is very different to any traditional file system - it does not work on top of block devices (like hard drives, MMC/SD cards, USB flash drives, SSDs, etc). UBIFS was designed to work on top of raw flash, which has nothing to do with block devices. This is why UBIFS does not work on MMC cards and the like - they look like block devices to the outside world because they implement FTL (Flash Translation Layer) support in hardware, which simply speaking emulates a block device on top of the built-in raw flash."
Although I do appreciate the attempt to answer my questions, this reply is very puzzling.
My question was why did Mr Taco make a *distinction* between SD and NAND as far as flash memory goes. The reply, quoted above, *lumps SD and NAND devices together*. Furthermore, the primary topic of the quotation is yet another filesystem, UBIFS, that is not even under discussion (at least, not in my post it wasn't).
So, I'm happy to read up on UBIFS, however, I don't see how that can possibly answer the specific questions I have. I hope someone with some familiarity with device drivers or filesystem internals will drop by this thread. Otherwise, it's back to the research .
-0
NAND flash is typically configured so that for each block of stored data, there is a sub-block for supplementary data. This is used for wear-levelling, etc. "Raw" NAND devices such as kernel mtd devices expose this as well as the data portion of each block, and filesystems like yaffs2 and ubifs incorporate wear-levelling into the filesystem design, and rely on having direct access to raw flash.
SD cards contain a controller that does its own wear-levelling, and expose only the data segment of each block. Yaffs2 and ubifs will simply not work on such a device, nor are they really appropriate choices for it. A block-oriented filesystem is needed here, and because we can't guarantee graceful shutdowns, it needs to have some sort of data integrity scheme. Ext3/4 are journalled, Samsung's RFS is FAT with copy-on-write added, and the other possibilities here are nilfs2 and btrfs - but there are no backports of reasonable recent btrfs versions for the kernel Samsung has provided to us.
Unhelpful said:
nilfs2 and btrfs - but there are no backports of reasonable recent btrfs versions for the kernel Samsung has provided to us.
Click to expand...
Click to collapse
Thanks. I'm a bit boring to explain the basic things ppl can google first.
About fs. Do we need metadata or block level journaling?
We can consider ReiserFS or JFS? btrfs sounds nice tho.
I want to be able to securely delete files(overwrite the whole file at least once) on my s3. I tried to install the app undelete which also has an option to securely delete files. But when I start the app it complains about not finding any fat formated file system. I also tried shredroid but this program crashes on startup, maybe because it also expects a fat file system? So is there a working way to secure delete files on the s3? It doesn't have to be an app, if there is some linux utility like srm compiled for android that would be great.
There's an app called Secure Deletion for Android in the Play Store that should do what you need. No free version, though, and it only has one review and 50+ downloads. FWIW, the single review was 5 stars. I'm gonna keep looking because I would like the same thing.
From what I can tell, stuff like that wont work on our devices' internal memory due to the method samsung has used to grant access to the partitions. Undelete never worked on my Xoom because of the same issue, possibly due to the fact both devices utilize MTP. However, it should work on any external SD cards you may have, however that is just me speculating as I have not tried it.
Sent from my SAMSUNG-SGH-I747 using xda premium
d3athsd00r said:
From what I can tell, stuff like that wont work on our devices' internal memory due to the method samsung has used to grant access to the partitions. Undelete never worked on my Xoom because of the same issue, possibly due to the fact both devices utilize MTP. However, it should work on any external SD cards you may have, however that is just me speculating as I have not tried it.
Sent from my SAMSUNG-SGH-I747 using xda premium
Click to expand...
Click to collapse
Now that you mention it, the internal storage is mounted as a FUSE filesystem, not the expected ext3/4, so it's probably because it's a "fake" filesystem. I wonder if there is a way to access the raw physical filesystem (/data is ext4, which should be compatible with most secure delete apps) instead of the userspace filesystem.
EndlessDissent said:
Now that you mention it, the internal storage is mounted as a FUSE filesystem, not the expected ext3/4, so it's probably because it's a "fake" filesystem. I wonder if there is a way to access the raw physical filesystem (/data is ext4, which should be compatible with most secure delete apps) instead of the userspace filesystem.
Click to expand...
Click to collapse
Thank you. Couldn't remember what it is called. But that is exactly the reason it can't find it with the app. Also, if journaling is enabled, then a secure delete still wont work on ext3/4. This isn't a bug, its due to the way ext with journaling handles reads/writes.
Sent from my SAMSUNG-SGH-I747 using xda premium
Anyone know why it's mounted via fuse? The same files are also available via /data/media and /data is mounted as ext4. Maybe it would be possible to unmount the fuse filesystem, remount the data filesystem as ext2 (no journaling) and then run some secure delete program?
octapod said:
Anyone know why it's mounted via fuse? The same files are also available via /data/media and /data is mounted as ext4. Maybe it would be possible to unmount the fuse filesystem, remount the data filesystem as ext2 (no journaling) and then run some secure delete program?
Click to expand...
Click to collapse
It's because of the MTP. If it were mounted as a normal filesystem, you could use USB Mass Storage, but you would lose access to all of your internal storage while it was in UMS mode.
Also, I don't think you can mount one filesystem as another, so mounting ext4 as ext2 will possibly corrupt your data. What you could do is mount the ext4 filesystem with the journal features turned off (which is practically the same as ext2 but better), then secure delete.
The issue for me is, that task seems a lot more involved on a phone/tablet than it is on an actual computer. Not sure if it would even work. And if you want to secure delete every time you delete something, that's a whole lot of effort to go through. Of course, if you want it deleted that badly, it's probably worth it to you.
This is completely pulled off of my other thread in the HOX+ section
our device is a data/media device which is why usb mount don't work
here's a link that explains it all
http://teamw.in/DataMedia
and part of the convo with dees_troy is below
<Dees_Troy> Nope, it will *never* work on a data/media device
<Dees_Troy> read and learn: http://teamw.in/DataMedia
<WinBot> [Link] http://tinyw.in/lstO :: What is a data media device? | TeamWin
<Dees_Troy> definitely worth understanding
<Dees_Troy> at some point we're going to try to kang in MTP for recovery
<Lloir> so for now then it's sideload or from inside the rom
<Dees_Troy> or adb push
<Lloir> aye
<Dees_Troy> or gtfo
<*****> so cant mount usb storage with newer devices...hmm one x did guess this is where confusion at least on my part came to be
<Lloir> lmao Dees_Troy
<Dees_Troy> one x wasn't a data media device
YOU MUST either transfer the rom\boot\porn\audio\mods while the phone is on or use adb push or even sideload when in recovery, THIS IS THE ONLY way
IF you want to read without clicking the link i'll whack it up in #2
What is a data media device?
I'm writing this page because there seems to be a lot of confusion about how many of the newer Android devices work. Starting in Honeycomb 3.0 with the Xoom, Google changed the way that they handled storage. Instead of having a "data" partition with your apps and a separate "sdcard" partition for storage, Google started giving you a single, very large data partition. Inside /data is a folder at /data/media that contains all of the contents of what you think of as your internal sdcard.
Since /data/media is part of /data, we pretty much never actually format the data partition. Formatting data, of course, also removes the media folder that contains the internal sdcard. When you choose a factory reset, instead of formatting, we use rm -rf commands to remove all the folders except for the media folder so that we can remove all of your apps and settings while leaving your "sdcard" intact. In TWRP we also have a wipe internal storage option that rm -rf's the media folder and a "Format Data" option that formats to recreate the entire file system in case something goes completely wrong or to remove device encryption.
When you're booted to Android, Android fuses the media folder to /sdcard and emulates a FAT files system that doesn't have permissions for legacy apps. We don't currently have fuse in recovery, so we just add an extra mount command to mount /data/media to /sdcard so in recovery you still have to worry about permissions on /sdcard.
Because the "internal sdcard" is not a true FAT file system, you can't mount it via USB storage. Well, that's not technically true, but the vast majority of people use Windows computers and Windows doesn't recognize ext4. If we were to allow you to mount the data partition via USB storage, Windows would claim that the device wasn't formatted and offer to format it for you, which, as you can imagine, would be a disaster. The whole ext4 setup is another reason that Android switched to using MTP for transferring files. Most of these devices don't have the necessary kernel configuration to even support USB storage mode, so it's not very easy to enable USB storage if we even wanted to try. Unfortunately at this time, MTP isn't available in recovery, so if you have no other option, you will have to use adb to push and pull files to/from your device.
As a special note, if you choose to do a factory reset from your ROM, even if the ROM says that it will wipe everything including the internal storage, well, that's not what TWRP will do. A stock AOSP recovery would format data including the "sdcard" but TWRP will use its regular factory reset setup that leaves the internal storage intact.
There are a couple of nice gains with using this setup vs the old data + FAT storage partition. With /data/media you, as the user get more control over how you use your storage. If you have a ton of apps, then that's no problem since you have a huge data partition to work with. If you don't have a lot of apps, you get more room to use for storing things like movies. Further, ext4 doesn't suffer from the 4GB file size limit that FAT has, so you can have a large, high-def movie on your device if you like. I'm sure another motivating factor was to get Android away from using FAT which is a Microsoft creation. Performance on ext4 in Android is also probably better than FAT. As a downside, data media devices tend to store a lot more app data in the "data" section and so backups on these devices tend to be larger.
Thank you for this post. I had been a bit curious about this. I have Evo LTE. But I also Boughy a family member a GNex that I have to maintain. I know that my Evo, I can plug into windows, and when I select USB transfer (not in recovery), it mounts my internal storage (not ext_SD) , and it shows as removable drive in windows. But,....when I plug in GNex, it shows up as GNex , not removable drive H. I always wondered a bit about what this all means.
Thanks for the info. :thumbup:
Sent from my EVO using xda premium
Thanks for the useful post.
If I make a nandroid backup for the whole device, will it exclude the /data/media folder? Because on my old phone the nandroid backup doesn't include /sdcard.
romitkin said:
Thanks for the useful post.
If I make a nandroid backup for the whole device, will it exclude the /data/media folder? Because on my old phone the nandroid backup doesn't include /sdcard.
Click to expand...
Click to collapse
Yes it does, we are past the point where that causes issues.
Everyone just needs to get used to the data/media thing. And learn how to use adb side load
I predict a whole bunch of "bricks" due to people not informing themselves on how the phone works and how to use adb
Sent from my One X using Tapatalk 2
A little bit from my side as well I hope someone will find this helpful. Virtual SD card on Android
Can't you use USB Host if you got recovery installed as well? I think I've read that somewhere...
mike1986. said:
A little bit from my side as well I hope someone will find this helpful. Virtual SD card on Android
Click to expand...
Click to collapse
Link broken, here is the correct URL:
http://android-revolution-hd.blogspot.com/2013/03/virtual-sd-card-on-android.html
Oh and bump for anyone new to the idea of a "data/media" device. You're not in Kansas anymore.
NxNW said:
Link broken, here is the correct URL:
http://android-revolution-hd.blogspot.com/2013/03/virtual-sd-card-on-android.html
Oh and bump for anyone new to the idea of a "data/media" device. You're not in Kansas anymore.
Click to expand...
Click to collapse
Lloir, thanks for sharing and good explanation. MTP makes sense and on my One, this way file transfers are simple and fast. However there are two things I dislike about MTP right now:
When you copy files to the device, on some file types it throws a warning that the device might not be able to read the file. This interrupts the file transfer and I have to confirm I want to copy the file to the device. On an older MP3 player I have, I could deactivate this by editing a device capabilities XML file. On the One, I did not find such a file. Perhaps an option in the MTP deamon?
When I am transferring files, I can't browse through other folders at the same time. It says the device is busy.
Is there a way to solve those two issues or "is it what it is"?
Lloir said:
Windows would claim that the device wasn't formatted and offer to format it for you, which, as you can imagine, would be a disaster
Click to expand...
Click to collapse
I did that on a device. Windows offered to format and I did it. Man that was not good! 2 hours later after manually restoring the partition values I was OK and the most amazing thing all my data was there perfect and untouched :silly: :highfive:
Thanks for that great explanation. Been having some issues with windows and android. Nothing big just had to sideload my ROM and killed my "sdcard" at least now I know why
Note from the Author -
I am moving on to the N5 now and ditching my S3. I will continue to maintain this thread, however - please do PM me if you think that something needs to be changed or updated in this thread as I doubt I will be answering questions within the thread as much. Please don't PM support questions to me. Only PM updates that need to be made in the thread.
It's been a blast!
Regards
Dan
S3 Storage (Data Loss Recovery / Prevention / Info)
This thread is intended to give you an overview of some of the Storage of the S3 from a Data Loss and recovery perspective. It is not intended to cover USB sticks or mods to Swap / Mount other storage. It is solely to cover day-to-day data concerns and give a background to how these things work
Please note, if you have recently swapped between Android 4.1 and 4.2 and cannot find your sdcard data, you need to read [Info] Flashed 4.2? Can't find your /sdcard data?
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Before we get started...
Here are a couple of threads you should get familiar with before posting on XDA.
Forum Rules - use Search before posting
Post Questions or Support queries in Q&A, NOT General
Backround of Android storage (Pre-S3)
Firstly, I think we need to understand how Android worked historically as this will help us to understand how the S3 works now.
A legacy android device, lets pretend the HTC desire on Android 2.2 as this was a standard configuration at the time. It had 2 major partitions (and several minor ones which are irrelevant to this topic). It has the /system partition and the /data partition. These were partitions of the internal NAND flash memory.
/system is where the Android operating system lives. The user could not delete or change anything in here (unless you were rooted). All the software that came with the phone was installed in the /system partition
/data is where all the userdata goes. Whenever you installed an app from the market, it installed to /data/app and it put all it's important data into /data/data. Also any system settings you changed (Wallpaper, ringtones etc) all were stored in /data/data. When you did a factory reset, it wiped /data and everything in it.
Of course, having these 2 partitions was not enough for everyday use. There was no where to store your music, photos, documents etc. /data is just for app data and settings. So this is where /sdcard comes in
/sdcard is the Android mount point for the External SD card in this legacy android device. This means that when you inserted a Micro SD card, Android used /sdcard as it's internal reference to where the card's storage is. The /sdcard was a necessity before you could take photos. Over time, bigger apps started to put other data here. For example, a GPS / Map application would store its apk (application package) in /data/app and store your personal configuration settings in /data/data but it may download and store offline maps somewhere on the /sdcard. In older devices, the internal Memory (NAND Flash) was usually too small to allow much data on it. Many users would have to root to get more storage space or keep uninstalling apps to keep the "low on space" warnings at bay
How the S3 is different
Well, the S3 is substantially different. There are of course SOME similarities. For example, the S3 still has internal NAND Flash Memory. This is often referred to as the eMMC (Embedded MultiMedia Card) - which still contains the /system and /data partitions, used in exactly the same way.
The main difference is /sdcard. The S3 is designed specifically so using an external micro sd card is NOT a necessity. It has a larger NAND Flash internal memory (eMMC) so it can also have an "internal SD card". This is where people start to get confused. The entire internal memory is an eMMC which is essentially an internal SD card, however a partition of that internal memory is /sdcard.
OK, I know - let me explain. /sdcard is a mount point that Android uses to know where to store /sdcard data. But on the S3, instead of storing it on a required external sd card, it points to an internal memory partition. Now here is the clever bit. The /sdcard actually points to /data/media (or /data/media/0 in Android 4.2 onwards). So you continue to have your /data partition, but within /data you have:
/data/app
/data/data
/data/media
The clever thing is that the file system that android uses for /system and /data is a Linux file system called Extended FS. In our case, we use the Ext 4 file system. This is important to understand because these file systems do not work with Windows so an external SD card would usually be Fat 32 file system, or exFAT so we could plug it into our windows computers and read the contents. What Samsung have had to do is use the FUSE file system to allow /sdcard (or /data/media) to exist as a FAT file system within the EXT 4 file system. Clever stuff. But it has it's pro's and cons...
You lost me at file system
All electronic systems that have an operating system and store data use a file system. Think of it in it's simplest form. Imagine a school text book. It has lots of chapters about different things. It has a "Table of Contents" in the first few pages, telling you where each chapter in the book is so if you want to know what page chapter 13 is on, you look in the contents and find the page and go straight there - The alternative is looking through each page individually to find the chapter. Not a quick process.
Well data storage works the same. When you put a file on a hard drive, sdcard, USB stick (or whatever) it is written to a specific location. When it is written to this location, the location is added to the file system. So when you put word.doc onto the drive, The file system is informed of the (very complicated) location of the file. When you tell Windows, Android (or whatever) that you want to open word.doc, the operating system consults the File system and goes to retrieve the data from its true and real (yet very complicated) location on the drive.
There are many file system types still in use today. Usually they are operating system specific. For example, Ext 4 is a Linux file system (and Android by Proxy as it is Linux kernel based) and Windows cannot read Ext file systems. Similarly, exFAT is a Microsoft file system (also used for sdcards on the S3) and cannot be used (easily) on Linux machines. Since most everyday users of the S3 are Windows users, you can see now hopefully why it was important for Sasmung to use "magic" and implement FUSE to allow an exFAT file system to be used for /sdcard, within the Ext 4 partition of /data
Did I REALLY need to know ALL that?
You know what? Probably not but it may go some way to help understand the limitations we will cover later on.
A bit more info for the S3
Obviously, the internal memory supplied with the S3 may not be enough for all users so they added the ability to add expanded memory in the form of the Micro SD card. Android uses the Mount point of /extSdCard now, instead of /sdcard like it used for legacy devices - because /sdcard is already in use elsewhere.
One thing many of you have probably noticed is that with the S3, there is no option to mount the /sdcard or /extSdCard as USB Mass storage on your computer. You must use MTP or PTP.
PTP - Photo Transfer Protocol. When you connect your S3 to your computer using PTP, Your computer sees it as a camera. It will show photos on your "camera" and will set about implementing the default camera options (such as suggesting you import your photos) etc. It won't show documents or other media necessarily.
MTP - Media Transfer Protocol. When you connect your S3 to your computer, it will be seen as a media player. This should allow you full access to all the files on there, including word documents and the such like.
One of the reasons for this is that because /sdcard points to something using the FUSE file system and is not a true partition, it would be difficult to allow it to be used as USB Mass Storage. It may or may not be possible but the biggest advantage of using MTP / PTP is that the computer and the S3 can both access the internal memory at the same time. With USB Mass Storage (UMS) this is quite awkward and can result in errors.
Deleting data
This is partially why we needed to understand a little about file systems. So I could explain to you how data is handled when it is deleted.
As I explained earlier, when you write a file to memory, a corresponding entry is written to the file system to advise the OS where the data is. Sure, you may think you are writing the file to /sdcard/documents/work directory on the internal memory, but in reality these directories or folders do not actually exist at a memory hardware level. The data is written to a block and the file system is informed where that block is, how big the file is, what directory it should appear in to the OS etc. When a file is written the memory, the OS see's the available space go down and the used space go up. All this information comes from the file system.
When you delete the file, the actual data is NOT deleted. It remains where it is on the memory. The block is not overwritten. When the OS is told by the user to delete the file, the File system entry is deleted. This changes the free/used space as the file system is no longer accounting for the data, however the truth is the data still exists. When the next request to write a file to the memory comes from the OS, the file system will think the block where the old data was is empty and will overwrite it.
It is this difference between the file system and reality that allows data to be recovered by external software. if you do not write any data to the memory, external software can scan the memory for data whilst bypassing the file system all together. Ff course the window is small. You only have a very limited time to recover data before the file system allows the data to be overwritten with a new entry.
This is not just true of a deleted file. Even formatting the memory (which is actually just re-creating a new, blank file system) leaves all the data in tact behind-the-scenes and can all still be recovered until you start writing data to the memory. Cool huh?
Wow, all this time I've been stressing, is it really that simple?
Awwww snap! You got me. No it is not that simple. All this PC software, example: Piriform recuva only works on a computer drive. In windows, imagine this is anything with a Drive Letter. C: drive etc.
The only way to get a drive letter on your sdcard is to use USB Mass Storage mode, which as previously discussed - is not possible on your S3 (unless you are rooted, you can mount USB mass storage in custom recovery or use a UMS app from Play). The alternative is to use a card reader on your PC and put the sdcard in it.
There are also apps like Undelete for root users - which again, you guessed it - requires root. So if you're not rooted, it's simplest to use a card reader which can be bought for peanuts.
It's worth mentioning, NONE OF THE ABOVE will work with /sdcard on internal memory. It is not possible to get your data back once deleted from internal memory. Once gone, it's gone forever. You can only restore from /extSdCard (removable, external SDcard)
Phone won't boot, can I get my data back from internal memory?
Let's start by saying, it depends why your phone won't boot. If it's an SDS (Sudden Death Syndrome) type issue, where your internal eMMC (NAND FLash memory) has failed, then no. However, if you believe this is not the case then you can get your /sdcard data using adb BUT you need a custom recovery to be flashed via Odin before you do this. Read [REF] Understanding the basics before rooting your S3
However, if out of curiosity - you do still want to get your data off, using adb , read below:
Pre requisite is having adb "installed" on your windows PC. Download THIS file and follow the instructions in the readme.
You need to observe the following. For android 4.1.x and earlier, /data/media for android 4.2.x and newer, /data/media/0 - I will assume 4.2.2 for this guide,.
1) Boot into recovery, connect usb and go to "mounts and storage". Toggle the "mount data" options to mount these partitions. Tip, when mounted, the option then becomes "unmount data"
2) Open "cmd" in Windows and type the below code, which will copy all your data to a folder called sdcard on your windows desktop
Code:
adb pull /data/media/0/ c:\users\rootsu\desktop\sdcard
Also note, this assumes you have windows vista or newer. Also, it assumes your windows username is rootsu.
That's it, simple.
Display and Digitiser won't work, can I get my data back from internal memory?
You can use adb and a custom recovery to pull data from your /sdcard or even app data from /data/data
Pre requisite is having adb "installed" on your windows PC. Download THIS file and follow the instructions in the readme.
You need to observe the following. For android 4.1.x and earlier, /data/media for android 4.2.x and newer, /data/media/0 - I will assume 4.2.2 for this guide,.
1) Boot into recovery, connect usb and go to "mounts and storage". Toggle the "mount data" options to mount these partitions. Tip, when mounted, the option then becomes "unmount data"
2) Open "cmd" in Windows and type the below code, which will copy all your data to a folder called sdcard on your windows desktop
Code:
adb pull /data/media/0/ c:\users\rootsu\desktop\sdcard
Other things you may want to pull.....
Code:
adb pull /data/data/com.android.providers.telephony/databases/mmssms.db c:\users\rootsu\desktop\sdcard
Code:
adb pull /data/data/com.android.providers.contacts/databases/contacts2.db c:\users\rootsu\desktop\sdcard
Also note, this assumes you have windows vista or newer. Also, it assumes your windows username is rootsu.
That's it, simple.
Data corruption
When data becomes corrupt, there's really not a lot you can do. The file system knows where the data is already. If it's corrupt, you're stuck. Most common causes of corruption are:
1) Dirty unmount of /sdcard. SD card pulled out whilst it is being written to / phone shuts off whilst being written to. SOMETIMES - Plugging the card into a card reader in windows, Windows will ask to fix it and MAY fix it.
2) Fake SD card. This is really the MOST common. Get a card reader ans use:
h2testw.exe for windows to test your card in a card reader. Set it to read the full size of the card, which will take hours but well worth it.
If you get a result like this:
Code:
Warning: Only 63995 of 63996 MByte tested.
The media is likely to be defective.
3.8 GByte OK (8072512 sectors)
58.6 GByte DATA LOST (122989248 sectors)
Details:2 MByte overwritten (4096 sectors)
...Then you have a fake card, that is really 4 GB. I'll explain this.
Commonly, fake cards are reprogrammed to "think" they are high capacity cards, such as 32 GB or 64 GB to defraud buyers out of money. This is common on eBay (Never buy cards from eBay).
When these cards are formatted, the file system also thinks it is this fake capacity. Normally, when a card is full, the file system will report to the OS there is no more space and this prevents additional writes to the card. However, in the case where the card is 4GB and the File system thinks it is 64 GB, the tricked file system doesn't know the card is full. The file system keeps allowing data to be written to the card, over writing the existing data but without replacing the file system entries. The file system thinks data that has been overwritten hasn't been overwritten so when you try to open one of these files, it is essentially "corrupt" or non-existent.
Preventing data loss
Time to wise up guys. It is possible to recover data off your removable media, but internal memory - very unlikely. No apps on your PC or Android will help with deleted data. So you need to backup.
Dropbox - Use dropbox to automatically upload your photos to online storage.
Foldersync - Use FolderSync to upload important sdcard files to your dropbox account, or better yet - got a computer thats always on at home? Set foldersync to schedule a sync over wifi whilst you're asleep.
Other info
Interesting tidbits
Quite an exhaustive reference guide you got here rootSU thanks this will sure come handy for all of us :good:
Cheers
Thank you very much for taking the time to write this. It's a non academic approach to a sum of keywords and all of them are explained in such a manner that it would be almost impossible to misunderstand
Nice!
Nice post!
There are a few other interesting tidbits of info that might be worth mentioning:
- eMMC has an internal micro-controller that runs very specific firmware (and SDS was mainly caused by a bug in that firmware)
- eMMC (just like SSD) has specific writing/erasing limits and commands to deal with that - as a very general idea it can write about 4k at a time but can only erase in much larger blocks - like 64k (at least, but a 16GB model could have a much bigger block); normally on the same erase-size block there is very special list maintained, and based on that list wear leveling is implemented;
- all flash-based memory AGES - there is only a limited amount of erase/writes cycles possible before a point where the info is no longer reliably-stored; in some models that value can be incredibly small! to avoid writing more to some regions than other a mechanism call wear leveling is implemented; that one can have a big impact on both speed and reliability (but really don't expect it to create miracles)
- since it is very important for the speed and reliability of the flash memory to return unused blocks to this internal lists there are special TRIM commands that informs the firmware that the block can be garbage-collected; with an OS that supports TRIM, when a file is erased the blocks are also TRIMmed; this is one extra level that makes recovery basically impossible under normal circumstances
- this does not really mean that things are completely impossible to recover, just that you might need to spend so much on it that recovery would be impractical for any item worth less than 100000 US$ to 1 million US$
EDIT
- also just as with SSD it is not a bad idea to keep a good percentage of the flash memory free - IMHO at least 4GB for 16GB models, 6-8GB for 32GB models - that will improve performance since fragmentation (CLARIFICATION - free-space fragmentation) will grow much slower
- unfortunately there is no program for eMMC similar to smartctl (or any other SMART-data reading program) on normal SATA/IDE/SCSI disks - there seem to be some proprietary commands that are somehow similar but those are generally undocumented.
xclub_101 said:
Nice post!
Click to expand...
Click to collapse
I've put a link to your post in post 1. Where as it's not strictly relevant to my point, it is interesting stuff....
Fragmentation isn't an issue on ssd's. Its an issue on hdd because the head must physically move to another area of the Platter to get the data. That's the slow down. Defrag of a hdd moves all the used blocks (data) together so the actuator doesn't need to move much.
Performance degrades over time on ssds because every write, if data already exists must be erased too. But this hasn't really been an issue so much since TRIM became widely available.
-----------------------
Sent via tapatalk.
I do NOT reply to support queries over PM. Please keep support queries to the Q&A section, so that others may benefit
rootSU said:
...
Fragmentation isn't an issue on ssd's.
...
Click to expand...
Click to collapse
Thank you.
Also sorry for the misunderstanding with my contraction - what I wanted to say was free space fragmentation - that one does matter a lot on solid-state memory because of the garbage collection and some controllers have been famous for having a huge drop in performance with little free space - I will try to also correct that post.
xclub_101 said:
Sorry, I used a misleading contraction - what I wanted to say was free space fragmentation - that one does matter a lot on solid-state memory because of the garbage collection and some controllers have been famous for having a huge drop in performance with little free space - I will try to also correct that post.
Click to expand...
Click to collapse
Yep, it's true about Garbage collection, but TRIM *should* handle this nicely as should "over provisioning" although probably, some cheap SSD's may not over provision.
EDIT> Actually (sorry everyone for off topic) if you're interested in SSD's, these articles are a "fun" read... (I put fun in speech marks as it depends how geeky you are )
http://en.wikipedia.org/wiki/Trim_(computing)
http://www.pcworld.com/article/2038...-ssds-what-makes-these-speedy-drives-hum.html
Update to post 1:
Note from the Author -
I am moving on to the N5 now and ditching my S3. I will continue to maintain this thread, however - please do PM me if you think that something needs to be changed or updated in this thread as I doubt I will be answering questions within the thread as much. Please don't PM support questions to me. Only PM updates that need to be made in the thread.
It's been a blast!
Regards
Dan
Awesome bits of info. This is the game changer. I learned a whole lot just by reading here in XDA. I've only been using Android for a few weeks but thanks to XDA, I've already rooted, installed a bunch of apps and kept my OCD in check.
my device memory has corrupted and when i start recovery mode i get "E: faild to mount /cash (invalid argument) "
Hello all,
Now that we have root on our aesthetically unique device, I believe our next problem is storage space. I don't know about you all, but my aquos crystal has about 4 gb to work with when it comes to apps. Folder mount works well for me, as far as linking the obb files of large apps to the external sd card, but I still have to store a portion of any particular app on the phone's internal storage, which for me is no bueno. There has to be a better way. (For the sarcastic among you, buying another phone is not an option for me.)
Back in the day of Gingerbread, I remember having a whopping 300 mb of app storage to work with. I was able to partition a portion of my external sd to be used as internal storage for the device, solving all my storage woes. It worked so well, that I was sure that I would be able to find a similar method for the aquos crystal.
Now, I see that Link2sd is able to work with ext2, ext3, ext4, and fat32 partitions, so I figured I would attempt to create a ext2 partition on the micro sd to link to my internal storage directory. I used the Partition Wizard mini tool to create a ext2 partition in windows. Unfortunately after starting up Link2sd and selecting the format of my ext partition (ext2), Link2ds gives me an error telling me that the mount script was unable to be created. I've read that if we get this error, we can select ext4, even if we are using a different partition format. This got rid of the error. Mount script was successfully created. Autolink is selected, which should install new apps directly to the new partition. It doesn't, not for me at least.
I've read several guides around the internet dealing with this issue, as well as completely different methods besides Link2sd, and nothing seems to work for me. My goal in this is to supplement the capacity for apps. I would love to find a way to store all the library, devlik, and app files for each app on the external card, rather than having to keep some of each app on internal, and its larger counterparts on external. As you know, apps are constantly updated, and what seems to happen is if you install an app, then link it to the sd card, and the app gets updated in the future, the updated portion will get installed on the internal card, making your 'linked' app kind of pointless. What I'd really like to do is disregard the phone's internal storage altogether and use something like a 64 gb card as the phones internal storage. This may or may not be possible.
I'm not a programmer. I have a very basic understanding of how these things work. When it comes to things like this, I am literally standing on the shoulders of giants, and benefiting from their tenacity and willingness to share the fruits of their labors. Their exploits, and the internet that tells their story, have allowed me to root and modify practically every console and android device I've ever owned. With that being said, thank you to all that have been involved in unraveling the real potential of this device.
So, do any of you fine saints and scholars know of a way to get around the feeble storage capacity of our aquos crystal?