Drivers for HW cards #39
Comments
|
I'm assuming your switching the hardware into non raid mode ? Anything else won't make much sense to nwipe. Doesn't nwipe then see the drives once the controller is in non raid mode? What sort of drives are they? If the drivers are available in buildroot they can be added to ShredOS quite easily (but still requires a compile of buildroot) And if they are not available in buildroot but the source is available it may be possible to compile and built into ShredOS anyway. |
|
Yes non Raid. A typical Freenas/Truenas setup. I use an Intel E91267-203 RAID Expander Storage Adaptor which then feeds a PERC H310 which has been crossflashed to IT mode LSI 9211-8i. One set of 4 drives is directly attached to the Dell Perc and they cannot be seen either. All drives are a mix of sizes and are sata disks. I suspect it’s the LSI 9211-8i (aka Dell Perc) that doesn’t have the drivers. Cheers |
|
The megaraid SAS driver which supports Dell Perc, LSI, Intel, FSC, ACER devices is available. I'll rebuild ShredOS with this driver if you can test it out for me? |
|
Yes no problem. |
|
The drivers to enable a variety of 3rd party RAID controllers (in HBA mode) are:
These should enable DELL server support (DELL-branded LSI controllers, possibly IBM-branded LSI controllers and a bunch of others), and also the unbranded/original LSI controllers. Those were used in NHellFire's DBAN fork and they worked with Dell hardware. |
|
@Firminator thanks, I'll get those added ASAP. |
|
You're the man. Thanks! Let us know. |
|
If you build an ISO like you did with https://github.com/PartialVolume/shredos.x86_64/releases/download/v2020.05.012_x86-64_0.31/shredos-2020.05.012_x86-64_0.31_20210612_legacy_bios.iso I'll be able to test it on my wiping box. |
|
Any updates on this for an iso to test with? I have a megaraid_sas device (9440-8i) I'd like to test with. |
|
He's pretty covered right now accd. to a recent comment: https://github.com/PartialVolume/shredos.x86_64/issues/38#issuecomment-926252171 |
I can also do some testing for you. I have several types of LSI based HBA controllers that i can try out. |
|
@Carl-Wilhelm much appreciated. I'm hopefully going to try and get a new release out by the end of this week, with a bit of luck. |
|
I'm currently working on the latest release, I'm just wondering whether adding iSCSI drivers would be of any practical use to somebody out there ? |
Yes, absolutely if it is not to much extra work for you. 😊 |
Just to add to it. The ability to use fibre Chanel external SAN enclosures would be more beneficial than iSCSI, but i assume that it is a whole New ballgame to do that type of integration. |
|
@skyline-65 @Firminator @Carl-Wilhelm See latest pre-release v2020.05.013_x86-64_0.31 and .img for writing to a USB flash drive. 2020.05.013_x86-64_0.31_20211013.img for USB flash drive. .ISO image for CD/DVD to follow. If you could give it a try with your raid hardware and let me know how you get on. Many Thanks. |
Did some quick testing now. When i say "Did not work", i mean that no disk was visible in nwipe after booting up. Booted up Ubuntu with the last controller, compiled nwipe and was able to run shedding on the disks. |
|
@Carl-Wilhelm I've made some changes to config so that the drivers are builtin to the kernel rather than modules. I'm currently compiling and should have a new version for you to try later today. (Takes about 3 hours to compile). |
|
@Carl-Wilhelm Can you try this one v2020.05.014_x86-64_0.31 The raid drivers are now builtin rather than moduler so auto loaded. You may also find you have an extra virtual drive called something like Linux SCSI debug, just ignore that. I'll remove that virtual drive once the raid controllers are recognised. Let me know how it goes. |
Hï again. Quick test this morning: |
|
@Carl-Wilhelm That's great news, thanks for checking those out. If you haven't already done so can you also use hexedit /dev/xxxx on the Shredos disc or your preferred hex editor to check the disks are being wiped correctly. Maybe a single PRNG stream, no blanking, no verification and then check with the hex editor, the first and last sectors and random sectors in the middle of the disc contain random data and if ok follow up with a single zero pass, no blanking, no verification. Then do the same checks with the hexeditor to confirm the disc has been zeroed. There is no reason why the disc's shouldn't be wiped, but I like to test these things as much as possible. I'll release a official version shortly (version shredos-2020.05.015_x86-64_0.31) that contains the new RAID drivers plus more USB keyboard drivers and the start of adding network support so wipe status can be uploaded to a server via a file or possibly via TCP/IP socket to a KDE program that can monitor a farm of nwipe's. |
Yes sir. I will come back to you tomorrow. No more time to fiddle with it today :-) |
Took me a couple of days, but checked now. After a PRNG i can confirm that the drive is random, and after blanking, it is only zero. |
|
Implemented in official release v2020.05.016_x86-64_0.32 |
|
LSI SAS2308-IT (FW 18.00.00/7.35.00.00) using the newly added mpt2sas driver: attached drives are recognized and the wiping starts. However it shows UNK instead of SCSI/SATA/IDE in the main NWIPE GUI: I see that SCSI is theoretically supported in https://github.com/martijnvanbrummelen/nwipe/blob/master/src/device.c but something doesn't play well. |
|
@Firminator The device name for SAS is missing from the table of valid devices that are displayed in the GUI. So just a GUI issue for the SAS device. I'm just wondering, but are you able to compile nwipe from source? The reason I ask is that I don't have any SAS drives to test with. What I want to do is add an extra line in the log code that displays the raw data returned by ioctl on HDIO_GET_IDENTITY. Once I know what value is being returned I can then add SAS to the device table. If you've not compiled nwipe from source then no problem, I can do another pre-release of Shredos, it just takes longer to find out if I've fixed it properly or not. |
|
We all appreciate you working on this I dare to say. Thanks so much. |
|
@Firminator Thanks, I've just commited the fix upstream on nwipe martijnvanbrummelen/nwipe#350 If you are able to compile the nwipe code and confirm that SAS as the bus type now appears against the appropriate drive. (When compiling the code I don't mean the Shredos code as that takes a lot longer with a lot more that can go wrong). Compiling nwipe from source on a Ubuntu or any Debian based Linux distro should be fairly straight forward https://github.com/martijnvanbrummelen/nwipe#debian--ubuntu-prerequisites and https://github.com/martijnvanbrummelen/nwipe#compilation |
|
I'm using this dedicated standalone box with no OS installed. Just the controller and the drives. Maybe someone else in this thread running their actual live (Ubuntu/Debian) OS off of a SAS controller connected disk can compile nwipe and test it? If you publish another beta/test USB .IMG I can test it within seconds today. |
|
No problem, I can do a ShredOS pre-release that uses the nwipe master. I'll let you know when it's ready. |
|
I just tried https://github.com/martijnvanbrummelen/nwipe#automating-the-download-and-compilation-process-for-debian-based-distros and then copied the freshly compiled nwipe binary with SAS support (v0.32.001) onto a flash drive. Then mounted the flash drive into a live session of ShredOS; copied nwipe from flash drive to /usr/bin to replace the old v0.32.000 with v0.32.001; then ran nwipe_launcher, but it error'd out with |
|
@Firminator Here's Shredos with the master version of nwipe with the fix for the SAS instead of UNK. https://github.com/PartialVolume/shredos.x86_64/releases/tag/v2020.05.016_x86-64_0.32.001 |
|
Interesting. |
|
@Firminator I'd guessed as much, as I was writing the code I was thinking "I wonder if SATA drives can also be attached to this controller" As before I'll need those nwipe logs for the SATA drive, I'm assuming it will say something like Transport protocol= SATA.. quick fix, but I wish I'd gone ahead and added SATA. Are SATA and SAS the only types of drive that work on that controller? |
Yes. If you are thinking SCSI drives... they have a different connector not compatible to SAS connectors. Also you can't connect SAS drives to SATA ports/cables btw :) |
Also the serial numbers don't populate for some reason for the first two drives in the |
I'm assuming the serial numbers therefore don't show in the GUI after the drive model number as well? Looks like I might have to extract serial numbers using smartctl as well. Are model numbers ok? I'm also assuming you've put the XXXXs in the model no and serial no. fields of smartctl output to anonymise the data. |
|
Yes exactly to all three. |
|
With SATA looks like Transport protocol doesn't exist, instead I need to search for "SATA Version is:" |
|
Well, should it show SATA in the GUI or ATA (like it shows now). This is quite a conundrum as you could also show the version number of each protocol to give some info about expected throughput (3GBit compared to 6GBit for example). SAS could also show SPL-2 or SPL-3 (https://www.t10.org/drafts.htm#SCSI3_SAS). Once this is all figured out I can provide you a screenshot for documentation that shows SAS, SATA, ATA, USB on one screen... plus if I find an old IDE drive I might be able to get that drive shown as well. All non-virtualized on real hardware. Ideal for testing. |
I do have Machines with SCSI, IDE and all kinds of SAS. Also have external USB drives. I have one box with debian and compiled nwipe, plus one machine that i boot from usb with shredos. Can do whatever testing you want me to. |
|
@Carl-Wilhelm That will be useful, thanks. |
That it is, nothing is ever simple. All the hardware I've come across so far have been consistent in the way they name the /sys/block/.. path, but not these controllers/drivers. Currently most IDE and SATA devices that follow the /sys/block.. naming convention are just end up being called ATA, but with the controller @Firminator has (sorry, I don't remember if you told me what the make/model was) I can identify SATA or SAS from smartctl but if I followed the previous naming convention, SATA would be called ATA and SAS would be called SCSI. So although technically correct they don't quite describe the drive correctly in terms of what we recognise them as. I think I might change the terminology previously used, i.e ATA was displayed for both SATA and IDE assuming smartctl is installed and provides the relevant field I would replace ATA with SATA. If smartmontools is not installed on a system (which can happen on a distro, although maybe most distros install smartmontools as default) nwipe will revert to it's fallback method which is ATA,USB, SCSI and UNK for unknown. If smartctl is available then hopefully the naming will be SATA, SAS, USB and maybe IDE or more correctly PATA. This would be initially a bit of an experiment but hopefully we can find some consistency irrespective of the controller and driver used. In a perfect world I would have liked to retrieve this info via /sys/block or ioctl calls but I think I would be trying to reinvent the wheel when it's so easy to leverage smartmontools to do the work. I hope most of that makes sense. By all means if you don't agree or I've missed something then let me know. It may be a week or so before I can code this as I'm busy for the rest of the week, but hopefully sometime next week I can do another release. And thanks for all you input, it's good to know Shredos and nwipe are useful. |
It was an LSI SAS2308-IT (with FW 18.00.00/7.35.00.00) We have to figure out what we are trying to show to the user here. We have the protocol, the bus, the interface, the formfactor and the actual physical connectors. I'd argue what we want to see is what interface the device is using and what the expected (theoretical max) speed of that interface is. That would be beneficial for the user although we have some constraints with GUI space and what can be shown on the main nwipe GUI screen without having to create and invoke a page 2/second screen somewhere. Then we have naming confusions. Each interface/bus/standard comes with a certain theor. max bandwidth in MBit/s or GBit/s. I've seen a list somewhere on Wikipedia that shows all this. Here's a list of the ones we would be interested in:
These naming conventions are fortunate since having 'InterfaceName-Version' gives us some optical consistency in the GUI. I'd also argue that we want to show MB or GB per second instead of MBit/s and GBit/s for the interfaces as we are wiping disks and pertaining to this we need to know how fast each interface is. No one wants to convert the raw data rate of the interface (GBit/s) into the actual value that we want to compare to. The nwipe GUI also uses MB/s to show the total 'throughput' of the wipe. Some commands that could be useful to get this info are: but of course there are intricacies when the device is not SATA (/dev/sda) and let's say NVMe (/dev/nvme0n1). NVMe devices would benefit from having nvme-cli installed in ShredOS. I'm sure with megacli or other vendor cli tools you can read out additional info behind a RAID controller in HBA mode as well. In other words we don't necessarily have to rely on just smartmontools. Haven't checked if lshw and hdparm is available in ShredOS. |
|
I'm thinking of a new interface something like this... We would still keep the original (smartmontools might not be installed, so we need a fall back) but maybe by pressing 'I' for info, the default drive selection changes to the screenshot below or maybe you could select this type of display as default from an option. |
|
Oh my. This is awesome. I'm all for it. Yes definitely need a fallback. Best would be to derive the data from kernel/systemtools and only use 3rd-party/additional tools as fallback. What sticks out to me is that it shows SATA 3.0 with a max of 3 GBit/s, but that's wrong. SATA-3 has 6 GBit/s. The drive must be using SATA-2 with a max of 3 GBit/s[1] and of course is backward compatible with the older SATA-1 (1.5GBit/s). There might be actually a jumper on that drive where you decide to run it in 1.5 or 3 GBit/s mode :) If there is GUI space I'd would like to see something like this (using the WD drive in your screenshot) [1] https://en.wikipedia.org/wiki/Serial_ATA#SATA_revision_2.0_(3_Gbit/s,_300_MB/s,_Serial_ATA-300) |
I wouldn't read too much into those numbers, the SATA references are fake, in reality the smartctl data was indeed from the drive mentioned but that drive was attached via a USB 3 port and a very old SATA to USB adapter on a RPI4. running nwipe. I guess that old SATA/USB adapter may have something to do with it? I should hook the drive up to an actual SATA connector on my Dell test system and see whether the smartctl reports differently. |
Looks good, I like it. It's a little longer than I'd like as I need to have some leading spaces on the disc capacity to keep the columns aligned for varying sizes in the list. However I don't think we need the 1., 2., 3 numbering, it's not like we reference that anywhere. Getting rid of the numbering would free up 3 characters so ” /dev/sdb SATA-2 (3.0 GBit/s) (2 TB) (33 C)" should fit in nicely. And if not I can always move the drive info over a character or two. |
|
Now I've counted the characters I think " /dev/sdb SATA-2 (3.0 GBit/s) (2 TB) (33 C)" is 19 characters too long for a 80 character column, but fine for 100+ columns. I would probably dynamically check column width and if it falls below 100 columns move the "3.0 GBit/s” and "(33 C)" into the info pane. What do you think? |
Sounds perfect.... although I have no understanding how it is determined if the layout is just 80 characters (which is what my test box uses) and when you get a 100+ character layout. Does that have something to do with the monitor resolution or is that a video driver thing? |
Well I think your example setup couldn't have been better. It very well has proven why you would want to show the interface max speed... it will put attention to a non-optimized setup using for example slow/bad/incompatible interface adapters like the old one you were using limiting throughput to 1.5GBit/s. For example I have an adapter that will connect 40/80pin IDE drives to SATA ports and it works for wiping, but I'm not sure if I reach full wiping speed due to faulty/buggy/imperfect interface translation from PATA to SATA. I could buy a second brand and then compare :) |
From what I've found on the different systems I have. If you boot on a legacy bios (no UEFI) Shredos will display nwipe in 80 column mode using the frame buffer (no Nvidia/Intel video drivers are used). If you boot UEFI Shredos will boot at full screen resolution assuming Shredos has a video driver that supports your graphics card, which for me, makes the text too small but even so, it's still legible. I started loading some graphics drivers into Shredos but realised real quick that the boot image was growing very fast so had second thoughts about that. Adding drivers for graphics cards was a rabbit hole I didn't want to go down until after all the wiping related features have been added to nwipe. So on my DELL optiplex 9010 it will boot high resolution on either an Nvidia (don't remember the model) or it's onboard graphics, but if I switch to legacy boot in the DELL configuration it runs Shredos in 80 column mode. Personally I like the 80 column mode as I can read the text from across the room, but others prefer the high resolution mode. Each to their own. |
The old SATA/USB adapter did indeed limit throughtput. Here's a snapshot of smartctl -i on the same model of disk attached to the SATA connectors on the Dell motherboard. Doubles current speed from 1.5 Gb/s to 3.0 Gb/s
|
|
However, in reality the actual sustained wipe speed (zero fill, no blanking, no verify) is about 120-140MB/s (0.96-1.12 Gb/s) so the old SATA/USB adapter might be an issue for small amounts of data being read/written to disk but when it comes to a sustained write to disk where the limiting factor is how fast we can write to the platter rather than the disc cache, then the old 1.5Gb/s adapter possibly has little affect for our particular use. |
|
@Firminator For the time being I've added bus type as ATA for a SATA (currently UNK) that is plugged into your LSI controller. We'll revisit this once the selection GUI layout has been changed, but for the time being I'll leave this as USB, SAS, ATA, NVME and UNK as the common names. Committed upstream in nwipe Check smartctl for unresolved bus types SATA #358 |
Is it possible to add drivers that support HW cards/adapters such as Dell Perc 310 so I can erase all 24 drives in my server case? Is there an easy way I could add to the USB boot stick?
The text was updated successfully, but these errors were encountered: