m4tdarkling/Knorrie: re: inappropriate ioctl for device. so the problematic snapshots are of a subvolume like: /storage/root/kvm where kvm is the subvol. there's also a nested subvol /storage/root/kvm/nocow which is +C
m4tit's the 'nocow' directory that's causing that issue in the snapshots
m4tlsattr: Inappropriate ioctl for device While reading flags on kvm.20171023/nocow
m4tlsattr /storage/root/kvm and lsattr /storage/root/kvm/nocow don't do it :|
m4tID 604 gen 84543 top level 5 path kvm
m4tID 605 gen 83635 top level 604 path kvm/nocow
m4tACTION scratches head
optyno cow no joy
zachary12hey everyone
zachary12i was trying to add another drive to my array using a balance, i had a power outage during, i boot but i'm getting lots of bad tree block errors
zachary12i'm able to mount with recovery and i'm running a scrub now
zachary12hopefully i'm able to correct all the errors
zachary12how can i gracefully add the new drive? remove then readd?
Nico__Hi all, I'm attempting (and failing) to set up btrfs send/receive between a generic linux system and a synology NAS. I'm hitting an error on the receiving (debian) side: "ERROR: invalid tlv in cmd. tlv_type = 25, tlv_len = 4".
Nico__Details about the synology: Linux TestNAS-Nico 3.10.102 #15152 SMP Fri Oct 6 18:15:25 CST 2017 x86_64 GNU/Linux synology_braswell_916+
Nico__btrfs version: v4.0
kdavetlv_type = 25 does not exist in the spec, afaics
Nico__what does the 'tlv' error mean?
kdavetlv is tag-length-value, encoding of the instructions in the stream
kdavehas synology implemented their own extensions to send stream?
Nico__that would be my next question :)
kdavethere are 22 commands and 24 item types
optyisn't 3.10.102 pretty old in the btrfs world?
optyand progs 4.0
Nico__yep. but it is also that what is current for a synology
optydoes it work the other way?
Nico__should have put quotes on current...
Nico__have not tested that. Give me a moment
kdaveno other search results of the message
Nico__no google came up empty
Nico__from linux --> synology works (testfile is created) but also gives an error: ERROR: attribute 12 requested but not present
Nico__for completeness the command used from the linux machine: btrfs send /srv/test2-base | ssh root@192.168.80.117 btrfs receive /volume1/test2
Nico__test2-base is a read-only snapshot
kdavethere are at least 3 people with @synology address in the btrfs git history, so we can ask them
Nico__that sounds like a good idea
kdaveyou could also report that in their forums
Nico__i'll do that
kdaveof course, 1 additional command an 1 attribute
kdavefound in the kernel sources they provide
Knorrieopen source vendor lock in!
Nico__So synology has implemented their own extension?
kdavea lot of them as I can see now
kdaveessentially means the synology-btrfs is not upstream-btrfs, there are on-disk format extensions and quite some changes to the runtime
Knorrieis it really based on 3.10?
Nico__thats what uname -a claims
Knorriesince backporting recent things that far sounds extremely painful... sounds scary
kdave3.10.77
kdavelet me do a diff
Nico__so replicating from a synology to an upstream-btrfs is not as straight forward as I had hoped :(
kdavewould be good to get their answer on that
kdaveas long as they call it 'btrfs' it should be compatible
Nico__that was the reason that we bought one
kdavetons of changes, not only to btrfs
kdaveand some of them look useful
Nico__they did not contribute back their work?
kdavenot much
kdave15 patches
kdavesince 3.0
darklingBetter than some appliance makers...
p4cmanis there a way to make a subvol under the top level fs when running a system with another subvol as its root
Knorriep4cman: mount -o subvolid=0 somewhere else
p4cmanwouldn't it be subvolid=5?
Knorrieyes
Knorriealso good
p4cmanthanks
p4cmanalso I am having some trouble in understanding how a subvolume like var/lib/machines works
p4cmanit is not listed in the toplevel
p4cmanit is under the root subvol
p4cmanit does show up in btrfs su list -p /
p4cmanlike it is named by path
p4cmanso when i snapshot the root subvol it is under, it will not be backed up right
p4cmanoh so it is just like any other subvolume
Knorrieyes
zachary12anyone around?
kilobytezachary12: just ask a question, most people don't keep watching the terminal with IRC constantly
zachary12i did a few hours ago
zachary12i was adding a drive and part way through a balance, i lost power
zachary12lots of checksum errors, running a scrub now, i can still mount in recovery
zachary12do i need to remove the drive and readd from the array before balancing again?
kilobyteadding a new device is nearly instant, so the array is in the same state as if had a balance interrupted for some other reason, no need to re-add anything
kilobytechecksum errors sound troubling though
kilobyteanything on smart or dmesg (lower layers)?
zachary12yeah i'm not sure what happened, dmesg is full of bad tree block
zachary12all drives passing smart
zachary12fi show has 5/6 drivers at 1.2tb used, and the new one is 800gb
zachary12can i just run balance start and it'll pick up and do the delta?
kilobyteany error recovered with scrub is an ex-error, so if scrub succeeded, all is fine for now (but if some hardware is bad, new errors can pop up)
zachary12https://i.imgur.com/gdZFssL.jpg that was ~16 hours ago
zachary12scrub is still going, might be a while
kilobyte"corrected errors", that's good
zachary12yeah it was at 86000 correct errors or something just before
zachary12so once the scrubs finished, what should i do to get this new drive added?
kilobyteif it's a 6-drive array, no need to do anything, unless all other disks are almost full
zachary12they're nowhere near full (40% or so used), raid10
zachary12i don't need even spread across drives?
kilobyteeh wait, bdev /dev/sdd errs: wr 115207923 sounds scary
kilobytewhat are these wr errors?
zachary12yikes ok
zachary12how do you mean
kilobyteit says /dev/sdd has _write_ errors
zachary12so that's bad?
kilobyteyeah.
multicorezachary12: "check" btrfs dev stats
darklingThat many write errors would tend to indicate that the drive's been disconnected for a while.
kilobyteas it's raid10, one device being bad means your data is still safe (and indeed, you get lots of correctable errors, 0 uncorrectable), but it's something to look at
zachary12so the drive (sdd) could be a dud?
kilobytedarkling: yeah, thanks for reminding us -- that's a less scary scenario
kilobytezachary12: or its cable (possibly unseated?)
zachary12ill check when i get home, it's a new HBA and sas breakout cable
kilobyteanyway, in such case scrub throwing and fixing plenty of correctable errors is expected, and it's raid doing its job
zachary12https://i.imgur.com/sdQC1v9.png
zachary12i think sdd is dead jim
kilobytethese errors are sticky (until -z), they could have happened in the past
zachary12?
zachary12so i need to let the scrub finish, then check sdds cable
kilobyteyeah; in theory it's faster to check sdd first, but by scrubbing first you're sure all blocks have two copies so if any of disks gets lost in the process, you lose nothing
zachary12okay, the scrub may take another day or two
zachary12they're 4tb disks
zachary12thanks everyone, appreciate it
kilobytethat long? Looking at single newish 4TB, I see 1.30T scrubbed within 2:20:41, a raid1 of four _ancient_ disks in a museal computer, 9.29G within 3:06:07
kilobytes/9.29/929/
zachary12well it's 3.5tb in 14 hours, assuming it scales linearly it'll be another day at least
zachary12https://hastebin.com/paseciyoxe.sql
zachary12that's the smart for sdd
zachary12i think there was a bug with load cycle counts for these drivers
kilobyteI mean 2×929, I divided it by half to show capacity not usage
zachary12ahh
kilobytebalance is ridiculously slow, scrub is fast
zachary12yes last time i added a drive and balanced it took 3 days
zachary12i scrub weekly via cron
kilobytebalancing on adding a new disk is required only if sum(free space on old disks)<(space on new disk)*(RAID1: 1, RAID10: 3)
gehidoreI should really enable scrub service again
zachary12hmm that's good to know