From owner-freebsd-current@FreeBSD.ORG Fri Sep 7 06:31:15 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C29E016A419; Fri, 7 Sep 2007 06:31:15 +0000 (UTC) (envelope-from kvs@binarysolutions.dk) Received: from solow.pil.dk (relay.pil.dk [195.41.47.164]) by mx1.freebsd.org (Postfix) with ESMTP id 7FC8113C459; Fri, 7 Sep 2007 06:31:14 +0000 (UTC) (envelope-from kvs@binarysolutions.dk) Received: from coruscant.local (naboo.binarysolutions.dk [80.196.17.173]) by solow.pil.dk (Postfix) with ESMTP id 819FD1CC0F6; Fri, 7 Sep 2007 08:31:11 +0200 (CEST) Received: by coruscant.local (Postfix, from userid 502) id 664CD60A14D; Fri, 7 Sep 2007 08:31:10 +0200 (CEST) To: Pawel Jakub Dawidek References: <20070905141759.GJ12013@garage.freebsd.pl> <20070905171741.GA15709@garage.freebsd.pl> From: Kenneth Vestergaard Schmidt Date: Fri, 07 Sep 2007 08:31:09 +0200 In-Reply-To: <20070905171741.GA15709@garage.freebsd.pl> (Pawel Jakub Dawidek's message of "Wed\, 5 Sep 2007 19\:17\:41 +0200") Message-ID: User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.1 (darwin) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailman-Approved-At: Fri, 14 Sep 2007 11:00:11 +0000 Cc: freebsd-current@freebsd.org Subject: Re: Unkillable and runaway processes X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Sep 2007 06:31:15 -0000 Pawel Jakub Dawidek writes: > On Wed, Sep 05, 2007 at 05:25:09PM +0200, Kenneth Vestergaard Schmidt wrote: >> Pawel Jakub Dawidek writes: >> >> I could use some pointers on what to do - is there some way I can debug >> >> this better, maybe give some better info? >> > >> > Try disabling ZIL. This looks like a bug was already reported by Kris. >> > This was already reported to OpenSolaris. >> >> We disabled ZIL, but the bug crept up again. I've just now rebooted the >> machine with DDB-support, so I can get some more info, per Kris's mail. That would be Kip's mail. >> The full state of the process hanging is 'zfs:(&tx->tx_quiesce_done_cv)' >> - it cycles between RUN, CPUx and this one. > > Hmm, this means it didn't deadlock... Here's some debugging output. PIDs 14870, 14933 and 12458 are just spinning and being useless. Funnily, all three have rename() in their backtrace. db> ps pid ppid pgrp uid state wmesg wchan cmd 20724 20720 20724 0 R+ CPU 0 sysctl 20720 1012 20720 0 S+ pause 0xd1e1a864 csh 20421 948 948 125 S kqread 0xc9ad1600 pickup 14933 14931 4351 0 R rsync 14932 14931 4351 0 S select 0xc07d68b8 ssh 14931 14930 4351 0 S select 0xc07d68b8 rsync 14930 14661 4351 0 S wait 0xd1c2b804 sh 14870 14866 4351 0 S zfs:(&tx 0xc73e6f24 rsync 14867 14866 4351 0 S select 0xc07d68b8 ssh 14866 14865 4351 0 S select 0xc07d68b8 rsync 14865 14856 4351 0 S wait 0xd26c62ac sh 14856 4355 4351 0 S piperd 0xc8f94630 perl5.8.8 14661 4355 4351 0 S piperd 0xd1fcb630 perl5.8.8 12458 12457 4351 0 R rsync 12457 12397 4351 0 S select 0xc07d68b8 rsync 12397 4355 4351 0 S piperd 0xd2090dec perl5.8.8 4360 4357 4357 0 S piperd 0xd1d55630 postdrop 4357 4348 4357 0 Ss piperd 0xd1fcbdec sendmail 4355 4354 4351 0 S wait 0xd20b0000 perl5.8.8 4354 4351 4351 0 S wait 0xd2050804 perl5.8.8 4351 4348 4351 0 Ss wait 0xd20482ac sh 4348 970 970 0 S piperd 0xc741d318 cron 1013 1 1 0 S nanslp 0xc07d07e4 getty 1012 1 1012 0 Ss+ wait 0xd1c2c000 login 1011 1 1011 0 Ss+ ttyin 0xc71d6410 getty 1010 1 1010 0 Ss+ ttyin 0xc71c5410 getty 995 1 995 0 Ss select 0xc07d68b8 inetd 970 1 970 0 Ss nanslp 0xc07d07e4 cron 963 1 963 0 Ss select 0xc07d68b8 sshd 957 948 948 125 S kqread 0xd1eff480 qmgr 948 1 948 0 Ss kqread 0xd1e14e80 master 866 1 866 0 Ss select 0xc07d68b8 ntpd 776 1 776 0 Ss select 0xc07d68b8 syslogd 721 1 721 0 Ss select 0xc07d68b8 devd 360 0 0 0 SL zfs:(&tq 0xd151877c [zil_clean] 359 0 0 0 SL zfs:(&tq 0xd1518848 [zil_clean] 358 0 0 0 SL zfs:(&tq 0xd1518914 [zil_clean] 357 0 0 0 SL zfs:(&tq 0xd15189e0 [zil_clean] 356 0 0 0 SL zfs:(&tq 0xd1518aac [zil_clean] 355 0 0 0 SL zfs:(&tq 0xd1518b78 [zil_clean] 354 0 0 0 SL zfs:(&tq 0xd10ba1e8 [zil_clean] 353 0 0 0 SL zfs:(&tq 0xd10ba2b4 [zil_clean] 352 0 0 0 SL zfs:(&tq 0xd10ba380 [zil_clean] 351 0 0 0 SL zfs:(&tq 0xd1518c44 [zil_clean] 350 0 0 0 SL zfs:(&tq 0xd1518d10 [zil_clean] 349 0 0 0 SL zfs:(&tq 0xd1518ddc [zil_clean] 348 0 0 0 SL zfs:(&tq 0xd1518ea8 [zil_clean] 347 0 0 0 SL zfs:(&tq 0xd1849050 [zil_clean] 346 0 0 0 SL zfs:(&tq 0xd184911c [zil_clean] 345 0 0 0 SL zfs:(&tq 0xd18491e8 [zil_clean] 344 0 0 0 SL zfs:(&tq 0xd10ba44c [zil_clean] 343 0 0 0 SL zfs:(&tq 0xd18492b4 [zil_clean] 342 0 0 0 SL zfs:(&tq 0xd10ba518 [zil_clean] 341 0 0 0 SL zfs:(&tq 0xd10ba5e4 [zil_clean] 340 0 0 0 SL zfs:(&tq 0xd1849380 [zil_clean] 339 0 0 0 SL zfs:(&tq 0xd184944c [zil_clean] 338 0 0 0 SL zfs:(&tq 0xd10ba6b0 [zil_clean] 337 0 0 0 SL zfs:(&tq 0xd10ba77c [zil_clean] 336 0 0 0 SL zfs:(&tq 0xd1849518 [zil_clean] 335 0 0 0 SL zfs:(&tq 0xd18495e4 [zil_clean] 334 0 0 0 SL zfs:(&tq 0xd10ba848 [zil_clean] 333 0 0 0 SL zfs:(&tq 0xd18496b0 [zil_clean] 332 0 0 0 SL zfs:(&tq 0xd184977c [zil_clean] 331 0 0 0 SL zfs:(&tq 0xd1849848 [zil_clean] 330 0 0 0 SL zfs:(&tq 0xd1849914 [zil_clean] 329 0 0 0 SL zfs:(&tq 0xd10ba914 [zil_clean] 328 0 0 0 SL zfs:(&tq 0xd18499e0 [zil_clean] 327 0 0 0 SL zfs:(&tq 0xd1849aac [zil_clean] 326 0 0 0 SL zfs:(&tq 0xd1849b78 [zil_clean] 325 0 0 0 SL zfs:(&tq 0xd10ba9e0 [zil_clean] 324 0 0 0 SL zfs:(&tq 0xd1849c44 [zil_clean] 323 0 0 0 SL zfs:(&tq 0xd1849d10 [zil_clean] 322 0 0 0 SL zfs:(&tq 0xd10baaac [zil_clean] 321 0 0 0 SL zfs:(&tq 0xd0c77848 [zil_clean] 320 0 0 0 SL zfs:(&tq 0xd10bab78 [zil_clean] 319 0 0 0 SL zfs:(&tq 0xd0c77914 [zil_clean] 318 0 0 0 SL zfs:(&tq 0xd0c779e0 [zil_clean] 317 0 0 0 SL zfs:(&tq 0xd0c77aac [zil_clean] 316 0 0 0 SL zfs:(&tq 0xd10bac44 [zil_clean] 315 0 0 0 SL zfs:(&tq 0xd0c77b78 [zil_clean] 314 0 0 0 SL zfs:(&tq 0xd10bad10 [zil_clean] 313 0 0 0 SL zfs:(&tq 0xd0c77c44 [zil_clean] 312 0 0 0 SL zfs:(&tq 0xd0c77d10 [zil_clean] 311 0 0 0 SL zfs:(&tq 0xd0c77ddc [zil_clean] 310 0 0 0 SL zfs:(&tq 0xd10baddc [zil_clean] 309 0 0 0 SL zfs:(&tq 0xd10baea8 [zil_clean] 308 0 0 0 SL zfs:(&tq 0xd1518050 [zil_clean] 307 0 0 0 SL zfs:(&tq 0xd151811c [zil_clean] 306 0 0 0 SL zfs:(&tq 0xd15181e8 [zil_clean] 305 0 0 0 SL zfs:(&tq 0xd0c77ea8 [zil_clean] 304 0 0 0 SL zfs:(&tq 0xd0ea7050 [zil_clean] 303 0 0 0 SL zfs:(&tq 0xd15182b4 [zil_clean] 302 0 0 0 SL zfs:(&tq 0xd1518380 [zil_clean] 301 0 0 0 SL zfs:(&tq 0xd151844c [zil_clean] 300 0 0 0 SL zfs:(&tq 0xd1518518 [zil_clean] 299 0 0 0 SL zfs:(&tq 0xd15185e4 [zil_clean] 298 0 0 0 SL zfs:(&tq 0xd15186b0 [zil_clean] 297 0 0 0 SL zfs:(&tq 0xd0ea7c44 [zil_clean] 296 0 0 0 SL zfs:(&tq 0xd0ea7d10 [zil_clean] 295 0 0 0 SL zfs:(&tq 0xd0ea7ddc [zil_clean] 294 0 0 0 SL zfs:(&tq 0xd0ea7ea8 [zil_clean] 293 0 0 0 SL zfs:(&tq 0xd0ea711c [zil_clean] 292 0 0 0 SL zfs:(&tq 0xd10b9050 [zil_clean] 291 0 0 0 SL zfs:(&tq 0xd10b911c [zil_clean] 290 0 0 0 SL zfs:(&tq 0xd0ea71e8 [zil_clean] 289 0 0 0 SL zfs:(&tq 0xd0ea72b4 [zil_clean] 288 0 0 0 SL zfs:(&tq 0xd10b91e8 [zil_clean] 287 0 0 0 SL zfs:(&tq 0xd0ea7380 [zil_clean] 286 0 0 0 SL zfs:(&tq 0xd10b92b4 [zil_clean] 285 0 0 0 SL zfs:(&tq 0xd0ea744c [zil_clean] 284 0 0 0 SL zfs:(&tq 0xd10b9380 [zil_clean] 283 0 0 0 SL zfs:(&tq 0xd0ea7518 [zil_clean] 282 0 0 0 SL zfs:(&tq 0xd0ea75e4 [zil_clean] 281 0 0 0 SL zfs:(&tq 0xd10b944c [zil_clean] 280 0 0 0 SL zfs:(&tq 0xd10b9518 [zil_clean] 279 0 0 0 SL zfs:(&tq 0xd10b95e4 [zil_clean] 278 0 0 0 SL zfs:(&tq 0xd10b96b0 [zil_clean] 277 0 0 0 SL zfs:(&tq 0xd10b977c [zil_clean] 276 0 0 0 SL zfs:(&tq 0xd10b9848 [zil_clean] 275 0 0 0 SL zfs:(&tq 0xd0ea76b0 [zil_clean] 274 0 0 0 SL zfs:(&tq 0xd10b9914 [zil_clean] 273 0 0 0 SL zfs:(&tq 0xd10b99e0 [zil_clean] 272 0 0 0 SL zfs:(&tq 0xd10b9aac [zil_clean] 271 0 0 0 SL zfs:(&tq 0xd10b9b78 [zil_clean] 270 0 0 0 SL zfs:(&tq 0xd10b9c44 [zil_clean] 269 0 0 0 SL zfs:(&tq 0xd10b9d10 [zil_clean] 268 0 0 0 SL zfs:(&tq 0xd10b9ddc [zil_clean] 267 0 0 0 SL zfs:(&tq 0xd10b9ea8 [zil_clean] 266 0 0 0 SL zfs:(&tq 0xd0ea777c [zil_clean] 265 0 0 0 SL zfs:(&tq 0xd10ba050 [zil_clean] 264 0 0 0 SL zfs:(&tq 0xd10ba11c [zil_clean] 263 0 0 0 SL zfs:(&tq 0xd0ea7848 [zil_clean] 262 0 0 0 SL zfs:(&tq 0xd08c7518 [zil_clean] 261 0 0 0 SL zfs:(&tq 0xd08c75e4 [zil_clean] 260 0 0 0 SL zfs:(&tq 0xd08c76b0 [zil_clean] 259 0 0 0 SL zfs:(&tq 0xd08c777c [zil_clean] 258 0 0 0 SL zfs:(&tq 0xd08c7848 [zil_clean] 257 0 0 0 SL zfs:(&tq 0xd08c7914 [zil_clean] 256 0 0 0 SL zfs:(&tq 0xd08c79e0 [zil_clean] 255 0 0 0 SL zfs:(&tq 0xd08c7aac [zil_clean] 254 0 0 0 SL zfs:(&tq 0xd0ea7914 [zil_clean] 253 0 0 0 SL zfs:(&tq 0xd0ea79e0 [zil_clean] 252 0 0 0 SL zfs:(&tq 0xd08c7b78 [zil_clean] 251 0 0 0 SL zfs:(&tq 0xd08c7c44 [zil_clean] 250 0 0 0 SL zfs:(&tq 0xd08c7d10 [zil_clean] 249 0 0 0 SL zfs:(&tq 0xd0ea7aac [zil_clean] 248 0 0 0 SL zfs:(&tq 0xd08c7ddc [zil_clean] 247 0 0 0 SL zfs:(&tq 0xd0ea7b78 [zil_clean] 246 0 0 0 SL zfs:(&tq 0xd08c7ea8 [zil_clean] 245 0 0 0 SL zfs:(&tq 0xc75d3ddc [zil_clean] 244 0 0 0 SL zfs:(&tq 0xd0c77050 [zil_clean] 243 0 0 0 SL zfs:(&tq 0xd0c7711c [zil_clean] 242 0 0 0 SL zfs:(&tq 0xc75d3d10 [zil_clean] 241 0 0 0 SL zfs:(&tq 0xd0c771e8 [zil_clean] 240 0 0 0 SL zfs:(&tq 0xc75d3b78 [zil_clean] 239 0 0 0 SL zfs:(&tq 0xc75d3aac [zil_clean] 238 0 0 0 SL zfs:(&tq 0xd0c772b4 [zil_clean] 237 0 0 0 SL zfs:(&tq 0xd0c77380 [zil_clean] 236 0 0 0 SL zfs:(&tq 0xd0c7744c [zil_clean] 235 0 0 0 SL zfs:(&tq 0xc75d39e0 [zil_clean] 234 0 0 0 SL zfs:(&tq 0xd0c77518 [zil_clean] 233 0 0 0 SL zfs:(&tq 0xd0c775e4 [zil_clean] 232 0 0 0 SL zfs:(&tq 0xc75d3914 [zil_clean] 231 0 0 0 SL zfs:(&tq 0xd0c776b0 [zil_clean] 230 0 0 0 SL zfs:(&tq 0xc75d3848 [zil_clean] 229 0 0 0 SL zfs:(&tq 0xd0c7777c [zil_clean] 228 0 0 0 SL zfs:(&tq 0xc7a6c2b4 [zil_clean] 227 0 0 0 SL zfs:(&tq 0xc75d36b0 [zil_clean] 226 0 0 0 SL zfs:(&tq 0xc75d35e4 [zil_clean] 225 0 0 0 SL zfs:(&tq 0xc7a6c380 [zil_clean] 224 0 0 0 SL zfs:(&tq 0xc75d411c [zil_clean] 223 0 0 0 SL zfs:(&tq 0xc7a6c44c [zil_clean] 222 0 0 0 SL zfs:(&tq 0xc7a6c518 [zil_clean] 221 0 0 0 SL zfs:(&tq 0xc7a6c5e4 [zil_clean] 220 0 0 0 SL zfs:(&tq 0xc7a6c6b0 [zil_clean] 219 0 0 0 SL zfs:(&tq 0xc75d41e8 [zil_clean] 218 0 0 0 SL zfs:(&tq 0xc7a6c77c [zil_clean] 217 0 0 0 SL zfs:(&tq 0xc7a6c848 [zil_clean] 216 0 0 0 SL zfs:(&tq 0xc75d42b4 [zil_clean] 215 0 0 0 SL zfs:(&tq 0xc7a6c914 [zil_clean] 214 0 0 0 SL zfs:(&tq 0xc7a6c9e0 [zil_clean] 213 0 0 0 SL zfs:(&tq 0xc75d4380 [zil_clean] 212 0 0 0 SL zfs:(&tq 0xc7a6caac [zil_clean] 211 0 0 0 SL zfs:(&tq 0xc75d444c [zil_clean] 210 0 0 0 SL zfs:(&tq 0xc7a6cb78 [zil_clean] 209 0 0 0 SL zfs:(&tq 0xc7a6cc44 [zil_clean] 208 0 0 0 SL zfs:(&tq 0xc7a6cd10 [zil_clean] 207 0 0 0 SL zfs:(&tq 0xc75d4518 [zil_clean] 206 0 0 0 SL zfs:(&tq 0xc7a6cddc [zil_clean] 205 0 0 0 SL zfs:(&tq 0xc7a6cea8 [zil_clean] 204 0 0 0 SL zfs:(&tq 0xd08c7050 [zil_clean] 203 0 0 0 SL zfs:(&tq 0xd08c711c [zil_clean] 202 0 0 0 SL zfs:(&tq 0xd08c71e8 [zil_clean] 201 0 0 0 SL zfs:(&tq 0xd08c72b4 [zil_clean] 200 0 0 0 SL zfs:(&tq 0xd08c7380 [zil_clean] 199 0 0 0 SL zfs:(&tq 0xd08c744c [zil_clean] 198 0 0 0 SL zfs:(&tq 0xc75d3ea8 [zil_clean] 197 0 0 0 SL zfs:(&tq 0xc75d3c44 [zil_clean] 196 0 0 0 SL zfs:(&tq 0xc75d377c [zil_clean] 195 0 0 0 SL zfs:(&tq 0xc75d3050 [zil_clean] 194 0 0 0 SL zfs:(&tq 0xc75d311c [zil_clean] 193 0 0 0 SL zfs:(&tq 0xc75d31e8 [zil_clean] 192 0 0 0 SL zfs:(&tq 0xc75d45e4 [zil_clean] 191 0 0 0 SL zfs:(&tq 0xc75d46b0 [zil_clean] 189 0 0 0 SL zfs:(&tq 0xc75d477c [zil_clean] 188 0 0 0 SL zfs:(&tq 0xc75d32b4 [zil_clean] 187 0 0 0 SL zfs:(&tq 0xc75d3380 [zil_clean] 186 0 0 0 SL zfs:(&tq 0xc75d344c [zil_clean] 185 0 0 0 SL zfs:(&tq 0xc75d3518 [zil_clean] 183 0 0 0 SL zfs:(&tx 0xc73e6f2c [txg_thread_enter] 182 0 0 0 SL zfs:(&tx 0xc73e6f0c [txg_thread_enter] 181 0 0 0 RL [txg_thread_enter] 180 0 0 0 SL vgeom:io 0xc73d8a08 [vdev:worker da11] 179 0 0 0 SL vgeom:io 0xc73d89c8 [vdev:worker da10] 178 0 0 0 SL vgeom:io 0xc73d8948 [vdev:worker da9] 177 0 0 0 SL vgeom:io 0xc7a1c748 [vdev:worker da8] 176 0 0 0 SL vgeom:io 0xc73d8888 [vdev:worker da7] 175 0 0 0 SL vgeom:io 0xc73d8848 [vdev:worker da6] 174 0 0 0 SL vgeom:io 0xc73d7848 [vdev:worker da5] 175 0 0 0 SL vgeom:io 0xc73d8848 [vdev:worker da6] 174 0 0 0 SL vgeom:io 0xc73d7848 [vdev:worker da5] 173 0 0 0 SL vgeom:io 0xc73d8788 [vdev:worker da4] 172 0 0 0 SL vgeom:io 0xc73d8708 [vdev:worker da3] 171 0 0 0 SL vgeom:io 0xc7a1c9c8 [vdev:worker da2] 170 0 0 0 SL vgeom:io 0xc7a1cdc8 [vdev:worker da1] 169 0 0 0 SL vgeom:io 0xc7a1cbc8 [vdev:worker da0] 168 0 0 0 SL zfs:(&tq 0xc75d4848 [spa_zio_intr_5] 167 0 0 0 SL zfs:(&tq 0xc75d4848 [spa_zio_intr_5] 166 0 0 0 SL zfs:(&tq 0xc75d4914 [spa_zio_issue_5] 165 0 0 0 SL zfs:(&tq 0xc75d4914 [spa_zio_issue_5] 164 0 0 0 SL zfs:(&tq 0xc75d49e0 [spa_zio_intr_4] 163 0 0 0 SL zfs:(&tq 0xc75d49e0 [spa_zio_intr_4] 162 0 0 0 SL zfs:(&tq 0xc75d4aac [spa_zio_issue_4] 161 0 0 0 SL zfs:(&tq 0xc75d4aac [spa_zio_issue_4] 160 0 0 0 SL zfs:(&tq 0xc75d4b78 [spa_zio_intr_3] 159 0 0 0 SL zfs:(&tq 0xc75d4b78 [spa_zio_intr_3] 158 0 0 0 SL zfs:(&tq 0xc75d4c44 [spa_zio_issue_3] 157 0 0 0 SL zfs:(&tq 0xc75d4c44 [spa_zio_issue_3] 156 0 0 0 SL zfs:(&tq 0xc75d4d10 [spa_zio_intr_2] 155 0 0 0 SL zfs:(&tq 0xc75d4d10 [spa_zio_intr_2] 154 0 0 0 SL zfs:(&tq 0xc75d4ddc [spa_zio_issue_2] 153 0 0 0 SL zfs:(&tq 0xc75d4ddc [spa_zio_issue_2] 152 0 0 0 SL zfs:(&tq 0xc75d4ea8 [spa_zio_intr_1] 151 0 0 0 SL zfs:(&tq 0xc75d4ea8 [spa_zio_intr_1] 150 0 0 0 SL zfs:(&tq 0xc7a6c050 [spa_zio_issue_1] 149 0 0 0 SL zfs:(&tq 0xc7a6c050 [spa_zio_issue_1] 148 0 0 0 SL zfs:(&tq 0xc7a6c11c [spa_zio_intr_0] 147 0 0 0 SL zfs:(&tq 0xc7a6c11c [spa_zio_intr_0] 146 0 0 0 SL zfs:(&tq 0xc7a6c1e8 [spa_zio_issue_0] 145 0 0 0 SL zfs:(&tq 0xc7a6c1e8 [spa_zio_issue_0] 108 0 0 0 SL zfs:(&ar 0xc759440c [arc_reclaim_thread] 105 0 0 0 SL zfs:(&tq 0xc75d4050 [system_taskq] 104 0 0 0 SL zfs:(&tq 0xc75d4050 [system_taskq] 43 0 0 0 SL - 0xc07d0614 [schedcpu] 42 0 0 0 SL sdflush 0xc07dbb24 [softdepflush] 41 0 0 0 SL syncer 0xc07d060c [syncer] 40 0 0 0 SL vlruwt 0xc71c7000 [vnlru] 39 0 0 0 SL psleep 0xc07d6d44 [bufdaemon] 38 0 0 0 SL pgzero 0xc07dc5d8 [pagezero] 37 0 0 0 SL psleep 0xc07dc2f4 [vmdaemon] 36 0 0 0 SL psleep 0xc07dc2bc [pagedaemon] 35 0 0 0 SL pftm 0xc047ab60 [pfpurge] 34 0 0 0 WL [irq12: psm0] 33 0 0 0 WL [irq1: atkbd0] 32 0 0 0 LL *Giant 0xc7088550 [swi0: sio] 31 0 0 0 SL - 0xc715043c [fdc0] 30 0 0 0 WL [irq25: bge1] 29 0 0 0 WL [irq24: bge0] 28 0 0 0 WL [irq18: ips0] 27 0 0 0 SL idle 0xc71962e4 [mpt_raid0] 26 0 0 0 SL idle 0xc7196000 [mpt_recovery0] 25 0 0 0 WL [irq22: mpt0] 24 0 0 0 WL [irq15: ata1] 23 0 0 0 WL [irq14: ata0] 22 0 0 0 WL [irq7: acpi0] 21 0 0 0 WL [swi2: cambio] 20 0 0 0 SL ccb_scan 0xc07b82f4 [xpt_thrd] 19 0 0 0 WL [swi6: task queue] 18 0 0 0 WL [swi6: Giant taskq] 9 0 0 0 SL - 0xc70e4e80 [thread taskq] 17 0 0 0 WL [swi5: +] 8 0 0 0 SL - 0xc70fb100 [acpi_task_2] 7 0 0 0 SL - 0xc70fb100 [acpi_task_1] 6 0 0 0 SL - 0xc70fb100 [acpi_task_0] 5 0 0 0 SL - 0xc70fb200 [kqueue taskq] 16 0 0 0 SL - 0xc07d0614 [yarrow] 4 0 0 0 SL - 0xc07ce6ec [g_down] 3 0 0 0 SL - 0xc07ce6e8 [g_up] 2 0 0 0 SL - 0xc07ce6e0 [g_event] 15 0 0 0 WL [swi1: net] 14 0 0 0 WL [swi3: vm] 13 0 0 0 RL CPU 1 [swi4: clock sio] 12 0 0 0 RL [idle: cpu0] 11 0 0 0 RL [idle: cpu1] 1 0 1 0 SLs wait 0xc709dab0 [init] 10 0 0 0 SL audit_wo 0xc07db588 [audit] 0 0 0 0 WLs [swapper] db> show proc 14870 Process 14870 (rsync) at 0xd26c3ab0: state: NORMAL uid: 0 gids: 0, 0, 5 parent: pid 14866 at 0xd1c2bab0 ABI: FreeBSD ELF32 arguments: /usr/local/bin/rsync threads: 1 100379 D zfs:(&tx 0xc73e6f24 rsync db> thread 100379 [thread pid 14870 tid 100379 ] sched_switch+0x19d: testb $0x20,0x68(%esi) db> bt Tracing pid 14870 tid 100379 td 0xd26c4e00 sched_switch(d26c4e00,0,1,181d9d4,8567e45,...) at sched_switch+0x19d mi_switch(1,0,d26c4e00,fc81da0c,c05b08c6,...) at mi_switch+0x146 sleepq_switch(d26c4e00,0,c07640c2,21b,0,...) at sleepq_switch+0x7f sleepq_wait(c73e6f24,c73e6eac,c758ddf9,1,0,...) at sleepq_wait+0x36 _cv_wait(c73e6f24,c73e6eac,c758dce6,19e,c73e6f1c,...) at _cv_wait+0x13c txg_wait_open(c73e6e00,4b65728e,0,d5,c88e3bfc,...) at txg_wait_open+0xd5 dmu_tx_wait(d0fe6800,2,0,0,0,...) at dmu_tx_wait+0xe0 zfs_freebsd_rename(fc81dba4,fc81db7c,fc81dbc0,fc81dc0c,fc81dc68,...) at zfs_freebsd_rename+0x41e VOP_RENAME_APV(c7592e00,fc81dba4,101,c05ee4d5,5009c10,...) at VOP_RENAME_APV+0x46 kern_rename(d26c4e00,bfbfd0c8,bfbfd8c8,0,fc81dd2c,...) at kern_rename+0x2e1 rename(d26c4e00,fc81dcfc,8,0,5a0a382,...) at rename+0x29 syscall(fc81dd38) at syscall+0x305 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (128, FreeBSD ELF32, rename), eip = 0x28103ccb, esp = 0xbfbfcbbc, ebp = 0xbfbfcbc8 --- db> show proc 14933 Process 14933 (rsync) at 0xd1f752ac: state: NORMAL uid: 0 gids: 0, 0, 5 parent: pid 14931 at 0xd1f6bab0 ABI: FreeBSD ELF32 arguments: /usr/local/bin/rsync threads: 1 100321 RunQ rsync db> thread 100321 [thread pid 14933 tid 100321 ] sched_switch+0x19d: testb $0x20,0x68(%esi) db> bt Tracing pid 14933 tid 100321 td 0xd1f72200 sched_switch(d1f72200,0,1,173a9d4,8567e2f,...) at sched_switch+0x19d mi_switch(1,0,d1f72200,fc73aa0c,c05b08c6,...) at mi_switch+0x146 sleepq_switch(d1f72200,0,c07640c2,21b,0,...) at sleepq_switch+0x7f sleepq_wait(c73e6f24,c73e6eac,c758ddf9,1,0,...) at sleepq_wait+0x36 _cv_wait(c73e6f24,c73e6eac,c758dce6,19e,c73e6f1c,...) at _cv_wait+0x13c txg_wait_open(c73e6e00,4b657007,0,d5,d91633b0,...) at txg_wait_open+0xd5 dmu_tx_wait(d160bd00,2,0,0,0,...) at dmu_tx_wait+0xe0 zfs_freebsd_rename(fc73aba4,fc73ab7c,fc73abc0,fc73ac0c,fc73ac68,...) at zfs_freebsd_rename+0x41e VOP_RENAME_APV(c7592e00,fc73aba4,101,119,5009c10,...) at VOP_RENAME_APV+0x46 kern_rename(d1f72200,bfbfd0c8,bfbfd8c8,0,fc73ad2c,...) at kern_rename+0x2e1 rename(d1f72200,fc73acfc,8,d1f72200,fc73ad2c,...) at rename+0x29 syscall(fc73ad38) at syscall+0x305 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (128, FreeBSD ELF32, rename), eip = 0x28103ccb, esp = 0xbfbfcbbc, ebp = 0xbfbfcbc8 --- db> show proc 12458 Process 12458 (rsync) at 0xd1e1c804: state: NORMAL uid: 0 gids: 0, 0, 5 parent: pid 12457 at 0xd1c2dab0 ABI: FreeBSD ELF32 arguments: /usr/local/bin/rsync threads: 1 100293 RunQ rsync db> thread 100293 [thread pid 12458 tid 100293 ] sched_switch+0x19d: testb $0x20,0x68(%esi) db> bt Tracing pid 12458 tid 100293 td 0xd18c5a00 sched_switch(d18c5a00,0,1,15d29d4,858acc6,...) at sched_switch+0x19d mi_switch(1,0,d18c5a00,fc5d2a0c,c05b08c6,...) at mi_switch+0x146 sleepq_switch(d18c5a00,0,c07640c2,21b,0,...) at sleepq_switch+0x7f sleepq_wait(c73e6f24,c73e6eac,c758ddf9,1,0,...) at sleepq_wait+0x36 _cv_wait(c73e6f24,c73e6eac,c758dce6,19e,c73e6f1c,...) at _cv_wait+0x13c txg_wait_open(c73e6e00,4b7d2f7f,0,d5,d26010ec,...) at txg_wait_open+0xd5 dmu_tx_wait(c987f980,2,0,0,0,...) at dmu_tx_wait+0xe0 zfs_freebsd_rename(fc5d2ba4,fc5d2b7c,fc5d2bc0,fc5d2c0c,fc5d2c68,...) at zfs_freebsd_rename+0x41e VOP_RENAME_APV(c7592e00,fc5d2ba4,101,c8c63c08,5009c10,...) at VOP_RENAME_APV+0x46 kern_rename(d18c5a00,bfbfd0a8,bfbfd8a8,0,fc5d2d2c,...) at kern_rename+0x2e1 rename(d18c5a00,fc5d2cfc,8,c714b400,7df1b67,...) at rename+0x29 syscall(fc5d2d38) at syscall+0x305 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (128, FreeBSD ELF32, rename), eip = 0x28103ccb, esp = 0xbfbfcb9c, ebp = 0xbfbfcba8 --- /Kenneth