Date: Sat, 19 Feb 2011 10:04:15 +0330 From: "Payam Poursaied" <me@payam124.com> To: "'Jeremy Chadwick'" <freebsd@jdc.parodius.com> Cc: freebsd-apache@freebsd.org Subject: RE: apache 100% cpu usage Message-ID: <008701cbcfff$0a721590$1f5640b0$@com> In-Reply-To: <20110219053032.GA76790@icarus.home.lan> References: <190201cbcfec$c76297f0$5627c7d0$@com> <20110219053032.GA76790@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. ------=_NextPart_000_0082_01CBD01C.5B4C4D60 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit > -----Original Message----- > From: Jeremy Chadwick [mailto:freebsd@jdc.parodius.com] > Sent: Saturday, February 19, 2011 9:01 AM > To: Payam Poursaied > Cc: freebsd-apache@freebsd.org > Subject: Re: apache 100% cpu usage > > On Sat, Feb 19, 2011 at 07:53:31AM +0330, Payam Poursaied wrote: > > Hi all > > I faced with a problem using apache 1.3.42 on FreeBSD 8.0-RELEASE. > > The problem is that httpd processes frequently reach 100% CPU usage and stay > > at this level of CPU usage. > > This is a sample of top output. > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 77051 www 1 118 0 208M 86080K CPU4 4 87.6H 100.00% httpd > > 9403 www 1 118 0 215M 93340K CPU2 2 65.0H 100.00% httpd > > 78430 www 1 118 0 226M 100M CPU0 0 58:57 100.00% httpd > > 95332 www 1 50 0 205M 83624K sbwait 12 0:02 4.49% httpd > > 89664 www 1 46 0 210M 88144K sbwait 11 0:05 2.59% httpd > > > > I think there might be a bug that caused something like endless loop but I > > am newbie to gdb and debugging and I could not find any relevant point to > > get into it. > > I tried to compile apache in debug mode and then wait to see a suspicious > > process and then attached GDB66 to that cpu intensive httpd process to find > > out what is going on. It seems the problem is related to something in > > /lib/libc.so.7 which I could not find out how to compile it in debug mode. > > > > The result of "list" and "where" command of GDB are as below, I appreciate > > if you could point me to right direction to examine this problem deeply. > > As an extra information, mod_perl is heavily used trough this apache > > > > (gdb) l > > 5640 > > 5641 #ifdef TPF > > 5642 EBW_AREA input_parms; > > 5643 ecbptr()->ebrout = PRIMECRAS; > > 5644 input_parms = * (EBW_AREA *)(&(ecbptr()->ebw000)); > > 5645 #endif > > 5646 > > 5647 MONCONTROL(0); > > 5648 > > 5649 common_init(); > > (gdb) w > > Ambiguous command "w": watch, wh, whatis, where, while, while-stepping, > > winheight, ws. > > (gdb) where > > #0 0x000000080093a9f5 in ?? () from /lib/libc.so.7 > > #1 0x000000080093d505 in ?? () from /lib/libc.so.7 > > #2 0x000000080093d702 in ?? () from /lib/libc.so.7 > > #3 0x000000080094352e in ?? () from /lib/libc.so.7 > > #4 0x0000000800944924 in realloc () from /lib/libc.so.7 > > #5 0x000000080307a50f in Perl_safesysrealloc () from > > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > > #6 0x00000008030a6156 in Perl_sv_grow () from > > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > > #7 0x000000080309e32e in Perl_sv_vcatpvfn () from > > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > > #8 0x00000008030a7b42 in Perl_vnewSVpvf () from > > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > > #9 0x00000008030a7bf8 in Perl_newSVpvf () from > > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > > #10 0x000000080303da32 in Perl_gv_fetchmethod_autoload () from > > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > > #11 0x000000080308edca in S_method_common () from > > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > > #12 0x000000080308f462 in Perl_pp_method () from > > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > > #13 0x000000080308eb9e in Perl_runops_standard () from > > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > > #14 0x0000000803036191 in Perl_call_sv () from > > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > > #15 0x0000000802eabe36 in perl_call_handler () from > > /usr/local/libexec/apache/libperl.so > > #16 0x0000000802eac87a in perl_run_stacked_handlers () from > > /usr/local/libexec/apache/libperl.so > > #17 0x0000000802eae2b0 in perl_handler () from > > /usr/local/libexec/apache/libperl.so > > #18 0x000000000040ee42 in ap_invoke_handler (r=0x804624060) at > > http_config.c:476 > > #19 0x0000000000427849 in process_request_internal (r=0x804624060) at > > http_request.c:1299 > > #20 0x00000000004278a4 in ap_process_request (r=0x804624060) at > > http_request.c:1315 > > #21 0x000000000041d423 in child_main (child_num_arg=14) at http_main.c:4885 > > #22 0x000000000041d734 in make_child (s=0x800c05060, slot=14, > > now=1297853738) at http_main.c:5055 > > #23 0x000000000041db6b in perform_idle_server_maintenance () at > > http_main.c:5256 > > #24 0x000000000041e159 in standalone_main (argc=1, argv=0x7fffffffebd8) at > > http_main.c:5520 > > #25 0x000000000041e7a4 in main (argc=1, argv=0x7fffffffebd8) at > > http_main.c:5773 > > This looks like you have a mod_perl problem, or a script that's doing > it. It's probably not an Apache issue. > > ktrace would be of more help here than gdb. It will at least show you > what the children are doing in real-time, syscall-wise. Correlating > this with underlying perl code is up to you/your problem, sadly. > > Good luck. > Thank you for your fast response. I used truss and ktrace, but there wasn't any output to show any system calls or .. may I know how I can get inside and explore the problem? How it is possible to build /lib/libc.so.7 in debug mode? Or any general how-to/instruction on debugging is appreciated. Best Regards -payam ------=_NextPart_000_0082_01CBD01C.5B4C4D60 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIQQjCCBDIw ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0 ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0 ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+ QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH 7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0 YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1 OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIHJzCCBg+gAwIBAgIQ fJ4K6co7FmGJoF7AP1XRaDANBgkqhkiG9w0BAQUFADCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgT AlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3 b3JrMSEwHwYDVQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VS Rmlyc3QtQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDAeFw0xMDA0MDQwMDAwMDBaFw0x MTA0MDQyMzU5NTlaMIHaMTUwMwYDVQQLEyxDb21vZG8gVHJ1c3QgTmV0d29yayAtIFBFUlNPTkEg Tk9UIFZBTElEQVRFRDFGMEQGA1UECxM9VGVybXMgYW5kIENvbmRpdGlvbnMgb2YgdXNlOiBodHRw Oi8vd3d3LmNvbW9kby5uZXQvcmVwb3NpdG9yeTEfMB0GA1UECxMWKGMpMjAwMyBDb21vZG8gTGlt aXRlZDEYMBYGA1UEAxMPUGF5YW0gUG91cnNhaWVkMR4wHAYJKoZIhvcNAQkBFg9tZUBwYXlhbTEy NC5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDHzG9pJwFE9yGZxr14vAyiybKW Rm2V955LwsEIPWKg/+FMrkPBck5vaUk+0BHBWhMroMHUZzbUkRbbrxQJItVc5dOVpWukjJWPC/kM UbNDOjB37tnGopMIbAtneLWDfJAnFG5Z/SfJ5fSOQqaeKSLR1axeZ2gvV+1+ZEFJIGtg/7OXeO/Q ASVD8QWdUOJ1+OxjLLoBR2KEBiG+LZWYg0NYWHuBi+R8cEC48GTM7G1EMgTPgRupxWrji6w+DsQ2 A72lyzNV2gvy/2a7ab5vQ2TXSon40YAjCIkRmLeNhIzaiJeCZPNjRfPlMw5uIssixCIHYUmdAArp UVC7uMFSY+9jecnL2PLlQXro8tGkHu4oTYHVC3yYn2wiSzzKq4l1dTlMFecsp5eO2gQfDdc6Kw9V QWUMhI69UW/0CGSJpdrG1ud79bsn+bCXHxudA+P1o1lpAB+Zii17/wFcRe88xK/IP1oy0l8J4chp IP9V/9KW7iMyDWDdoEyoU1WBVcQMO7lHtaQF2UbpVjPvqP0M/tGoE6P40HSIJXF8apiKn/0DH14g 83Pcx1yCvWZ3WPBThNX3eWMpWMW2JnAOApW3Y9LQT8O8XF3ohsnpwoJpZsIoaJK7j1+WW/MVeoSk sC8oSlyCu7D6eIRkurwQNNS8ymz2eg3xd4kYzcId3X9NwnxUHQIDAQABo4ICETCCAg0wHwYDVR0j BBgwFoAUiYJnfcSdJnAAS7RQSHzePa4Ebn0wHQYDVR0OBBYEFE+jVEkNnPkveugYQP4EJM3VejNg MA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEE AbIxAQMFAjARBglghkgBhvhCAQEEBAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzAp BggrBgEFBQcCARYdaHR0cHM6Ly9zZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBM oEqgSIZGaHR0cDovL2NybC5jb21vZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50 aWNhdGlvbmFuZEVtYWlsLmNybDBKoEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VS Rmlyc3QtQ2xpZW50QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYG CCsGAQUFBzAChipodHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYI KwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAaBgNVHREEEzARgQ9tZUBwYXlhbTEy NC5jb20wDQYJKoZIhvcNAQEFBQADggEBAEhaBABgM+NZXZM0JOmtCEIF55E6B7aHfkZuZA5lD8qq DXvGUb4RHDSQUn31NmMX0PDoirPY6xIgAIZKNKZCo/AUOBVyfOQIxxI3/GwyrndySZsT7o7t5qIy mFWNPUzTCtqgBJehttrvnLH9cBL1mRlSJobsSTYVltFxtygxahqGuX0y0iaYYqZKWHbvLNde4scL QYUN9pw/vyHWjOAu2QMNbZ2/RNNd+cIM42B2jZwhgF21R/ZnSRMm8MV9+1heZQ1Ddx0Hv9eUS+aN WoRV62OwHkI/eez76g7KIq8kkQZaZQElVSICp+o3Mv367pg/dH5th/Kpu/lW4+RZq310Zb8xggW2 MIIFsgIBATCBwzCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExh a2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8v d3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1dGhlbnRp Y2F0aW9uIGFuZCBFbWFpbAIQfJ4K6co7FmGJoF7AP1XRaDAJBgUrDgMCGgUAoIICxzAYBgkqhkiG 9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xMTAyMTkwNjM0MTFaMCMGCSqGSIb3 DQEJBDEWBBS8XtfheoFYWrzc7/QhUfEh1rO3qzCBtwYJKoZIhvcNAQkPMYGpMIGmMAsGCWCGSAFl AwQBKjALBglghkgBZQMEARYwCgYIKoZIhvcNAwcwCwYJYIZIAWUDBAECMA4GCCqGSIb3DQMCAgIA gDAHBgUrDgMCBzANBggqhkiG9w0DAgIBQDANBggqhkiG9w0DAgIBKDAHBgUrDgMCGjALBglghkgB ZQMEAgMwCwYJYIZIAWUDBAICMAsGCWCGSAFlAwQCATAKBggqhkiG9w0CBTCB1AYJKwYBBAGCNxAE MYHGMIHDMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBD aXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cu dXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRp b24gYW5kIEVtYWlsAhB8ngrpyjsWYYmgXsA/VdFoMIHWBgsqhkiG9w0BCRACCzGBxqCBwzCBrjEL MAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UE ChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5j b20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFp bAIQfJ4K6co7FmGJoF7AP1XRaDANBgkqhkiG9w0BAQEFAASCAgCNQfe7euUsQDr8so8EMus/JZai LRdjl8C9ZTzy1xIHHgS6m0PgjKei8V4YjUVSb6s5wjkCDEB+odWaz0WVua/ZsYms7h/yvRqXxpJC R9WMmTNuqVbYn+hAlRTG2vVgDYWoJZ3t7thq5evjOryrdMjiNdu4k3J677Fhlo630nWTAryuLxSF p9FbrD1MsgfKNK2bBd0BuMPzQ/E3xvXKyk2cl8dLhNnBqtYNsquIAj2Hr4Pi4ns8AHGVJvEcfsaN DCASHO8p7UBszxlVpJuQCpV+Z2DbvmVRWfBTGPvXPKFUVxGEzG9369aMi1qX8/kRE4XYFHde7AUj oF723Capek9c+yG7xJofrj6Jlq8SPayoP5iHFYUw6Ii0jCeLxVOCKWmq4xIENtJKcAPcgGpKBsR3 lrepZPhXLi63h35wHsptxtryGOOf4u+vAE1PjkAdCNbuwABQDFmxKpuSYxRJfvpp9fOA6zxxzLYN 6kfPLmCotS0Q2O66QyHVMLTAuRqvHwESoZxowQLCf0wb97bT9x6F7BZ+kt+M1alQTTX85yH3VbG9 skR3/GEP9R1uD5DMi/CGINr86NgGWIu/m2KLR6GWKLGXlmiadqFZP1jcns7FzKVa9V0N2qpI743w udWlHTf2c9F1fdmkpNtdh/sJZzR5BEVt9fTnSh3QJLgCAUiWAgAAAAAAAA== ------=_NextPart_000_0082_01CBD01C.5B4C4D60--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?008701cbcfff$0a721590$1f5640b0$>