Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iscsi-scst crash when use iser link #243

Closed
jumpapex opened this issue Jul 1, 2024 · 1 comment
Closed

iscsi-scst crash when use iser link #243

jumpapex opened this issue Jul 1, 2024 · 1 comment

Comments

@jumpapex
Copy link

jumpapex commented Jul 1, 2024

When test iscsi-scst with iser link, it will crash when initiator send an abormal NOP packet.

scst version: 3.8 release
initiator and target side are all RockyLinux 8.10

initiator:

  1. e1000 nic with IP: 192.168.1.57
  2. Mellanox ConnectX-3 Pro with IP: 192.168.2.57
    initiator rpms
    libiscsi-1.18.0-8.module+el8.10.0+1590+a67ab969.x86_64
    iscsi-initiator-utils-6.2.1.4-8.git095f59c.el8_8.x86_64

target:

  1. e1000 nic with IP: 192.168.1.58
  2. Mellanox ConnectX-3 Pro with IP: 192.168.2.58

If iscsi connection is on ethernet e1000, initiator and target work normally.

while use iser link, iscsi-scst crashed.
commands:

iscsiadm -m discovery -t sendtargets -p 192.168.2.58 -I iser
iscsiadm -m node --login -p 192.168.2.58 

it shows "Login to [iface: iser, target: iqn.2020-22.com.sh-tangram:647c6666e7f:tgt, portal: 192.168.2.58,3260] successful"

Then server crashs.
It seems iscsi-scst receive an abnormal NOP:
"Initiator sent Nop-Out with not a single PDU"

BUG check code:

static void close_conn(struct iscsi_conn *conn)
{
...
	iscsi_extracheck_is_rd_thread(conn);

	sBUG_ON(!conn->closing);

	if (conn->active_close) {
		/* We want all our already send operations to complete */
		conn->transport->iscsit_conn_close(conn, RCV_SHUTDOWN);
	} else {
		conn->transport->iscsit_conn_close(conn,
				RCV_SHUTDOWN|SEND_SHUTDOWN);
	}
...
}

I expect the iscsi-scst should not crash in this case.

I will check why initiator sends bad NOP or some modules handle it incorrectly.
BTW, are there any hints to check why iser send/receive bad NOP?

some of the dmesg:
[ 14.904391] [427]: iscsi-scst: isert_cm_evt_handler:1629:isert_cm_evt:connect request(4) status:0 portal:00000000b1227cd9 cm_id:000000006b9b5a4e
[ 14.918804] [427]: iscsi-scst: isert_device_create:1044:iser created device:000000007e67a844
[ 14.920517] [427]: iscsi-scst: isert_conn_qp_create:1172:iser created cm_id:000000006b9b5a4e qp:0x21A
[ 14.920805] [427]: iscsi-scst: isert_cm_conn_req_handler:1460:iser accepted connection cm_id:000000006b9b5a4e 192.168.2.57:32905->192.168.2.58:3260
[ 14.921274] [555]: iscsi-scst: isert_cm_evt_handler:1629:isert_cm_evt:established(9) status:0 portal:00000000b1227cd9 cm_id:000000006b9b5a4e
[ 15.912335] [1357]: scst: Using security group "default" for initiator "iqn.1994-05.com.redhat:647c6666e7f" (target iqn.2020-22.com.sh-tangram:647c6666e7f:tgt)
[ 15.913220] [1357]: iscsi-scst: Session 00000000ec6e9b8a created: target 00000000ce2644ce, tid 1, sid 0x10400003d0200, initiator iqn.1994-05.com.redhat:647c6666e7f
[ 15.913232] [1357]: iscsi-scst: Negotiated parameters: InitialR2T Yes, ImmediateData No, MaxConnections 1, MaxRecvDataSegmentLength 8192, MaxXmitDataSegmentLength 8192,
[ 15.913238] [1357]: iscsi-scst: MaxBurstLength 1048576, FirstBurstLength 65536, DefaultTime2Wait 0, DefaultTime2Retain 0,
[ 15.913242] [1357]: iscsi-scst: MaxOutstandingR2T 1, DataPDUInOrder Yes, DataSequenceInOrder Yes, ErrorRecoveryLevel 0,
[ 15.913246] [1357]: iscsi-scst: HeaderDigest None, DataDigest None, OFMarker No, IFMarker No, OFMarkInt 2048, IFMarkInt 2048, RDMAExtensions Yes
[ 15.913252] [1357]: iscsi-scst: Target parameters set for session 10400003d0200: QueuedCommands 32, Response timeout 90, Nop-In interval 30, Nop-In timeout 30
[ 18.450080] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 21.319613] [427]: iscsi-scst: ERROR: Initiator sent Nop-Out with not a single PDU
[ 21.319623] [427]: iscsi-scst: ERROR: Error -4 (iSCSI opcode 0, ITT 0)
[ 26.440611] [555]: iscsi-scst: isert_cm_evt_handler:1629:isert_cm_evt:disconnected(10) status:0 portal:00000000b1227cd9 cm_id:000000006b9b5a4e
[ 26.441526] ------------[ cut here ]------------
[ 26.441526] ------------[ cut here ]------------
[ 26.441531] kernel BUG at /root/source/scst-3.8/iscsi-scst/kernel/nthread.c:256!
[ 26.441544] invalid opcode: 0000 [#1] SMP PTI
[ 26.441550] CPU: 1 PID: 2759 Comm: iscsi_conn_clea Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.el8_10.x86_64 #1
[ 26.441557] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.21100432.B64.2301110304 01/11/2023
[ 26.441560] RIP: 0010:close_conn+0x849/0x8a0 [usistgt]
[ 26.441578] Code: 00 48 c7 c1 38 05 04 c1 ff b3 68 01 00 00 48 c7 c2 3c 34 04 c1 48 c7 c6 4a 34 04 c1 e8 70 24 cb ff 41 5a 41 5b e9 b1 f8 ff ff <0f> 0b 31 f6 48 8d 7c 24 38 4c 8d ab 18 02 00 00 e8 62 70 31 d0 eb
[ 26.441584] RSP: 0018:ffffb3640480fe70 EFLAGS: 00010246
[ 26.441590] RAX: 0000000000000000 RBX: ffff978b585dd100 RCX: 0000000000000000
[ 26.441594] RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffff978ad7752a80
[ 26.441598] RBP: ffff978ae5d6e680 R08: ffff978defc73bb8 R09: 000000062810e680
[ 26.441602] R10: 0000000000000000 R11: ffff978defc71dc4 R12: ffff978b1c7a5500
[ 26.441604] R13: ffffb3640219fda8 R14: ffffffffc1037540 R15: ffff978ad7752a80
[ 26.441608] FS: 0000000000000000(0000) GS:ffff978defc40000(0000) knlGS:0000000000000000
[ 26.441612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 26.441615] CR2: 00007f2247988974 CR3: 00000003e6c10003 CR4: 00000000003706e0
[ 26.441662] Call Trace:
[ 26.441670] ? __die_body+0x1a/0x60
[ 26.441683] ? die+0x2a/0x50
[ 26.441690] ? do_trap+0xe7/0x110
[ 26.441696] ? close_conn+0x849/0x8a0 [usistgt]
[ 26.441707] ? do_invalid_op+0x36/0x40
[ 26.441712] ? close_conn+0x849/0x8a0 [usistgt]
[ 26.441722] ? invalid_op+0x14/0x20
[ 26.441731] ? start_close_conn+0x70/0x70 [usistgt]
[ 26.441741] ? close_conn+0x849/0x8a0 [usistgt]
[ 26.441752] ? start_close_conn+0x70/0x70 [usistgt]
[ 26.441761] close_conn_thr+0xa/0x20 [usistgt]
[ 26.441771] kthread+0x134/0x150
[ 26.441780] ? set_kthread_struct+0x50/0x50
[ 26.441787] ret_from_fork+0x35/0x40

@jumpapex
Copy link
Author

jumpapex commented Jul 1, 2024

sorry, the issue is resolved after I rebuilt all modules

@jumpapex jumpapex closed this as completed Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant