Comments for CXL.cache transaction layer specification--clarifications needed

The version we are using: CXL 3.0 spec

Relevant sections in spec:

Table of Contents

Suggestion 1

Rule 11 in

3.2.5.14 General Assumptions

11. The Host must not send a second snoop request to an address until all responses, plus data if required, for the prior snoop are collected.
seems verbose as

3.2.5.5 Multiple Snoops to the Same Address

The host is only allowed to have one snoop pending at a time per cacheline address per device. The host must wait until it has received both the snoop response and all IWB data (if any) before dispatching the next snoop to that address.
already says the same thing.

Proposed Change

Remove rule 11 in section 3.2.5.14:

3.2.5.14 General Assumptions

11. The Host must not send a second snoop request to an address until all responses, plus data if required, for the prior snoop are collected.

Suggestion 2

According to table 3-18, a RdShared can only result in GO-S, not GO-E.
RdShared could get GO-E

However, it may be beneficial to grant the device Exclusive ownership so that it can silently upgrade to Modified state if needed.

RdShared could get GO-E

In the above picture the requestor sends RdShared, Host sends SnpData to the owner (in M state), and the owner responds with RspIFwdM. Now we know the requestor will have the sole copy. However the spec says RdShared can only get GO-S/I/Err back, not GO-E.

Proposed change:

Allowing RdShared to get E state.

RdShared could get GO-E

Suggestion 3

Section 3.2.5.2 allows us to delete half of section 3.2.5.3.

Details

Here is the second part of 3.2.5.3:

3.2.5.3 Device/Host Snoop/WritePull Assumptions

... the host may not launch a WritePull for a write until the host has received the snoop response (including data in case of Rsp*Fwd*) for any snoops to the pending writes address. Any violation of these requirements will mean that the Bogus field on the D2H Data channel will be unreliable.
We believe that the only purpose of this part of the rule is to disallow examples such as the following:
Rd and Evict should 
				     not interrupt each other 2
In this example, a RdOwn request from device 2 is issued in parallel with a DirtyEvict from device 1. The host receives the RdOwn first, and responds sending a SnpInv to device 1. It then receives the DirtyEvict, and responds by sending GO_WritePull to device 1.

However, this example is already disallowed by the following text in rule 3.2.5.2:

3.2.5.2 Device/Host Snoop-GO-Data Assumptions

... When the host is sending a snoop to the device, the requirement is that no GO response will be sent to any requests with that address in the device until after the Host has received a response for the snoop and all implicit writeback (IWB) data (dirty data forwarded in response to a snoop) has been received. ...
Whenever a WritePull is sent, a GO is always accompanying, either as a separate message (in the case of WrInv), or as a combined message (WrCur, ItoMWr, DirtyEvict, CleanEvict). Therefore section 3.2.5.3 is implied by 3.2.5.2. We believe the CXL specification would be clearer if either:

Proposed change:

Concretely, we propose the following change to the specification:

3.2.5.3 Device/Host Snoop/WritePull Assumptions

The device requires that the host cannot have both a WritePull and H2D Snoop active on CXL.cache to a given 64-byte address. The host may not launch a snoop to a 64- byte address until all WritePull data from that address has been received by the host. Conversely, the host may not launch a WritePull for a write until the host has received the snoop response (including data in case of Rsp*Fwd*) for any snoops to the pending writes address. Any violation of these requirements will mean that the Bogus field on the D2H Data channel will be unreliable.

Suggestion 4

Section 3.2.5.4 can be removed.

Details

Here is section 3.2.5.4:

3.2.5.4 Snoop Responses and Data Transfer on CXL.cache Evicts
......If a device Evict transaction has been issued on the CXL.cache D2H request channel, but has not yet processed its WritePull from the host, and a snoop hits the writeback, the device must track this snoop hit if cache state is changed, which excludes the case when SnpCur results in a RspVFwdV response. When the device begins to process the WritePull, if snoop hit is tracked the device must set the Bogus field in all the D2H data messages sent to the host. The intent is to communicate to the host that the request data was already sent as IWB data, so the data from the Evict is potentially stale.

This rule applies to transaction sequences of two forms, and its purpose is to mandate the bogus field to be set in certain D2H Data messages. The first form of transaction sequences is the following:

Rd and Evict should 
					  not interrupt each other 2

However, we observe that transaction sequences like this have already been disallowed under 3.2.5.2, as the GO_WritePull will not be sent until after the snoop returns Data and response.

The second form of transaction sequence is the following:

Rd and Evict should 
							 not interrupt each other 2

In this situation, we propose to allow the host to send GO_WritePull_Drop instead of GO_WritePull:

Rd and Evict should 
					  not interrupt each other 2

because this means that no data will be sent and hence there is no need to require the bogus field to be set.

Proposed change

First, we propose to update table 3-18, allowing DirtyEvict to get GO_WritePull_Drop, to accommodate the above transaction sequence.

Rd and Evict should 
					  not interrupt each other 2

We also propose to delete 3.2.5.4 entirely:

3.2.5.4 Snoop Responses and Data Transfer on CXL.cache Evicts

To snoop cache evictions (for example, DirtyEvict) and maintain an orderly transfer of snoop ownership from the device to the host, cache evictions on CXL.cache must adhere to the following protocol. If a device Evict transaction has been issued on the CXL.cache D2H request channel, but has not yet processed its WritePull from the host, and a snoop hits the writeback, the device must track this snoop hit if cache state is changed, which excludes the case when SnpCur results in a RspVFwdV response. When the device begins to process the WritePull, if snoop hit is tracked the device must set the Bogus field in all the D2H data messages sent to the host. The intent is to communicate to the host that the request data was already sent as IWB data, so the data from the Evict is potentially stale.